update docs

docs/tutorial/extend/callback.md

 
 ## Write a Callback
 
+__Everything__ other than the training iterations happen in the callbacks.
+Most of the fancy things you want to do will probably end up here.
+
 The time where each callback method gets called is demonstrated in this snippet.
 ```python
 def train(self):
@@ -37,13 +40,13 @@ You can overwrite any of the following methods to define a new callback:
   To access tensors/ops which are already defined,
   you can use TF methods such as
   [`graph.get_tensor_by_name`](https://www.tensorflow.org/api_docs/python/tf/Graph#get_tensor_by_name).
-  If you're using a `TowerTrainer` instance, more tools are available:
+  If you're using a `TowerTrainer`, more tools are available:
 
   - Use `self.trainer.tower_func.towers` to access the
   	[tower handles](../../modules/tfutils.html#tensorpack.tfutils.tower.TowerTensorHandles),
   	and therefore the tensors in each tower.
   - [self.get_tensors_maybe_in_tower()](../../modules/callbacks.html#tensorpack.callbacks.Callback.get_tensors_maybe_in_tower)
-  	is a helper function to access tensors in the first training tower.
+  	is a helper function to look for tensors first globally, then in the first training tower.
   - [self.trainer.get_predictor()](../../modules/train.html#tensorpack.train.TowerTrainer.get_predictor)
   	is a helper function to create a callable under inference mode.
 
@@ -105,7 +108,7 @@ You can overwrite any of the following methods to define a new callback:
 * Write stuff to the monitor backend, by `self.trainer.monitors.put_xxx`.
   The monitors might direct your events to TensorFlow events file, JSON file, stdout, etc.
   You can access history monitor data as well. See the docs for [Monitors](../../modules/callbacks.html#tensorpack.callbacks.Monitors)
-* Access the current status of training, such as `self.epoch_num`, `self.global_step`. See [here](../../modules/callbacks.html#tensorpack.callbacks.Callback)
+* Access the current status of training, such as `self.epoch_num`, `self.global_step`. See docs of [Callback](../../modules/callbacks.html#tensorpack.callbacks.Callback)
 * Stop training by `raise StopTraining()` (with `from tensorpack.train import StopTraining`).
 * Anything else that can be done with plain python.
 
@@ -120,4 +123,5 @@ You can overwrite any of the following methods to define a new callback:
   [PeriodicTrigger](../../modules/callbacks.html#tensorpack.callbacks.PeriodicTrigger),
   [PeriodicCallback](../../modules/callbacks.html#tensorpack.callbacks.PeriodicCallback),
   or [EnableCallbackIf](../../modules/callbacks.html#tensorpack.callbacks.EnableCallbackIf).
+	Of course you also have the freedom to implement "what to do" and "when to do" altogether.
 

docs/tutorial/extend/model.md

@@ -3,7 +3,7 @@
 
 The first thing to note: __you never have to write a layer__.
 Tensorpack layers are nothing but wrappers of symbolic functions.
-You can use any symbolic functions you have written or seen elsewhere with or without tensorpack layers.
+In tensorpack, you can use any symbolic functions you have written or seen elsewhere with or without tensorpack layers.
 
 If you would like, you can make a symbolic function become a "layer" by following some simple rules, and then gain benefits from the framework.
 
@@ -32,8 +32,7 @@ By making a symbolic function a "layer", the following things will happen:
 + `argscope` will work for all its arguments except the input tensor(s).
 + It will work with `LinearWrap`: you can use it if the output of one layer matches the input of the next layer.
 
-There are also some (non-layer) symbolic functions in the `tfutils.symbolic_functions` module.
-There is not a rule about what kind of symbolic functions should be made a layer -- they are quite
+There is no rule about what kind of symbolic functions should be made a layer -- they are quite
 similar anyway. However, in general, I define the following symbolic functions as layers:
 + Functions which contain variables. A variable scope is almost always helpful for such functions.
 + Functions which are commonly referred to as "layers", such as pooling. This makes a model

examples/FasterRCNN/NOTES.md

@@ -40,7 +40,7 @@ Model:
 
 Speed:
 
-1. The training will start very slow due to convolution warmup, until about 3k steps to reach a maximum speed.
+1. The training will start very slow due to convolution warmup, until about 10k steps to reach a maximum speed.
 	 Then the training speed will slowly decrease due to more accurate proposals.
 
 2. Inference is not quite fast, because either you disable convolution autotune and end up with

examples/FasterRCNN/README.md

@@ -53,12 +53,12 @@ MaskRCNN results contain both bbox and segm mAP.
 
 |Backbone|`FASTRCNN_BATCH`|resolution |schedule|mAP (bbox/segm)|Time         |
 |   -    |    -           |    -      |   -    |   -           |   -         |
-|R-50    |64              |(600, 1024)|280k    |33.1           |18h on 8 V100|
-|R-50    |512             |(800, 1333)|280k    |35.6           |55h on 8 P100|
-|R-50    |512             |(800, 1333)|360k    |36.7           |49h on 8 V100|
-|R-50    |256             |(800, 1333)|280k    |36.8/32.1      |39h on 8 P100|
-|R-50    |512							|(800, 1333)|360k    |37.7/33.0      |72h on 8 P100|
-|R-101   |512             |(800, 1333)|280k    |40.1/34.4      |70h on 8 P100|
+|R-50    |64              |(600, 1024)|280k    |33.1           |18h on 8 V100s|
+|R-50    |512             |(800, 1333)|280k    |35.6           |55h on 8 P100s|
+|R-50    |512             |(800, 1333)|360k    |36.6           |49h on 8 V100s|
+|R-50    |256             |(800, 1333)|280k    |36.8/32.1      |39h on 8 P100s|
+|R-50    |512							|(800, 1333)|360k    |37.8/33.2      |51h on 8 V100s|
+|R-101   |512             |(800, 1333)|280k    |40.1/34.4      |70h on 8 P100s|
 
 The two 360k models have identical configurations with
 `R50-C4-2x` configuration in

tensorpack/utils/logger.py

@@ -98,8 +98,12 @@ def set_logger_dir(dirname, action=None):
         # unload and close the old file handler, so that we may safely delete the logger directory
         _logger.removeHandler(_FILE_HANDLER)
         del _FILE_HANDLER
-    # If directory exists and nonempty (ignore hidden files), prompt for action
-    if os.path.isdir(dirname) and len([x for x in os.listdir(dirname) if x[0] != '.']):
+
+    def dir_nonempty(dirname):
+        # If directory exists and nonempty (ignore hidden files), prompt for action
+        return os.path.isdir(dirname) and len([x for x in os.listdir(dirname) if x[0] != '.'])
+
+    if dir_nonempty(dirname):
         if not action:
             _logger.warn("""\
 Log directory {} exists! Use 'd' to delete it. """.format(dirname))
@@ -114,13 +118,9 @@ Press any other key to exit. """)
             shutil.move(dirname, backup_name)
             info("Directory '{}' backuped to '{}'".format(dirname, backup_name))  # noqa: F821
         elif act == 'd':
-            try:
-                shutil.rmtree(dirname)
-            except OSError:
-                num_files = len([x for x in os.listdir(dirname) if x[0] != '.'])
-                if num_files > 0:
-                    raise
-
+            shutil.rmtree(dirname, ignore_errors=True)
+            if dir_nonempty(dirname):
+                shutil.rmtree(dirname, ignore_errors=False)
         elif act == 'n':
             dirname = dirname + _get_time_str()
             info("Use a new log directory {}".format(dirname))  # noqa: F821