fix batchnorm

docs/user/callbacks.md

@@ -4,7 +4,7 @@
 Apart from the actual training iterations that minimizes the cost,
 you almost surely would like to do something else during training.
 Callbacks are such an interface to describe what to do besides the
-training iteration defined by the trainers.
+training iterations defined by the trainers.
 
 There are several places where you might want to do something else:
 
@@ -14,39 +14,39 @@ There are several places where you might want to do something else:
 * Between epochs (e.g. save the model, run some validation)
 * After the training (e.g. send the model somewhere, send a message to your phone)
 
-By writing callbacks to implement these tasks, you can reuse them as long as
+By writing callbacks to implement these tasks, you can reuse the code as long as
 you're using tensorpack trainers. For example, these are the callbacks I used when training
 a ResNet:
 
 ```python
 TrainConfig(
-		# ...
-		callbacks=[
-				# save the model every epoch
-				ModelSaver(),
-				# run inference on another Dataflow every epoch, compute top1/top5 classification error and save them
-				InferenceRunner(dataset_val, [
-						ClassificationError('wrong-top1', 'val-error-top1'),
-						ClassificationError('wrong-top5', 'val-error-top5')]),
-				# schedule the learning rate based on epoch number
-				ScheduledHyperParamSetter('learning_rate',
-																	[(30, 1e-2), (60, 1e-3), (85, 1e-4), (95, 1e-5)]),
-				# allow manually setting the learning rate during training
-				HumanHyperParamSetter('learning_rate'),
-			  # send validation error to my phone through pushbullet
-				SendStat('curl -u your_id_xxx: https://api.pushbullet.com/v2/pushes \\
-								 -d type=note -d title="validation error" \\
-								 -d body={val-error-top1} > /dev/null 2>&1',
-								 'val-error-top1')
-		],
-		extra_callbacks=[		# these are already enabled by default
-			# maintain and summarize moving average of some tensors (e.g. training loss, training error)
-			MovingAverageSummary(),
-			# draw a nice progress bar
-			ProgressBar(),
-			# print all the statistics I've created and scalar tensors I've summarized
-			StatPrinter(),
-		]
+  # ...
+  callbacks=[
+    # save the model every epoch
+    ModelSaver(),
+    # run inference on another Dataflow every epoch, compute top1/top5 classification error and save them
+    InferenceRunner(dataset_val, [
+        ClassificationError('wrong-top1', 'val-error-top1'),
+        ClassificationError('wrong-top5', 'val-error-top5')]),
+    # schedule the learning rate based on epoch number
+    ScheduledHyperParamSetter('learning_rate',
+                              [(30, 1e-2), (60, 1e-3), (85, 1e-4), (95, 1e-5)]),
+    # allow manually setting the learning rate during training
+    HumanHyperParamSetter('learning_rate'),
+    # send validation error to my phone through pushbullet
+    SendStat('curl -u your_id_xxx: https://api.pushbullet.com/v2/pushes \\
+               -d type=note -d title="validation error" \\
+               -d body={val-error-top1} > /dev/null 2>&1',
+               'val-error-top1')
+  ],
+  extra_callbacks=[    # these are already enabled by default
+    # maintain and summarize moving average of some tensors (e.g. training loss, training error)
+    MovingAverageSummary(),
+    # draw a nice progress bar
+    ProgressBar(),
+    # print all the statistics I've created and scalar tensors I've summarized
+    StatPrinter(),
+  ]
 )
 ```
 

docs/user/trainer.md

@@ -5,14 +5,15 @@
 
 Training is basically **running something again and again**.
 Tensorpack base trainer implements the logic of *running the iteration*,
-and other trainers implement what the iteration is.
+and other trainers implement *what the iteration is*.
 
 Most neural network training tasks are single-cost optimization.
 Tensorpack provides some trainer implementations for such tasks.
 These trainers will by default minimizes `ModelDesc.cost`,
-therefore you can use these trainers as long as you set `self.cost` in `ModelDesc._build_graph()`.
+therefore you can use these trainers as long as you set `self.cost` in `ModelDesc._build_graph()`,
+as did in most examples.
 
-The existing trainers were implemented with a TensorFlow queue to prefetch and buffer
+Most existing trainers were implemented with a TensorFlow queue to prefetch and buffer
 training data, which is significantly faster than
 a naive `sess.run(..., feed_dict={...})`.
 There are also multi-GPU trainers which includes the logic of data-parallel multi-GPU training,
@@ -23,11 +24,11 @@ To use trainers, pass a `TrainConfig` to configure them:
 
 ````python
 config = TrainConfig(
-					 dataflow=my_dataflow,
-					 optimizer=tf.train.AdamOptimizer(0.01),
-					 callbacks=[...]
+           dataflow=my_dataflow,
+           optimizer=tf.train.AdamOptimizer(0.01),
+           callbacks=[...]
            model=MyModel()
-				 )
+         )
 
 # start training:
 # SimpleTrainer(config).train()
@@ -47,7 +48,7 @@ Some trainer takes data from a TensorFlow reading pipeline instead of a Dataflow
 ([PTB example](../examples/PennTreebank)).
 
 
-## Write trainers
+## Write a trainer
 
 The existing trainers should be enough for single-cost optimization tasks. If you
 want to do something inside the trainer, consider writing it as a callback, or

examples/CTC-TIMIT/train-timit.py

@@ -22,8 +22,8 @@ from timitdata import TIMITBatch
 BATCH = 64
 NLAYER = 2
 HIDDEN = 128
-NR_CLASS = 61 + 1
-FEATUREDIM = 39
+NR_CLASS = 61 + 1   # 61 phoneme + epsilon
+FEATUREDIM = 39     # MFCC feature dimension
 
 
 class Model(ModelDesc):
@@ -100,8 +100,9 @@ def get_config(ds_train, ds_test):
             StatMonitorParamSetter('learning_rate', 'error',
                                    lambda x: x * 0.2, 0, 5),
             HumanHyperParamSetter('learning_rate'),
-            PeriodicCallback(
-                InferenceRunner(ds_test, [ScalarStats('error')]), 2),
+            PeriodicTrigger(
+                InferenceRunner(ds_test, [ScalarStats('error')]),
+                every_k_epochs=2),
         ],
         model=Model(),
         steps_per_epoch=steps_per_epoch,

examples/DeepQNetwork/common.py

@@ -97,7 +97,6 @@ def eval_model_multithread(cfg, nr_eval):
 
 
 class Evaluator(Callback):
-
     def __init__(self, nr_eval, input_names, output_names):
         self.eval_episode = nr_eval
         self.input_names = input_names

tensorpack/dataflow/imgaug/noname.py

@@ -102,23 +102,29 @@ class RandomResize(ImageAugmentor):
             yrange (tuple): (min, max) range of scaling ratio for h
             minimum (tuple): (xmin, ymin). avoid scaling down too much.
             aspect_ratio_thres (float): discard samples which change aspect ratio
-                larger than this threshold.
+                larger than this threshold. Set to 0 to keep aspect ratio.
             interp: cv2 interpolation method
         """
         super(RandomResize, self).__init__()
+        assert aspect_ratio_thres >= 0
+        if aspect_ratio_thres == 0:
+            assert xrange == yrange
         self._init(locals())
 
     def _get_augment_params(self, img):
         cnt = 0
         while True:
             sx = self._rand_range(*self.xrange)
-            sy = self._rand_range(*self.yrange)
+            if self.aspect_ratio_thres == 0:
+                sy = sx
+            else:
+                sy = self._rand_range(*self.yrange)
             destX = int(max(sx * img.shape[1], self.minimum[0]))
             destY = int(max(sy * img.shape[0], self.minimum[1]))
             oldr = img.shape[1] * 1.0 / img.shape[0]
             newr = destX * 1.0 / destY
             diff = abs(newr - oldr) / oldr
-            if diff <= self.aspect_ratio_thres:
+            if diff <= self.aspect_ratio_thres + 1e-7:
                 return (destX, destY)
             cnt += 1
             if cnt > 50:

tensorpack/models/batch_norm.py

@@ -97,7 +97,7 @@ def BatchNormV1(x, use_local_stat=None, decay=0.9, epsilon=1e-5):
 
 
 @layer_register(log_shape=False)
-def BatchNormV2(x, use_local_stat=None, decay=0.9, epsilon=1e-5):
+def BatchNorm(x, use_local_stat=None, decay=0.9, epsilon=1e-5):
     """
     Batch normalization layer, as described in the paper:
     `Batch Normalization: Accelerating Deep Network Training by
@@ -188,6 +188,3 @@ def BatchNormV2(x, use_local_stat=None, decay=0.9, epsilon=1e-5):
             return tf.identity(xn, name='output')
     else:
         return tf.identity(xn, name='output')
-
-
-BatchNorm = BatchNormV2

tensorpack/models/linearwrap.py

@@ -57,7 +57,9 @@ class LinearWrap(object):
             return f
         else:
             if layer_name != 'tf':
-                logger.warn("You're calling LinearWrap.__getattr__ with something neither a layer nor 'tf'!")
+                logger.warn(
+                    "You're calling LinearWrap.__getattr__ with {}:"
+                    " neither a layer nor 'tf'!".format(layer_name))
             assert isinstance(layer, ModuleType)
             return LinearWrap._TFModuleFunc(layer, self._t)
 

tensorpack/train/input_data.py

@@ -188,6 +188,9 @@ class TensorInput(FeedfreeInput):
             size(int): size of this input. Use None to leave it undefined.
         """
         self.get_tensor_fn = get_tensor_fn
+        if size is not None:
+            size = int(size)
+            assert size > 0
         self._size = size
 
     def size(self):