update docs; add BN WD in resnet

README.md

@@ -34,9 +34,9 @@ See [tutorials and documentations](http://tensorpack.readthedocs.io/tutorial/ind
 
 ## Examples:
 
-We refuse toy examples.
-Instead of showing you 10 arbitrary networks trained on toy datasets,
-[Tensorpack examples](examples) faithfully replicate papers and care about reproducing numbers,
+We refuse toy examples. We refuse low-quality implementations.
+Unlike most open source repos which only __implement__ papers,
+[Tensorpack examples](examples) faithfully __reproduce__ papers,
 demonstrating its __flexibility__ for actual research.
 
 ### Vision:

examples/DeepQNetwork/README.md

@@ -20,9 +20,9 @@ Claimed performance in the paper can be reproduced, on several games I've tested
 
 ![DQN](curve-breakout.png)
 
-On one (Maxwell) TitanX, Double-DQN took ~18 hours of training to reach a score of 400 on breakout.
-
-Double-DQN with nature paper setting runs at 60 batches (3840 trained frames, 240 seen frames, 960 game frames) per second on TitanX.
+On one GTX 1080Ti, the ALE version took ~3 hours of training to reach 21 (maximum) score on
+Pong, ~15 hours of training to reach 400 score on Breakout.
+It runs at 50 batches (~3.2k trained frames, 200 seen frames, 800 game frames) per second on GTX 1080Ti.
 
 ## How to use
 

examples/FasterRCNN/train.py

@@ -423,7 +423,8 @@ if __name__ == '__main__':
     if args.visualize or args.evaluate or args.predict:
         if not tf.test.is_gpu_available():
             from tensorflow.python.framework import test_util
-            assert test_util.IsMklEnabled(), "Inference requires either GPU support or MKL support!"
+            assert get_tf_version_tuple() >= (1, 7) and test_util.IsMklEnabled(), \
+                "Inference requires either GPU support or MKL support!"
         assert args.load
         finalize_configs(is_training=False)
 

examples/README.md

@@ -9,14 +9,15 @@ Github is full of deep learning code that "implements" but does not "reproduce"
 methods, and you'll not know whether the implementation is actually correct.
 See [Unawareness of Deep Learning Mistakes](https://medium.com/@ppwwyyxx/unawareness-of-deep-learning-mistakes-d5b5774da0ba).
 
-We refuse toy examples.
-Instead of showing you 10 arbitrary networks trained on toy datasets with random final performance,
-tensorpack examples try to faithfully replicate experiments and performance in the paper,
+We refuse toy examples. We refuse low-quality implementations.
+Unlike most open source repos which only __implement__ methods,
+[Tensorpack examples](examples) faithfully __reproduce__ 
+experiments and performance in the paper,
 so you're confident that they are correct.
 
 
 ## Getting Started:
-These are all the toy examples in tensorpack. They are supposed to be just demos.
+These are the only toy examples in tensorpack. They are supposed to be just demos.
 + [An illustrative MNIST example with explanation of the framework](basics/mnist-convnet.py)
 + Tensorpack supports any symbolic libraries. See the same MNIST example written with [tf.layers](basics/mnist-tflayers.py), [tf-slim](basics/mnist-tfslim.py), and [with weights visualizations](basics/mnist-visualizations.py)
 + A tiny [Cifar ConvNet](basics/cifar-convnet.py) and [SVHN ConvNet](basics/svhn-digit-convnet.py)

examples/ResNet/imagenet-resnet.py

@@ -103,16 +103,25 @@ def get_config(model):
 
 if __name__ == '__main__':
     parser = argparse.ArgumentParser()
+    # generic:
     parser.add_argument('--gpu', help='comma separated list of GPU(s) to use. Default to use all available ones')
-    parser.add_argument('--data', help='ILSVRC dataset dir')
+    parser.add_argument('--eval', action='store_true', help='run offline evaluation instead of training')
     parser.add_argument('--load', help='load a model for training or evaluation')
+
+    # data:
+    parser.add_argument('--data', help='ILSVRC dataset dir')
     parser.add_argument('--fake', help='use FakeData to debug or benchmark this model', action='store_true')
     parser.add_argument('--symbolic', help='use symbolic data loader', action='store_true')
-    parser.add_argument('--data-format', help='image data format',
+
+    # model:
+    parser.add_argument('--data-format', help='the image data layout used by the model',
                         default='NCHW', choices=['NCHW', 'NHWC'])
     parser.add_argument('-d', '--depth', help='ResNet depth',
                         type=int, default=50, choices=[18, 34, 50, 101, 152])
-    parser.add_argument('--eval', action='store_true', help='run offline evaluation instead of training')
+    parser.add_argument('--weight-decay-norm', action='store_true',
+                        help="apply weight decay on normalization layers (gamma & beta)."
+                             "This is used in torch/pytorch, and slightly "
+                             "improves validation accuracy of large models.")
     parser.add_argument('--batch', default=256, type=int,
                         help="total batch size. "
                         "Note that it's best to keep per-GPU batch size in [32, 64] to obtain the best accuracy."
@@ -126,6 +135,9 @@ if __name__ == '__main__':
 
     model = Model(args.depth, args.mode)
     model.data_format = args.data_format
+    if model.weight_decay_norm:
+        model.weight_decay_pattern = ".*/W|.*/gamma|.*/beta"
+
     if args.eval:
         batch = 128    # something that can run on one gpu
         ds = get_imagenet_dataflow(args.data, 'val', batch)

tensorpack/models/conv2d.py

@@ -198,7 +198,7 @@ def Conv2DTranspose(
     else:
         # Our own implementation, to avoid Keras bugs. https://github.com/tensorflow/tensorflow/issues/25946
         assert kernel_regularizer is None and bias_regularizer is None and activity_regularizer is None, \
-            "Unsupported arguments due to bug in TensorFlow 1.13"
+            "Unsupported arguments due to Keras bug in TensorFlow 1.13"
         data_format = get_data_format(data_format, keras_mode=False)
         shape_dyn = tf.shape(inputs)
         strides2d = shape2d(strides)

tensorpack/tfutils/varmanip.py

@@ -226,7 +226,7 @@ def is_training_name(name):
         return True
     if name.endswith('/Adagrad'):
         return True
-    if name.startswith('EMA/'):  # all the moving average summaries
+    if name.startswith('EMA/') or '/EMA/' in name:  # all the moving average summaries
         return True
     if name.startswith('AccumGrad') or name.endswith('/AccumGrad'):
         return True

tensorpack/train/interface.py

@@ -56,7 +56,7 @@ def launch_train_with_config(config, trainer):
     2. Call `trainer.setup_graph` with the input as well as `config.model`.
     3. Call `trainer.train` with rest of the attributes of config.
 
-    See tutorial at
+    See the `related tutorial
     <https://tensorpack.readthedocs.io/tutorial/training-interface.html#with-modeldesc-and-trainconfig>`_
     to learn more.