update example

README.md

@@ -43,9 +43,10 @@ Unlike most open source repos which only __implement__ papers,
 demonstrating its __flexibility__ for actual research.
 
 ### Vision:
-+ [Train ResNet](examples/ResNet) and [other models](examples/ImageNetModels) on ImageNet.
++ [Train ResNet](examples/ResNet) and [other models](examples/ImageNetModels) on ImageNet
 + [Train Mask/Faster R-CNN on COCO object detection](examples/FasterRCNN)
-+ [Generative Adversarial Network(GAN) variants](examples/GAN), including DCGAN, InfoGAN, Conditional GAN, WGAN, BEGAN, DiscoGAN, Image to Image, CycleGAN.
++ [Unsupervised learning with Momentum Contrast](https://github.com/ppwwyyxx/moco.tensorflow) (MoCo)
++ [Generative Adversarial Network(GAN) variants](examples/GAN), including DCGAN, InfoGAN, Conditional GAN, WGAN, BEGAN, DiscoGAN, Image to Image, CycleGAN
 + [DoReFa-Net: train binary / low-bitwidth CNN on ImageNet](examples/DoReFa-Net)
 + [Fully-convolutional Network for Holistically-Nested Edge Detection(HED)](examples/HED)
 + [Spatial Transformer Networks on MNIST addition](examples/SpatialTransformer)

examples/FasterRCNN/config.py

@@ -142,7 +142,7 @@ _C.TRAIN.NUM_GPUS = None         # by default, will be set from code
 _C.TRAIN.WEIGHT_DECAY = 1e-4
 _C.TRAIN.BASE_LR = 1e-2  # defined for total batch size=8. Otherwise it will be adjusted automatically
 _C.TRAIN.WARMUP = 1000   # in terms of iterations. This is not affected by #GPUs
-_C.TRAIN.WARMUP_INIT_LR = 1e-2 * 0.33  # defined for total batch size=8. Otherwise it will be adjusted automatically
+_C.TRAIN.WARMUP_INIT_LR = 0.  # defined for total batch size=8. Otherwise it will be adjusted automatically
 _C.TRAIN.STEPS_PER_EPOCH = 500
 _C.TRAIN.STARTING_EPOCH = 1  # the first epoch to start with, useful to continue a training
 

examples/FasterRCNN/modeling/generalized_rcnn.py

@@ -31,7 +31,7 @@ class GeneralizedRCNN(ModelDesc):
         return tf.transpose(image, [0, 3, 1, 2])
 
     def optimizer(self):
-        lr = tf.get_variable('learning_rate', initializer=0.003, trainable=False)
+        lr = tf.get_variable('learning_rate', initializer=0., trainable=False)
         tf.summary.scalar('learning_rate-summary', lr)
 
         # The learning rate in the config is set for 8 GPUs, and we use trainers with average=False.

examples/README.md

@@ -9,13 +9,13 @@ Github is full of deep learning code that "implements" but does not "reproduce"
 methods, and you'll not know whether the implementation is actually correct.
 See [Unawareness of Deep Learning Mistakes](https://medium.com/@ppwwyyxx/unawareness-of-deep-learning-mistakes-d5b5774da0ba).
 
-We refuse toy examples. 
-Instead of showing tiny CNNs trained on MNIST/Cifar10, 
+We refuse toy examples.
+Instead of showing tiny CNNs trained on MNIST/Cifar10,
 we provide training scripts that reproduce well-known papers.
 
 We refuse low-quality implementations.
 Unlike most open source repos which only __implement__ methods,
-[Tensorpack examples](examples) faithfully __reproduce__ 
+[Tensorpack examples](examples) faithfully __reproduce__
 experiments and performance in the paper,
 so you're confident that they are correct.
 
@@ -33,6 +33,7 @@ These are the only toy examples in tensorpack. They are supposed to be just demo
 | ---                                                                                                                                                   | ---                         |
 | Train [ResNet](ResNet), [ShuffleNet and other models](ImageNetModels) on ImageNet                                                                     | reproduce 10 papers         |
 | [Train Mask/Faster R-CNN on COCO](FasterRCNN)                                                                                                         | reproduce 7 papers          |
+| [Unsupervised learning with Momentum Contrast](https://github.com/ppwwyyxx/moco.tensorflow) (MoCo)                                                    | reproduce 2 papers          |
 | [Generative Adversarial Network(GAN) variants](GAN), including DCGAN, InfoGAN, <br/> Conditional GAN, WGAN, BEGAN, DiscoGAN, Image to Image, CycleGAN | visually reproduce 8 papers |
 | [DoReFa-Net: training binary / low-bitwidth CNN on ImageNet](DoReFa-Net)                                                                              | reproduce 4 papers          |
 | [Adversarial training with state-of-the-art robustness](https://github.com/facebookresearch/ImageNet-Adversarial-Training)                            | official code for the paper |