update examples

README.md

@@ -7,29 +7,6 @@ Tensorpack is a training interface based on TensorFlow.
 [![Gitter chat](https://badges.gitter.im/gitterHQ/gitter.png)](https://gitter.im/tensorpack/users)
 [![model-zoo](https://img.shields.io/badge/model-zoo-brightgreen.svg)](http://models.tensorpack.com)
 
-See some [examples](examples) to learn about the framework. Everything runs on multiple GPUs, because why not?
-
-### Vision:
-+ [Train ResNet/SE-ResNet on ImageNet](examples/ResNet)
-+ [Train Faster-RCNN / Mask-RCNN on COCO object detection](examples/FasterRCNN)
-+ [Generative Adversarial Network(GAN) variants](examples/GAN), including DCGAN, InfoGAN, Conditional GAN, WGAN, BEGAN, DiscoGAN, Image to Image, CycleGAN.
-+ [DoReFa-Net: train binary / low-bitwidth CNN on ImageNet](examples/DoReFa-Net)
-+ [Fully-convolutional Network for Holistically-Nested Edge Detection(HED)](examples/HED)
-+ [Spatial Transformer Networks on MNIST addition](examples/SpatialTransformer)
-+ [Visualize CNN saliency maps](examples/Saliency)
-+ [Similarity learning on MNIST](examples/SimilarityLearning)
-
-### Reinforcement Learning:
-+ [Deep Q-Network(DQN) variants on Atari games](examples/DeepQNetwork), including DQN, DoubleDQN, DuelingDQN.
-+ [Asynchronous Advantage Actor-Critic(A3C) with demos on OpenAI Gym](examples/A3C-Gym)
-
-### Speech / NLP:
-+ [LSTM-CTC for speech recognition](examples/CTC-TIMIT)
-+ [char-rnn for fun](examples/Char-RNN)
-+ [LSTM language model on PennTreebank](examples/PennTreebank)
-
-Examples are not only for demonstration of the framework -- you can train them and reproduce the results in papers.
-
 ## Features:
 
 It's Yet Another TF wrapper, but different in:
@@ -39,7 +16,6 @@ It's Yet Another TF wrapper, but different in:
 	  On various CNNs, it runs 1.5~1.7x faster than the equivalent Keras code.
 
 	+ Data-parallel multi-GPU training is off-the-shelf to use. It is as fast as Google's [official benchmark](https://www.tensorflow.org/performance/benchmarks).
-		You cannot beat its speed unless you're a TensorFlow expert.
 
 	+ See [tensorpack/benchmarks](https://github.com/tensorpack/benchmarks) for some benchmark scripts.
 
@@ -54,6 +30,32 @@ It's Yet Another TF wrapper, but different in:
 
 See [tutorials](http://tensorpack.readthedocs.io/en/latest/tutorial/index.html) to know more about these features.
 
+## [Examples](examples):
+
+Instead of showing you 10 different ways to train MNIST to random accuracy,
+[tensorpack examples](examples) faithfully replicate papers and care about performance.
+And everything runs on multiple GPUs. Some highlights:
+
+### Vision:
++ [Train ResNet on ImageNet](examples/ResNet)
++ [Train Faster-RCNN / Mask-RCNN on COCO object detection](examples/FasterRCNN)
++ [Generative Adversarial Network(GAN) variants](examples/GAN), including DCGAN, InfoGAN, Conditional GAN, WGAN, BEGAN, DiscoGAN, Image to Image, CycleGAN.
++ [DoReFa-Net: train binary / low-bitwidth CNN on ImageNet](examples/DoReFa-Net)
++ [Fully-convolutional Network for Holistically-Nested Edge Detection(HED)](examples/HED)
++ [Spatial Transformer Networks on MNIST addition](examples/SpatialTransformer)
++ [Visualize CNN saliency maps](examples/Saliency)
++ [Similarity learning on MNIST](examples/SimilarityLearning)
+
+### Reinforcement Learning:
++ [Deep Q-Network(DQN) variants on Atari games](examples/DeepQNetwork), including DQN, DoubleDQN, DuelingDQN.
++ [Asynchronous Advantage Actor-Critic(A3C) with demos on OpenAI Gym](examples/A3C-Gym)
+
+### Speech / NLP:
++ [LSTM-CTC for speech recognition](examples/CTC-TIMIT)
++ [char-rnn for fun](examples/Char-RNN)
++ [LSTM language model on PennTreebank](examples/PennTreebank)
+
+
 ## Install:
 
 Dependencies:

examples/README.md

@@ -3,7 +3,7 @@
 
 Training examples with __reproducible performance__.
 
-__"Reproduce" should always means reproduce performance__.
+__The word "reproduce" should always means reproduce performance__.
 Reproducing a method is usually easy, but you don't know whether you've made mistakes, because wrong code will often appear to work.
 Reproducing __performance__ results is what really matters, and is something that's hardly seen on github.
 See [Unawareness of Deep Learning Mistakes](https://medium.com/@ppwwyyxx/unawareness-of-deep-learning-mistakes-d5b5774da0ba).
@@ -11,11 +11,11 @@ See [Unawareness of Deep Learning Mistakes](https://medium.com/@ppwwyyxx/unaware
 
 ## Getting Started:
 These examples don't have meaningful performance numbers. They are supposed to be just demos.
-+ [An illustrative mnist example with explanation of the framework](mnist-convnet.py)
-+ The same mnist example using [tf-slim](mnist-tfslim.py), and [with weights visualizations](mnist-visualizations.py)
-+ A tiny [Cifar ConvNet](cifar-convnet.py) and [SVHN ConvNet](svhn-digit-convnet.py)
++ [An illustrative MNIST example with explanation of the framework](Basics/mnist-convnet.py)
++ The same MNIST example written with [tf.layers](Basics/mnist-tflayers.py), [tf-slim](Basics/mnist-tfslim.py), and [with weights visualizations](Basics/mnist-visualizations.py)
++ A tiny [Cifar ConvNet](Basics/cifar-convnet.py) and [SVHN ConvNet](Basics/svhn-digit-convnet.py)
 + [A boilerplate file to start with, for your own tasks](boilerplate.py)
-+ If you've used Keras, check out [Keras examples](keras).
++ If you've used Keras, check out [Keras examples](keras)
 
 ## Vision:
 | Name | Performance |

examples/basics/mnist-tflayers.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# File: mnist-tflayers.py
+
+import os
+import argparse
+import tensorflow as tf
+"""
+MNIST ConvNet example using tf.layers
+Mostly the same as 'mnist-convnet.py',
+the only differences are:
+    1. use tf.layers
+    2. use tf.layers variable names to summarize weights
+"""
+
+
+# Just import everything into current namespace
+from tensorpack import *
+from tensorpack.tfutils import summary, get_current_tower_context
+from tensorpack.dataflow import dataset
+
+IMAGE_SIZE = 28
+
+
+class Model(ModelDesc):
+    def _get_inputs(self):
+        """
+        Define all the inputs (with type, shape, name) that
+        the graph will need.
+        """
+        return [InputDesc(tf.float32, (None, IMAGE_SIZE, IMAGE_SIZE), 'input'),
+                InputDesc(tf.int32, (None,), 'label')]
+
+    def _build_graph(self, inputs):
+        """This function should build the model which takes the input variables
+        and define self.cost at the end"""
+
+        # inputs contains a list of input variables defined above
+        image, label = inputs
+
+        # In tensorflow, inputs to convolution function are assumed to be
+        # NHWC. Add a single channel here.
+        image = tf.expand_dims(image, 3)
+
+        image = image * 2 - 1   # center the pixels values at zero
+
+        # The context manager `argscope` sets the default option for all the layers under
+        # this context. Here we use 32 channel convolution with shape 3x3
+        with argscope(Conv2D, kernel_shape=3, nl=tf.nn.relu, out_channel=32):
+            l = tf.layers.conv2d(image, 32, 3, padding='same', activation=tf.nn.relu, name='conv0')
+            l = tf.layers.max_pooling2d(l, 2, 2, padding='valid')
+            l = tf.layers.conv2d(l, 32, 3, padding='same', activation=tf.nn.relu, name='conv1')
+            l = tf.layers.conv2d(l, 32, 3, padding='same', activation=tf.nn.relu, name='conv2')
+            l = tf.layers.max_pooling2d(l, 2, 2, padding='valid')
+            l = tf.layers.conv2d(l, 32, 3, padding='same', activation=tf.nn.relu, name='conv3')
+            l = tf.layers.flatten(l)
+            l = tf.layers.dense(l, 512, activation=tf.nn.relu, name='fc0')
+            l = tf.layers.dropout(l, rate=0.5,
+                                  training=get_current_tower_context().is_training)
+            logits = tf.layers.dense(l, 10, activation=tf.identity, name='fc1')
+
+        tf.nn.softmax(logits, name='prob')   # a Bx10 with probabilities
+
+        # a vector of length B with loss of each sample
+        cost = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=label)
+        cost = tf.reduce_mean(cost, name='cross_entropy_loss')  # the average cross-entropy loss
+
+        correct = tf.cast(tf.nn.in_top_k(logits, label, 1), tf.float32, name='correct')
+        accuracy = tf.reduce_mean(correct, name='accuracy')
+
+        # This will monitor training error (in a moving_average fashion):
+        # 1. write the value to tensosrboard
+        # 2. write the value to stat.json
+        # 3. print the value after each epoch
+        train_error = tf.reduce_mean(1 - correct, name='train_error')
+        summary.add_moving_summary(train_error, accuracy)
+
+        # Use a regex to find parameters to apply weight decay.
+        # Here we apply a weight decay on all W (weight matrix) of all fc layers
+        wd_cost = tf.multiply(1e-5,
+                              regularize_cost('fc.*/kernel', tf.nn.l2_loss),
+                              name='regularize_loss')
+        self.cost = tf.add_n([wd_cost, cost], name='total_cost')
+        summary.add_moving_summary(cost, wd_cost, self.cost)
+
+        # monitor histogram of all weight (of conv and fc layers) in tensorboard
+        summary.add_param_summary(('.*/kernel', ['histogram', 'rms']))
+
+    def _get_optimizer(self):
+        lr = tf.train.exponential_decay(
+            learning_rate=1e-3,
+            global_step=get_global_step_var(),
+            decay_steps=468 * 10,
+            decay_rate=0.3, staircase=True, name='learning_rate')
+        # This will also put the summary in tensorboard, stat.json and print in terminal
+        # but this time without moving average
+        tf.summary.scalar('lr', lr)
+        return tf.train.AdamOptimizer(lr)
+
+
+def get_data():
+    train = BatchData(dataset.Mnist('train'), 128)
+    test = BatchData(dataset.Mnist('test'), 256, remainder=True)
+    return train, test
+
+
+def get_config():
+    dataset_train, dataset_test = get_data()
+    # How many iterations you want in each epoch.
+    # This is the default value, don't actually need to set it in the config
+    steps_per_epoch = dataset_train.size()
+
+    # get the config which contains everything necessary in a training
+    return TrainConfig(
+        model=Model(),
+        dataflow=dataset_train,  # the DataFlow instance for training
+        callbacks=[
+            ModelSaver(),   # save the model after every epoch
+            MaxSaver('validation_accuracy'),  # save the model with highest accuracy (prefix 'validation_')
+            InferenceRunner(    # run inference(for validation) after every epoch
+                dataset_test,   # the DataFlow instance used for validation
+                ScalarStats(['cross_entropy_loss', 'accuracy'])),
+        ],
+        steps_per_epoch=steps_per_epoch,
+        max_epoch=100,
+    )
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--gpu', help='comma separated list of GPU(s) to use.')
+    parser.add_argument('--load', help='load model')
+    args = parser.parse_args()
+    if args.gpu:
+        os.environ['CUDA_VISIBLE_DEVICES'] = args.gpu
+
+    # automatically setup the directory train_log/mnist-convnet for logging
+    logger.auto_set_dir()
+
+    config = get_config()
+    if args.load:
+        config.session_init = SaverRestore(args.load)
+    # SimpleTrainer is slow, this is just a demo.
+    # You can use QueueInputTrainer instead
+    launch_train_with_config(config, SimpleTrainer())