update docs

.travis.yml

@@ -48,11 +48,17 @@ script:
 - cd $TRAVIS_BUILD_DIR && python tests/test_examples.py
 
 notifications:
-  email:
+- email:
     recipients:
     - ppwwyyxxc@gmail.com
     on_success: never
     on_failure: change
+- webhooks:
+    urls:
+      - https://webhooks.gitter.im/e/cede9dbbf6630b3704b3
+    on_success: change  # options: [always|never|change] default: always
+    on_failure: always  # options: [always|never|change] default: always
+    on_start: never     # options: [always|never|change] default: always
 
 deploy:
 - provider: pypi

docs/user/callbacks.md

@@ -24,7 +24,7 @@ TrainConfig(
   callbacks=[
     # save the model every epoch
     ModelSaver(),
-    # run inference on another Dataflow every epoch, compute top1/top5 classification error and save them
+    # run inference on another Dataflow every epoch, compute top1/top5 classification error and save them in log
     InferenceRunner(dataset_val, [
         ClassificationError('wrong-top1', 'val-error-top1'),
         ClassificationError('wrong-top5', 'val-error-top5')]),
@@ -39,12 +39,12 @@ TrainConfig(
                -d body={val-error-top1} > /dev/null 2>&1',
                'val-error-top1')
   ],
-  extra_callbacks=[    # these are already enabled by default
+  extra_callbacks=[    # these callbacks are already enabled by default
     # maintain and summarize moving average of some tensors (e.g. training loss, training error)
     MovingAverageSummary(),
     # draw a nice progress bar
     ProgressBar(),
-    # print all the statistics I've created and scalar tensors I've summarized
+    # print all the statistics I've created, and scalar tensors I've summarized
     StatPrinter(),
   ]
 )

docs/user/dataflow.md

@@ -7,8 +7,9 @@ A Dataflow has a `get_data()` generator method,
 which yields a `datapoint` when called.
 A datapoint must be a **list** of Python objects which I called the `components` of this datapoint.
 
-For example, to train on MNIST dataset, you can define a Dataflow
-that produces datapoints of two elements: a numpy array of shape (64, 28, 28), and an array of shape (64,).
+For example, to train on MNIST dataset, you can build a Dataflow
+that produces datapoints of two elements (components):
+a numpy array of shape (64, 28, 28), and an array of shape (64,).
 
 ### Composition of DataFlow
 One good thing about having a standard interface is to be able to provide
@@ -25,10 +26,10 @@ df = CaffeLMDB('/path/to/caffe/lmdb', shuffle=False)
 df = AugmentImageComponent(df, [imgaug.Resize((225, 225))])
 # group data into batches of size 128
 df = BatchData(df, 128)
-# start 3 processes to run the dataflow in parallel, and transfer the data with ZeroMQ
+# start 3 processes to run the dataflow in parallel, and transfer data with ZeroMQ
 df = PrefetchDataZMQ(df, 3)
 ````
-Another complicated example is the [ResNet training script](../examples/ResNet/imagenet-resnet.py)
+A more complicated example is the [ResNet training script](../examples/ResNet/imagenet-resnet.py)
 with all the data preprocessing.
 
 All these modules are written in Python,
@@ -60,7 +61,9 @@ A Dataflow has a `get_data()` method which yields a datapoint every time.
 class MyDataFlow(DataFlow):
   def get_data(self):
     for k in range(100):
-		  yield datapoint
+			digit = np.random.rand(28, 28)
+			label = np.random.randint(10)
+      yield [digit, label]
 ```
 
 Optionally, Dataflow can implement the following two methods:

docs/user/glance.md

@@ -3,6 +3,3 @@
 
 The following guide introduces some core concepts of TensorPack. In contrast to several other libraries TensorPack contains of several modules to build complex deep learning algorithms and train models with high accuracy and high speed.
 
-### Callbacks
-
-The use of callbacks add the flexibility to execute code during training. These callbacks are triggered on several events such as after each step or at the end of one training epoch.

docs/user/trainer.md

 
 # Trainers
 
-## Trainer
-
 Training is basically **running something again and again**.
 Tensorpack base trainer implements the logic of *running the iteration*,
 and other trainers implement *what the iteration is*.
@@ -54,7 +52,7 @@ The existing trainers should be enough for single-cost optimization tasks. If yo
 want to do something inside the trainer, consider writing it as a callback, or
 write an issue to see if there is a better solution than creating new trainers.
 
-For other tasks, you might need a new trainer.
+For certain tasks, you might need a new trainer.
 The [GAN trainer](../examples/GAN/GAN.py) is one example of how to implement
 new trainers.
 

examples/ResNet/imagenet-resnet.py

@@ -175,7 +175,7 @@ def get_data(train_or_test):
     ds = AugmentImageComponent(ds, augmentors)
     ds = BatchData(ds, BATCH_SIZE, remainder=not isTrain)
     if isTrain:
-        ds = PrefetchDataZMQ(ds, min(30, multiprocessing.cpu_count()))
+        ds = PrefetchDataZMQ(ds, min(20, multiprocessing.cpu_count()))
     return ds
 
 

tensorpack/dataflow/common.py

@@ -41,7 +41,7 @@ class TestDataSpeed(ProxyDataFlow):
         with get_tqdm(total=self.test_size, leave=True) as pbar:
             for idx, dp in enumerate(self.ds.get_data()):
                 pbar.update()
-                if idx == self.test_size:
+                if idx == self.test_size - 1:
                     break
 
 
@@ -439,7 +439,8 @@ class LocallyShuffleData(ProxyDataFlow, RNGDataFlow):
         Args:
             ds (DataFlow): input DataFlow.
             cache_size (int): size of the cache.
-            nr_reuse (int): reuse each datapoints several times to improve speed.
+            nr_reuse (int): reuse each datapoints several times to improve
+                speed, but may hurt your model.
         """
         ProxyDataFlow.__init__(self, ds)
         self.q = deque(maxlen=cache_size)