
# DataFlow

### What is DataFlow

DataFlow is a pure-Python library to create iterators for efficient data loading.

**Definition**: A DataFlow is a idiomatic Python iterator object that has a `__iter__()` method
which yields `datapoints`, and optionally a `__len__()` method returning the size of the DataFlow.
A datapoint is a **list or dict** of Python objects, each of which are called the `components` of a datapoint.

**Example**: to train on MNIST dataset, you may need a DataFlow with a `__iter__()` method
that yields datapoints (lists) of two components:
a numpy array of shape (64, 28, 28), and an array of shape (64,).

As you saw,
DataFlow is __independent of TensorFlow__ since it produces any python objects
(usually numpy arrays).
To `import tensorpack.dataflow`, you don't even have to install TensorFlow.
You can simply use DataFlow as a data processing pipeline and plug it into any other frameworks.


### Composition of DataFlow
There are a lot of existing DataFlow utilities in tensorpack, which you can use to compose
one DataFlow with complex data pipeline. A common pipeline usually
would __read from disk (or other sources), apply transformations (possibly in parallel), group into batches,
prefetch data__, etc, and all __run in parallel__. A simple example is as the following:

````python
# a DataFlow you implement to produce [tensor1, tensor2, ..] lists from whatever sources:
df = MyDataFlow(dir='/my/data', shuffle=True)
# resize the image component of each datapoint
df = AugmentImageComponent(df, [imgaug.Resize((225, 225))])
# group data into batches of size 128
df = BatchData(df, 128)
# start 3 processes to run the dataflow in parallel
df = MultiProcessRunnerZMQ(df, 3)
````
You can find more complicated DataFlow in the [ImageNet training script](../examples/ImageNetModels/imagenet_utils.py)
with all the data preprocessing.

### Work with Your Data
We do not make any assumptions about your data format.
You would usually want to write the source DataFlow (`MyDataFlow` in the above example) for your own data format.
See [another tutorial](extend/dataflow.html) for simple instructions on writing a DataFlow.
Once you have the source reader, all the [built-in
DataFlows](../modules/dataflow.html) are ready for you to assemble the rest of the data pipeline.

### Why DataFlow

1. It's easy: write everything in pure Python, and reuse existing utilities.
	 On the contrary, writing data loaders in TF operators is usually painful, and performance is hard to tune.
	 See more discussions in [Python Reader or TF Reader](extend/input-source.html#python-reader-or-tf-reader).
2. It's fast: see [Efficient DataFlow](efficient-dataflow.html)
	on how to build a fast DataFlow with parallelism.
	If you're using DataFlow with tensorpack, also see [Input Pipeline tutorial](extend/input-source.html)
	on how tensorpack further accelerates data loading in the graph.

Nevertheless, tensorpack supports data loading with native TF operators / TF datasets as well.

### Use DataFlow in Your Own Code

When training with tensorpack, typically it is the `InputSource` interface that runs the DataFlow.
However, DataFlow can be used without other tensorpack components.
To run a DataFlow by yourself, call `reset_state()` first to initialize it,
and then use the generator however you like:
```python
df = SomeDataFlow()

df.reset_state()
for dp in df:
    # dp is now a list. do whatever
```

Read the [API documentation](../../modules/dataflow.html#tensorpack.dataflow.DataFlw)
to see API details of DataFlow.
