Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
S
seminar-breakout
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Shashank Suhas
seminar-breakout
Commits
d4f925a9
Commit
d4f925a9
authored
Apr 26, 2016
by
Yuxin Wu
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
fix recommonmark version
parent
3975ae05
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
6 additions
and
5 deletions
+6
-5
docs/requirements.txt
docs/requirements.txt
+1
-1
docs/user/dataflow.md
docs/user/dataflow.md
+5
-4
No files found.
docs/requirements.txt
View file @
d4f925a9
...
...
@@ -2,4 +2,4 @@ termcolor
tqdm
nltk
decorator
recommonmark
recommonmark
==0.4.0
docs/user/dataflow.md
View file @
d4f925a9
## Dataflow
Dataflow uses Python generator to produce data
in an efficient way
.
Dataflow uses Python generator to produce data.
A Dataflow has to implement the
`get_data()`
generator method, which generates
`datapoints`
when called.
A datapoint must be a list of picklable Python object.
A Dataflow has to implement the
`get_data()`
generator method, which generates a
`datapoint`
when called.
A datapoint must be a list of picklable Python object, each is called a
`component`
of the datapoint.
For example to train on MNIST dataset, you can define a Dataflow that produces datapoints of shape
`[(BATCH, 28, 28), (BATCH,)]`
.
Then, multiple Dataflows can be composed together to build a complex data-preprocessing pipeline,
including __reading from disk, batching, augmentations, prefetching__, etc. These components written in Python
can provide a m
uch m
ore flexible data pipeline than with TensorFlow operators.
can provide a more flexible data pipeline than with TensorFlow operators.
Take a look at
[
common Dataflow
](
../../tensorpack/dataflow/common.py
)
and a
[
example of use
](
../../examples/ResNet/cifar10-resnet.py#L125
)
.
Optionally, Dataflow can implement the following two methods:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment