Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
S
seminar-breakout
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Shashank Suhas
seminar-breakout
Commits
2b076314
Commit
2b076314
authored
Feb 20, 2017
by
Yuxin Wu
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
just docs
parent
a2934281
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
8 additions
and
8 deletions
+8
-8
docs/tutorial/efficient-dataflow.md
docs/tutorial/efficient-dataflow.md
+7
-7
docs/tutorial/tf-queue.md
docs/tutorial/tf-queue.md
+1
-1
No files found.
docs/tutorial/efficient-dataflow.md
View file @
2b076314
...
@@ -3,21 +3,21 @@
...
@@ -3,21 +3,21 @@
This tutorial gives an overview of how to build an efficient DataFlow, using ImageNet
This tutorial gives an overview of how to build an efficient DataFlow, using ImageNet
dataset as an example.
dataset as an example.
Our goal in the end is to have
Our goal in the end is to have
a
generator which yields ImageNet datapoints (after proper preprocessing)
as fast as possible.
a
__generator__ which yields preprocessed ImageNet images and labels
as fast as possible.
Since it is simply a generator interface, you can use the DataFlow in other frameworks (e.g. Keras)
Since it is simply a generator interface, you can use the DataFlow in other frameworks (e.g. Keras)
or your own code as well.
or your own code as well.
We use ILSVRC12 training set, which contains 1.28 million images.
We use ILSVRC12 training set, which contains 1.28 million images.
Following the
[
ResNet example
](
../examples/ResNet
)
, our pre-processing need images in their original resolution,
so we'll read the original dataset instead of a down-sampled version here.
The average resolution is about 400x350
<sup>
[[1]]
</sup>
.
The original images (JPEG compressed) are 140G in total.
The original images (JPEG compressed) are 140G in total.
The average resolution is about 400x350
<sup>
[[1]]
</sup>
.
Following the
[
ResNet example
](
../examples/ResNet
)
, we need images in their original resolution,
so we'll read the original dataset instead of a down-sampled version.
We'll need to reach a speed of, roughly 1k images per second, to keep GPUs busy.
Note that the actual performance would depend on not only the disk, but also
Note that the actual performance would depend on not only the disk, but also
memory (for caching) and CPU (for data processing).
memory (for caching) and CPU (for data processing).
You'll need to tune the parameters (#processes, #threads, size of buffer, etc.)
You'll definitely need to tune the parameters (#processes, #threads, size of buffer, etc.)
or change the pipeline for new tasks and new machines
or change the pipeline for new tasks and new machines to achieve best performance.
to achieve better performance.
## Random Read
## Random Read
...
...
docs/tutorial/tf-queue.md
View file @
2b076314
...
@@ -15,7 +15,7 @@ while True:
...
@@ -15,7 +15,7 @@ while True:
minimize_op
.
run
(
feed_dict
=
{
'X'
:
X
,
'y'
:
y
})
minimize_op
.
run
(
feed_dict
=
{
'X'
:
X
,
'y'
:
y
})
```
```
However, when you need to load data from Python-side, this is the only available interface in frameworks such as Keras, tflearn.
However, when you need to load data from Python-side, this is the only available interface in frameworks such as Keras, tflearn.
This is part of the reason why
[
tensorpack is fast
](
https://gist.github.com/ppwwyyxx/8d95da79f8d97036a7d67c2416c851b6
)
.
This is part of the reason why
[
tensorpack is fast
er
](
https://gist.github.com/ppwwyyxx/8d95da79f8d97036a7d67c2416c851b6
)
.
You should use something like this instead:
You should use something like this instead:
```
python
```
python
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment