Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
S
seminar-breakout
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Shashank Suhas
seminar-breakout
Commits
8dbb84e2
Commit
8dbb84e2
authored
Jun 21, 2017
by
Yuxin Wu
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Allow LMDBData to take DataFlow
parent
661abb69
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
21 additions
and
6 deletions
+21
-6
tensorpack/dataflow/format.py
tensorpack/dataflow/format.py
+20
-5
tensorpack/train/input_source.py
tensorpack/train/input_source.py
+1
-1
No files found.
tensorpack/dataflow/format.py
View file @
8dbb84e2
...
@@ -157,9 +157,20 @@ class LMDBDataDecoder(MapData):
...
@@ -157,9 +157,20 @@ class LMDBDataDecoder(MapData):
class
LMDBDataPoint
(
MapData
):
class
LMDBDataPoint
(
MapData
):
""" Read a LMDB file and produce deserialized datapoints.
"""
Read a LMDB file and produce deserialized datapoints.
It reads the database produced by
It reads the database produced by
:func:`tensorpack.dataflow.dftools.dump_dataflow_to_lmdb`.
:func:`tensorpack.dataflow.dftools.dump_dataflow_to_lmdb`.
Example:
.. code-block:: python
ds = LMDBDataPoint("/data/ImageNet.lmdb", shuffle=False)
# alternatively:
ds = LMDBData("/data/ImageNet.lmdb", shuffle=False)
ds = LocallyShuffleData(ds, 50000)
ds = LMDBDataPoint(ds)
"""
"""
def
__init__
(
self
,
*
args
,
**
kwargs
):
def
__init__
(
self
,
*
args
,
**
kwargs
):
...
@@ -167,6 +178,10 @@ class LMDBDataPoint(MapData):
...
@@ -167,6 +178,10 @@ class LMDBDataPoint(MapData):
Args:
Args:
args, kwargs: Same as in :class:`LMDBData`.
args, kwargs: Same as in :class:`LMDBData`.
"""
"""
if
isinstance
(
args
[
0
],
DataFlow
):
ds
=
args
[
0
]
else
:
ds
=
LMDBData
(
*
args
,
**
kwargs
)
ds
=
LMDBData
(
*
args
,
**
kwargs
)
def
f
(
dp
):
def
f
(
dp
):
...
...
tensorpack/train/input_source.py
View file @
8dbb84e2
...
@@ -464,7 +464,7 @@ class StagingInputWrapper(FeedfreeInput):
...
@@ -464,7 +464,7 @@ class StagingInputWrapper(FeedfreeInput):
self
.
get_stage_op
(),
self
.
get_unstage_op
(),
self
.
_nr_stage
))
self
.
get_stage_op
(),
self
.
get_unstage_op
(),
self
.
_nr_stage
))
def
setup_staging_areas
(
self
):
def
setup_staging_areas
(
self
):
logger
.
info
(
"Setting up
the StageAreas
for GPU prefetching ..."
)
logger
.
info
(
"Setting up
StagingArea
for GPU prefetching ..."
)
for
idx
,
device
in
enumerate
(
self
.
_devices
):
for
idx
,
device
in
enumerate
(
self
.
_devices
):
with
tf
.
device
(
device
):
with
tf
.
device
(
device
):
inputs
=
self
.
_input
.
get_input_tensors
()
inputs
=
self
.
_input
.
get_input_tensors
()
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment