Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
S
seminar-breakout
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Shashank Suhas
seminar-breakout
Commits
4521b1f7
Commit
4521b1f7
authored
Jun 06, 2017
by
Yuxin Wu
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
add docs and fix initialization in ThreadedMapData (fix #293)
parent
815e9439
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
7 additions
and
3 deletions
+7
-3
tensorpack/dataflow/prefetch.py
tensorpack/dataflow/prefetch.py
+7
-3
No files found.
tensorpack/dataflow/prefetch.py
View file @
4521b1f7
...
...
@@ -212,8 +212,9 @@ class ThreadedMapData(ProxyDataFlow):
Note that the threads will only start in the process which calls
`reset_state()`.
"""
class
WorkerThread
(
StoppableThread
):
class
_
WorkerThread
(
StoppableThread
):
def
__init__
(
self
,
inq
,
outq
,
map_func
):
super
(
ThreadedMapData
.
_WorkerThread
,
self
)
.
__init__
()
self
.
inq
=
inq
self
.
outq
=
outq
self
.
func
=
map_func
...
...
@@ -228,7 +229,10 @@ class ThreadedMapData(ProxyDataFlow):
def
__init__
(
self
,
ds
,
nr_thread
,
map_func
,
buffer_size
=
200
):
"""
Args:
pass
ds (DataFlow): the dataflow to map
nr_thread (int): number of threads to use
map_func (callable): datapoint -> datapoint | None
buffer_size (int): number of datapoints in the buffer
"""
super
(
ThreadedMapData
,
self
)
.
__init__
(
ds
)
self
.
infinite_ds
=
RepeatedData
(
ds
,
-
1
)
...
...
@@ -244,7 +248,7 @@ class ThreadedMapData(ProxyDataFlow):
t
.
join
()
self
.
_in_queue
=
queue
.
Queue
()
self
.
_out_queue
=
queue
.
Queue
()
self
.
_threads
=
[
ThreadedMapData
.
WorkerThread
(
self
.
_threads
=
[
ThreadedMapData
.
_
WorkerThread
(
self
.
_in_queue
,
self
.
_out_queue
,
self
.
map_func
)
for
_
in
range
(
self
.
nr_thread
)]
for
t
in
self
.
_threads
:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment