Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
S
seminar-breakout
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Shashank Suhas
seminar-breakout
Commits
ae7b0774
Commit
ae7b0774
authored
May 28, 2019
by
Yuxin Wu
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
update dataflow docs about windows (#1219)
parent
afba8dee
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
18 additions
and
1 deletion
+18
-1
docs/tutorial/efficient-dataflow.md
docs/tutorial/efficient-dataflow.md
+11
-0
tensorpack/dataflow/parallel.py
tensorpack/dataflow/parallel.py
+7
-1
No files found.
docs/tutorial/efficient-dataflow.md
View file @
ae7b0774
...
@@ -272,6 +272,17 @@ df = RemoteDataZMQ('ipc://@my-socket', 'tcp://0.0.0.0:8877')
...
@@ -272,6 +272,17 @@ df = RemoteDataZMQ('ipc://@my-socket', 'tcp://0.0.0.0:8877')
TestDataSpeed
(
df
)
.
start
()
TestDataSpeed
(
df
)
.
start
()
```
```
## Common Issues on Windows:
1.
Windows does not support ZMQ. You can only use
`MultiProcessRunner`
,
`MultiThreadRunner`
, and
`MultiThreadMapData`
. But you cannot use
`MultiProcessRunnerZMQ`
or
`MultiProcessMapData`
(which is an alias of
`MultiProcessMapDataZMQ`
).
2.
Windows needs to pickle your dataflow to run it in multiple processes.
As a result you cannot use lambda functions for mappings, like the examples above.
You need to write a new function in global scope that does the mapping.
This issue also exist on Linux if you do not use the 'fork' start method.
[
1
]:
#ref
[
1
]:
#ref
<div
id=
ref
>
</div>
<div
id=
ref
>
</div>
...
...
tensorpack/dataflow/parallel.py
View file @
ae7b0774
...
@@ -178,7 +178,8 @@ class MultiProcessRunner(ProxyDataFlow):
...
@@ -178,7 +178,8 @@ class MultiProcessRunner(ProxyDataFlow):
Args:
Args:
ds (DataFlow): input DataFlow.
ds (DataFlow): input DataFlow.
num_prefetch (int): size of the queue to hold prefetched datapoints.
num_prefetch (int): size of the queue to hold prefetched datapoints.
num_proc (int): number of processes to use.
Required.
num_proc (int): number of processes to use. Required.
nr_prefetch, nr_proc: deprecated argument names
nr_prefetch, nr_proc: deprecated argument names
"""
"""
if
nr_prefetch
is
not
None
:
if
nr_prefetch
is
not
None
:
...
@@ -187,6 +188,8 @@ class MultiProcessRunner(ProxyDataFlow):
...
@@ -187,6 +188,8 @@ class MultiProcessRunner(ProxyDataFlow):
if
nr_proc
is
not
None
:
if
nr_proc
is
not
None
:
log_deprecated
(
"MultiProcessRunner(nr_proc)"
,
"Renamed to 'num_proc'"
,
"2020-01-01"
)
log_deprecated
(
"MultiProcessRunner(nr_proc)"
,
"Renamed to 'num_proc'"
,
"2020-01-01"
)
num_proc
=
nr_proc
num_proc
=
nr_proc
if
num_prefetch
is
None
or
num_proc
is
None
:
raise
TypeError
(
"Missing argument num_prefetch or num_proc in MultiProcessRunner!"
)
# https://docs.python.org/3.6/library/multiprocessing.html?highlight=process#the-spawn-and-forkserver-start-methods
# https://docs.python.org/3.6/library/multiprocessing.html?highlight=process#the-spawn-and-forkserver-start-methods
if
os
.
name
==
'nt'
:
if
os
.
name
==
'nt'
:
...
@@ -424,6 +427,9 @@ class MultiThreadRunner(DataFlow):
...
@@ -424,6 +427,9 @@ class MultiThreadRunner(DataFlow):
if
nr_thread
is
not
None
:
if
nr_thread
is
not
None
:
log_deprecated
(
"MultiThreadRunner(nr_thread)"
,
"Renamed to 'num_thread'"
,
"2020-01-01"
)
log_deprecated
(
"MultiThreadRunner(nr_thread)"
,
"Renamed to 'num_thread'"
,
"2020-01-01"
)
num_thread
=
nr_thread
num_thread
=
nr_thread
if
num_prefetch
is
None
or
num_thread
is
None
:
raise
TypeError
(
"Missing argument num_prefetch or num_thread in MultiThreadRunner!"
)
assert
num_thread
>
0
,
num_thread
assert
num_thread
>
0
,
num_thread
assert
num_prefetch
>
0
,
num_prefetch
assert
num_prefetch
>
0
,
num_prefetch
self
.
num_thread
=
num_thread
self
.
num_thread
=
num_thread
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment