Commit ca06ba07 authored by Yuxin Wu's avatar Yuxin Wu

rename 'prefetch.py' to 'parallel.py'

parent efaf9d7b
# -*- coding: UTF-8 -*- # -*- coding: UTF-8 -*-
# File: prefetch.py # File: parallel.py
from __future__ import print_function from __future__ import print_function
...@@ -130,11 +130,10 @@ class PrefetchData(ProxyDataFlow): ...@@ -130,11 +130,10 @@ class PrefetchData(ProxyDataFlow):
b. When ``nr_proc>1``, the dataflow produces the same distribution b. When ``nr_proc>1``, the dataflow produces the same distribution
of data as ``ds`` if each sample from ``ds`` is i.i.d. (e.g. fully shuffled). of data as ``ds`` if each sample from ``ds`` is i.i.d. (e.g. fully shuffled).
You probably only want to use it for training. You probably only want to use it for training.
2. This is significantly slower than :class:`PrefetchDataZMQ` when data is large. 2. This has more serialization overhead than :class:`PrefetchDataZMQ` when data is large.
3. When nesting like this: ``PrefetchDataZMQ(PrefetchData(df, nr_proc=a), nr_proc=b)``. 3. You can nest like this: ``PrefetchDataZMQ(PrefetchData(df, nr_proc=a), nr_proc=b)``.
A total of ``a`` instances of ``df`` worker processes will be created. A total of ``a`` instances of ``df`` worker processes will be created.
This is different from the behavior of :class:`PrefetchDataZMQ` 4. fork happens in `__init__`. `reset_state()` is a no-op. The worker processes won't get called.
4. `reset_state()` is a no-op. The worker processes won't get called.
""" """
class _Worker(mp.Process): class _Worker(mp.Process):
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment