Distex documentation¶
Release 0.7.1.
Pool¶
-
class
distex.pool.
Pool
(num_workers=0, hosts=None, qsize=2, initializer=None, initargs=(), localhost='', localport=0, lazy_create=False, worker_loop=<LoopType.default: 0>, func_pickle=<PickleType.dill: 2>, data_pickle=<PickleType.pickle: 0>)[source]¶ Pool of local and remote workers that can run tasks.
To create a process pool of 4 local workers:
pool = Pool(4)
To create 8 remote workers on host
maxi
, using SSH (unix only):pool = Pool(0, 'ssh://maxi/8')
distex
must be installed on all remote hosts and thedistex_proc
script must be in the path. Test this withssh <host> distex_proc
. When using SSH it is not necessary to have a distex server running on the hosts.When not using SSH a spawning server has to be started first on all hosts involved:
python3 -m distex.server
Warning
Only use this in a trusted network environment.
With the server running on host
mini
, to create a pool of 2 workers running there:pool = Pool(0, 'mini/2')
Local, remote SSH and remote non-SSH workers can all be combined in one pool:
pool = Pool(4, ['ssh://maxi/8', 'mini/2'])
To give a SSH username or a non-default port such as 10022, specify the host as
'ssh://username@maxi:10022/8'
. It is not possible to give a password, use SSH keys instead: ssh-keygen can be used to create a key and ssh-copy-id to copy it to all hosts.- Parameters
num_workers (
int
) – Number of local process workers. The default of 0 will use the number of CPUs.hosts – List of remote host specification strings in the format
[ssh://][username@]hostname[:portnumber]/num_workers
.qsize (
int
) – Number of pending tasks per worker. To improve the throughput of small tasks this can be increased from the default of 2. If no queueing is desired then it can be set to 1.initializer – Callable to initialize worker processes.
initargs (
tuple
) – Arguments tuple that is unpacked into the initializer.localhost (
str
) – Local TCP server (if any) will listen on this address.localport (
int
) – Local TCP server (if any) will listen on this port (default: random open port).lazy_create (
bool
) – If True then no workers will be created until the first task is submitted.worker_loop (
int
) –LoopType
to use for workers:default (=uvloop when available, proactor on Windows)
asyncio (standard selector event loop)
uvloop (Unix only)
proactor (Windows only)
quamash (PyQt)
func_pickle (
int
) –PickleType
to to use for serializing functions:pickle
cloudpickle
dill
data_pickle (
int
) –PickleType
to to use for data:pickle
cloudpickle
dill
distex.Pool
implements theconcurrent.futures.Executor
interface and can be used in the place of ProcessPoolExecutor.
PoolMap¶
-
class
distex.poolmap.
PoolMap
(pool, func, timeout=None, chunksize=1, ordered=True, source=None)[source]¶ Map a function using a distributed pool with eventkit.
- Parameters
func – Function to map. If it returns an awaitable then the result will be awaited and returned.
timeout – Timeout in seconds since map is started.
chunksize – Source emits are chunked up to this size. A larger chunksize can greatly improve efficiency for small tasks.
ordered –
True
: The order of results preserves the source order.False
: Results are in order of completion.