Name	Size	Mode	Actions
backports.cpython-36.pyc	2555	0644	editdlrm
compressor.cpython-36.pyc	16616	0644	editdlrm
disk.cpython-36.pyc	3050	0644	editdlrm
executor.cpython-36.pyc	3166	0644	editdlrm
format_stack.cpython-36.pyc	844	0644	editdlrm
func_inspect.cpython-36.pyc	8729	0644	editdlrm
hashing.cpython-36.pyc	6265	0644	editdlrm
logger.cpython-36.pyc	3960	0644	editdlrm
memory.cpython-36.pyc	28834	0644	editdlrm
my_exceptions.cpython-36.pyc	1130	0644	editdlrm
numpy_pickle.cpython-36.pyc	15564	0644	editdlrm
numpy_pickle_compat.cpython-36.pyc	7052	0644	editdlrm
numpy_pickle_utils.cpython-36.pyc	6516	0644	editdlrm
parallel.cpython-36.pyc	32416	0644	editdlrm
pool.cpython-36.pyc	12435	0644	editdlrm
testing.cpython-36.pyc	2037	0644	editdlrm
_cloudpickle_wrapper.cpython-36.pyc	543	0644	editdlrm
_dask.cpython-36.pyc	10528	0644	editdlrm
_deprecated_format_stack.cpython-36.pyc	8216	0644	editdlrm
_deprecated_my_exceptions.cpython-36.pyc	2751	0644	editdlrm
_memmapping_reducer.cpython-36.pyc	16124	0644	editdlrm
_multiprocessing_helpers.cpython-36.pyc	1210	0644	editdlrm
_parallel_backends.cpython-36.pyc	19671	0644	editdlrm
_store_backends.cpython-36.pyc	13946	0644	editdlrm
_utils.cpython-36.pyc	1126	0644	editdlrm
__init__.cpython-36.pyc	4538	0644	editdlrm

3 �Eg�7�@s
dZd
dl
Z
d
dlZd
dlZd
dlmZ
ye
Wnek rL


ed�
ZYnXd
dl m Z
d
dl mZ
d
dlm Z
ddlmZ
dd lmZ
dd lmZmZ
d
dlmZ
yd
dlZWnek r�


dZYnXGdd �d e �ZGdd�de�ZGdd�de�ZGdd�de�ZdS)a�
Custom implementation of multiprocessing.Pool with custom pickler. This module provides efficient ways of working with data stored in shared memory with numpy.memmap arrays without inducing any memory copy between the parent and child processes. This module should not be imported if multiprocessing is not available as it implements subclasses of multiprocessing Pool that uses a custom alternative to SimpleQueue. �N)
�sleep)
�Pickler)
�HIGHEST_PROTOCOL)
�BytesIO�
)
�get_memmapping_reducers)
�TemporaryResourcesManager)�mp�assert_spawning)
�Poolc@s&eZ
dZd
Zdefdd�
Zdd�ZdS)�CustomizablePicklera�Pickler that accepts custom reducers. TODO python2_drop : can this be simplified ? HIGHEST_PROTOCOL is selected by default as this pickler is used to pickle ephemeral datastructures for interprocess communication hence no backward compatibility is required. `reducers` is expected to be a dictionary with key/values being `(type, callable)` pairs where `callable` is a function that give an instance of `type` will return a tuple `(constructor, tuple_of_objects)` to rebuild an instance out of the pickled `tuple_of_objects` as would return a `__reduce__` method. See the standard library documentation on pickling for more details. NcCsftj
||
|d
�
|dkri}ttd�r4tjj�|_ntjj�|_x |j�D]\}}|j||�
qJWdS)N)
�protocol�dispatch) r�__init__�hasattrr�copy�copyreg�dispatch_table�items�register)�self�writer�reducersr �type�reduce_func�r�5/usr/local/lib/python3.6/site-packages/joblib/pool.pyrHs





zCustomizablePickler.__init__cs0tt
d
�r"�f
dd�}||j|
<n �|j|
<dS)z@Attach a reducer function to a given type in the dispatch table.rcs�|
�
}|j|d
|
i
�

dS)N�obj)
�save_reduce)rrZreduced)
rrr� dispatcher\s

z0CustomizablePickler.register..dispatcherN)rrrr)rrrrr)
rrrWs zCustomizablePickler.register)�__name__� __module__�__qualname__�__doc__rrrrrrrr0src@s:eZ
dZd
Zd dd�
Zdd�Zdd�Zd d �Zdd�ZdS)�CustomizablePicklingQueuea�Locked Pipe implementation that uses a customizable pickler. This class is an alternative to the multiprocessing implementation of SimpleQueue in order to make it possible to pass custom pickling reducers, for instance to avoid memory copy when passing memory mapped datastructures. `reducers` is expected to be a dict with key / values being `(type, callable)` pairs where `callable` is a function that, given an instance of `type`, will return a tuple `(constructor, tuple_of_objects)` to rebuild an instance out of the pickled `tuple_of_objects` as would return a `__reduce__` method. See the standard library documentation on pickling for more details. NcCsL||_|
j
d
d�
\|_|_|
j�|_tjdkr6d|_n |
j�|_|j �
dS)NF)
�duplex�win32) � _reducers�Pipe�_reader�_writer�Lock�_rlock�sys�platform�_wlock� _make_methods)r�contextrrrrrus





z"CustomizablePicklingQueue.__init__c

Cs t|�

|j
|j|j|j|jfS)
N)r r)r*r,r/r')
rrrr�__getstate__s


z&CustomizablePicklingQueue.__getstate__cCs$|
\|_|_
|_|_|_|j�
dS)
N)r)r*r,r/r'r0)r�staterrr�__setstate__�s
z&CustomizablePicklingQueue.__setstate__c


Cs|jj
�S)
N)r)�poll)
rrrr�empty�s
zCustomizablePicklingQueue.emptyc
s��jj
�_�
�jj�jj����
�fd
d�}
|
�_�jrP�f
dd����_n�j j �_��jdkrp��_n(�jj�jj�����fdd�}|�_dS)Nc s��
z�
�S��
XdS)
Nrr)�racquire�recv�rreleaserr�get�s


z4CustomizablePicklingQueue._make_methods..getc
s,t�}
t
|
�j�j|�

�jj|
j��

dS)
N)rrr'�dumpr*� send_bytes�getvalue)r�buffer)
rrr�send�s


z5CustomizablePicklingQueue._make_methods..sendc

s�
�
z�|�
S��
XdS)
Nr)
r)r?� wlock_acquire� wlock_releaserr�put�s


z4CustomizablePicklingQueue._make_methods..put) r)r8�_recvr,�acquire�releaser:r'�_sendr*r?r/rB)rr:rBr)r7r8r9rr?r@rArr0�s



z'CustomizablePicklingQueue._make_methods)
N) r r!r"r#rr2r4r6r0rrrrr$ds r$cs*eZ
dZd
Zd�f
dd� Zdd�Z�ZS)�PicklingPoola�Pool implementation with customizable pickling reducers. This is useful to control how data is shipped between processes and makes it possible to use shared memory without useless copies induces by the default pickling methods of the original objects passed as arguments to dispatch. `forward_reducers` and `backward_reducers` are expected to be dictionaries with key/values being `(type, callable)` pairs where `callable` is a function that, given an instance of `type`, will return a tuple `(constructor, tuple_of_objects)` to rebuild an instance out of the pickled `tuple_of_objects` as would return a `__reduce__` method. See the standard library documentation about pickling for more details. NcsR|dkrt�}|dkrt�}||_
||_t|
d
�
}|j|�

tt|�jf|�

dS)N)
� processes)�dict�_forward_reducers�_backward_reducers�update�superrGr)rrH�forward_reducers�backward_reducers�kwargs�poolargs)
� __class__rrr�s







zPicklingPool.__init__c
Cs@t|d
t
�}
t|
|j�|_t|
|j�|_|jj|_|jj |_ dS)NZ_ctx)�getattrr r$rJZ_inqueuerKZ _outqueuerFZ _quick_putrCZ _quick_get)rr1rrr� _setup_queues�s






zPicklingPool._setup_queues)NNN)r r!r"r#rrT� __classcell__rr)
rRrrG�s
rGc s:eZ
dZd
Zd �f
dd� Z�f
d d �Zedd��
Z�ZS)�MemmappingPoola:Process pool that shares large arrays to avoid memory copy. This drop-in replacement for `multiprocessing.pool.Pool` makes it possible to work efficiently with shared memory in a numpy context. Existing instances of numpy.memmap are preserved: the child suprocesses will have access to the same shared memory in the original mode except for the 'w+' mode that is automatically transformed as 'r+' to avoid zeroing the original data upon instantiation. Furthermore large arrays from the parent process are automatically dumped to a temporary folder on the filesystem such as child processes to access their content via memmapping (file system backed shared memory). Note: it is important to call the terminate method to collect the temporary folder used by the pool. Parameters ---------- processes: int, optional Number of worker processes running concurrently in the pool. initializer: callable, optional Callable executed on worker process creation. initargs: tuple, optional Arguments passed to the initializer callable. temp_folder: (str, callable) optional If str: Folder to be used by the pool for memmapping large arrays for sharing memory with worker processes. If None, this will try in order: - a folder pointed by the JOBLIB_TEMP_FOLDER environment variable, - /dev/shm if the folder exists and is writable: this is a RAMdisk filesystem available by default on modern Linux distributions, - the default system temporary folder that can be overridden with TMP, TMPDIR or TEMP environment variables, typically /tmp under Unix operating systems. if callable: An callable in charge of dynamically resolving a temporary folder for memmapping large arrays. max_nbytes int or None, optional, 1e6 by default Threshold on the size of arrays passed to the workers that triggers automated memory mapping in temp_folder. Use None to disable memmapping of large arrays. mmap_mode: {'r+', 'r', 'w+', 'c'} Memmapping mode for numpy arrays passed to workers. See 'max_nbytes' parameter documentation for more details. forward_reducers: dictionary, optional Reducers used to pickle objects passed from master to worker processes: see below. backward_reducers: dictionary, optional Reducers used to pickle return values from workers back to the master process. verbose: int, optional Make it possible to monitor how the communication of numpy arrays with the subprocess is handled (pickling or memmapping) prewarm: bool or str, optional, "auto" by default. If True, force a read on newly memmapped array to make sure that OS pre-cache it in memory. This can be useful to avoid concurrent disk access when the same data array is passed to different worker processes. If "auto" (by default), prewarm is set to True, unless the Linux shared memory partition /dev/shm is available and used as temp folder. `forward_reducers` and `backward_reducers` are expected to be dictionaries with key/values being `(type, callable)` pairs where `callable` is a function that give an instance of `type` will return a tuple `(constructor, tuple_of_objects)` to rebuild an instance out of the pickled `tuple_of_objects` as would return a `__reduce__` method. See the standard library documentation on pickling for more details. N���.A�
rr
Fc sn|dk rtj
d
t�
t|�
}||_t|j|||||d| d�\}}t|
||d�}|j| �

t t |�jf|�

dS)NzPcontext_id is deprecated and ignored in joblib 0.9.4 and will be removed in 0.11F)Ztemp_folder_resolver� max_nbytes� mmap_moderNrO�verboseZunlink_on_gc_collect�prewarm)rHrNrO)�warnings�warn�DeprecationWarningr�_temp_folder_managerr�resolve_temp_folder_namerIrLrMrVr) rrHZtemp_folderrYrZrNrOr[Z context_idr\rP�managerrQ)
rRrrr&
s"











zMemmappingPool.__init__c
s�d
}
xtt|
�
D]h}yt
t|�j�
PWqtk rt
}
z2t|t�rdtd�

|d|
krdtj d|�

WYdd}~XqXqW|j j�
dS)N� g�������?rz@Failed to terminate worker processes in multiprocessing pool: %r)�rangerMrV� terminate�OSError� isinstance�WindowsErrorrr]r^r`Z_unlink_temporary_resources)rZ n_retries�
i�
e)
rRrrreE
s










zMemmappingPool.terminatec

Cs,t|d
d�dk r|j
S|jj�|_
|j
SdS)N�_cached_temp_folder)rSrkr`ra)
rrrr�_temp_folderU
s

zMemmappingPool._temp_folder) NNrWrXNNr
NF) r r!r"r#rre�propertyrlrUrr)
rRrrV�sK

rV)r#rr-r]�timerrh� NameErrorr�picklerr�iorZ_memmapping_reducerrrZ_multiprocessing_helpersr r Zmultiprocessing.poolrZnumpy�np�ImportErrorr�objectr$rGrVrrrr�s.











4N'