/usr/local/lib/python3.6/site-packages/joblib/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/joblib/__pycache__/memory.cpython-36.pyc (28834B)
3
Eg1 @ st d Z ddlmZ ddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl Z ddl
mZ ddlmZ ddlmZmZmZ ddlmZ dd lmZ dd
lmZmZmZ dd l m Z m Z d Z d
d Z G dd de!Z"de iZ#dd Z$d&ddZ%dd Z&dd Z'd'ddZ(e j) Z*G d d d eZ+G d d d e,Z-G d d! d!e,Z.G d"d# d#eZ/G d$d% d%eZ0dS )(zo
A context object for caching a function's return value each time it
is called with the same input arguments.
)with_statementN)open )hashing)
get_func_code
get_func_name filter_args) format_call)format_signature)Logger format_timepformat)StoreBackendBaseFileSystemStoreBackendz
# first line:c C sL | j tr@| jd} t| d ttd }dj| dd } nd}| |fS )zZ Extract the first line information from the function code
text if available.
r Nr )
startswithFIRST_LINE_TEXTsplitintlenjoin) func_code
first_line r 7/usr/local/lib/python3.6/site-packages/joblib/memory.pyextract_first_line2 s
r c @ s e Zd ZdZdS )JobLibCollisionWarningzF Warn that there might be a collision between names of functions.
N)__name__
__module__ __qualname____doc__r r r r r ? s r localc C sF t | tstdj| |dks,t|t r:tdj||t| < dS )a Extend available store backends.
The Memory, MemorizeResult and MemorizeFunc objects are designed to be
agnostic to the type of store used behind. By default, the local file
system is used but this function gives the possibility to extend joblib's
memory pattern with other types of storage such as cloud storage (S3, GCS,
OpenStack, HadoopFS, etc) or blob DBs.
Parameters
----------
backend_name: str
The name identifying the store backend being registered. For example,
'local' is used with FileSystemStoreBackend.
backend: StoreBackendBase subclass
The name of a class that implements the StoreBackendBase interface.
z3Store backend name should be a string, '{0}' given.Nz;Store backend should inherit StoreBackendBase, '{0}' given.)
isinstancestr
ValueErrorformat
issubclassr _STORE_BACKENDS) backend_namebackendr r r register_store_backendG s
r+ c C s |dkr i }t |tjr t|}t |tr.|S t |trd}tjj|}x"tj D ]\}}| |krR| }qRW |dkrt
dj || |j |||d |S |dk rt
jdj |jjt dS )z7Return the correct store object for the given location.Nz#Unknown location {0} or backend {1})verbosebackend_optionszdInstanciating a backend using a {} as a location is not supported by joblib. Returning None instead.)r# pathlibPathr$ r ospath
expanduserr( items TypeErrorr& configurewarningswarn __class__r UserWarning)r* locationr, r- objZ backend_keyZ backend_objr r r _store_backend_factoryd s.
r< c C s" t | \}}|j| tjj| S )z4Compute the part of part associated with a function.)r appendr0 r1 r )funcmodulesfuncnamer r r _get_func_fullname s
rA c C s4 g }t | tr|j| n|jt| tjj| S )z:Build a roughly unique identifier for the cached function.)r# r$ r= rA r0 r1 r )r> partsr r r _build_func_identifier s
rC c C s d}yJ|dk r@dj dd |d j D }dtjj| |f }n tjj| }W n tk
rb Y nX |dk rdjttj | }nd}d j|t |S )
zE Helper function to format the message when loading the results.
Nz, c S s g | ]\}}d ||f qS )z%s=%sr ).0namevaluer r r
s z$_format_load_msg..
input_argsz%s(%s)z{0: <16}z[Memory]{0}: Loading {1})
r r3 r0 r1 basenameKeyErrorr& r timer$ )func_idargs_id timestampmetadata signatureargsZ ts_stringr r r _format_load_msg s
rS c @ sF e Zd ZdZdddZedd Zd d
Zd d Zd
d Z dd Z
dS )MemorizedResultal Object representing a cached value.
Attributes
----------
location: str
The location of joblib cache. Depends on the store backend used.
func: function or str
function whose output is cached. The string case is intended only for
instanciation based on the output of repr() on another instance.
(namely eval(repr(memorized_instance)) works).
argument_hash: str
hash of the function arguments.
backend: str
Type of store backend for reading/writing cache files.
Default is 'local'.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
The memmapping mode used when loading from cache numpy arrays. See
numpy.load for the meaning of the different values.
verbose: int
verbosity level (0 means no message).
timestamp, metadata: string
for internal use only.
r" Nr c C s t j| t|| _t|tr&|| _n| j| _|| _t|||d| _ || _
|d k rZ|| _ n| j j | j| jg| _ | j j
dd | _|| _|| _d S )N)r, duration)r __init__rC rM r# r$ r> rN r<
store_backend mmap_moderP get_metadatagetrU r, rO ) selfr: r> rN r* rX r, rO rP r r r rV s
zMemorizedResult.__init__c C s t jdtdd | jS )NzThe 'argument_hash' attribute has been deprecated in version 0.12 and will be removed in version 0.14.
Use `args_id` attribute instead. )
stacklevel)r6 r7 DeprecationWarningrN )r[ r r r
argument_hash s
z MemorizedResult.argument_hashc C s | j r t| j| j| j| jd}nd}y | jj| j| jg|| j dS tk
r } z.t dj
t j j
| jj| j| j}||W Y dd}~X nX dS )z$Read value from cache and return it.)rO rP N)msgr, zaError while trying to load a MemorizedResult's value. It seems that this folder is corrupted : {})r, rS rM rN rO rP rW load_itemr% rK r& r0 r1 r r: )r[ r` excnew_excr r r rZ s
zMemorizedResult.getc C s | j j| j| jg dS )zClear value from cacheN)rW Z
clear_itemrM rN )r[ r r r clear s zMemorizedResult.clearc C s dj | jj| jj| j| jdS )NzG{class_name}(location="{location}", func="{func}", args_id="{args_id}"))
class_namer: r> rN )r& r8 r rW r: r> rN )r[ r r r __repr__ s
zMemorizedResult.__repr__c C s | j j }d |d< |S )NrO )__dict__copy)r[ stater r r __getstate__ s
z MemorizedResult.__getstate__)r" Nr NN) r r r r! rV propertyr_ rZ rd rf rj r r r r rT s
rT c @ sD e Zd ZdZdZdd Zdd Zdd Zd
d Zd d
Z dd Z
dS )NotMemorizedResultzxClass representing an arbitrary value.
This class is a replacement for MemorizedResult when there is no cache.
rG validc C s || _ d| _d S )NT)rG rm )r[ rG r r r rV + s zNotMemorizedResult.__init__c C s | j r | jS tdd S )NzNo value stored.)rm rG rK )r[ r r r rZ / s zNotMemorizedResult.getc C s d| _ d | _d S )NF)rm rG )r[ r r r rd 5 s zNotMemorizedResult.clearc C s. | j r dj| jjt| jdS | jjd S d S )Nz{class_name}({value}))re rG z with no value)rm r& r8 r r
rG )r[ r r r rf 9 s
zNotMemorizedResult.__repr__c C s | j | jdS )N)rm rG )rm rG )r[ r r r rj B s z NotMemorizedResult.__getstate__c C s |d | _ |d | _d S )Nrm rG )rm rG )r[ ri r r r __setstate__E s
z NotMemorizedResult.__setstate__N)rG rm ) r r r r! __slots__rV rZ rd rf rj rn r r r r rl $ s rl c @ sJ e Zd ZdZdd Zdd Zdd Zdd Zdd d Zd
d Z dd Z
dS )NotMemorizedFunca No-op object decorating a function.
This class replaces MemorizedFunc when there is no cache. It provides an
identical API but does not write anything on disk.
Attributes
----------
func: callable
Original undecorated function.
c C s
|| _ d S )N)r> )r[ r> r r r rV Y s zNotMemorizedFunc.__init__c O s | j ||S )N)r> )r[ rR kwargsr r r __call__\ s zNotMemorizedFunc.__call__c O s t | j||S )N)rl r> )r[ rR rq r r r call_and_shelve_ s z NotMemorizedFunc.call_and_shelvec C s dj | jj| jS )Nz
{0}(func={1}))r& r8 r r> )r[ r r r rf b s zNotMemorizedFunc.__repr__Tc C s d S )Nr )r[ r7 r r r rd e s zNotMemorizedFunc.clearc O s | j ||S )N)r> )r[ rR rq r r r calli s zNotMemorizedFunc.callc O s dS )NFr )r[ rR rq r r r check_call_in_cachel s z$NotMemorizedFunc.check_call_in_cacheN)T) r r r r! rV rr rs rf rd rt ru r r r r rp M s
rp c @ s e Zd ZdZd)ddZd*dd Zed
d Zd d
Zdd Z dd Z
dd Z dd Z dd Z
dd Zdd Zd+d d Zd,d d!Zd"d# Zd-d%d&Zd'd( ZdS ).
MemorizedFunca Callable object decorating a function for caching its return value
each time it is called.
Methods are provided to inspect the cache or clean it.
Attributes
----------
func: callable
The original, undecorated, function.
location: string
The location of joblib cache. Depends on the store backend used.
backend: str
Type of store backend for reading/writing cache files.
Default is 'local', in which case the location is the path to a
disk storage.
ignore: list or None
List of variable names to ignore when choosing whether to
recompute.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
The memmapping mode used when loading from cache
numpy arrays. See numpy.load for the meaning of the different
values.
compress: boolean, or integer
Whether to zip the stored data on disk. If an integer is
given, it should be between 1 and 9, and sets the amount
of compression. Note that compressed arrays cannot be
read by memmapping.
verbose: int, optional
The verbosity flag, controls messages that are issued as
the function is evaluated.
r" NFr c
C s t j| || _|| _|| _|d kr(g }|| _|| _t|||t||dd| _ | j d k rl| j j
t | jg |d kr|t j }|| _
ytj| | W n Y nX tj|rtj j|} | jddd} tjdd| } n|j} d| | _d | _d | _d S ) N)compressrX )r, r- r z
r z.rD zMemoized version of %s)r rV rX rw r> ignore_verboser< dictrW store_cached_func_coderC rL rO functoolsupdate_wrapperinspect
isfunctionpydocZTextDocdocumentreplaceresubr! _func_code_info
_func_code_id)
r[ r> r: r* rx rX rw r, rO docr r r rV s>
zMemorizedFunc.__init__c C s | j ||\}}d}d}d}| jddo6| jj||gsx| jdkrrt| j\} }
| jdj|
|| jj |gd d}nyt
j
} | jrt ||| j |d }|s| jj
||g|| jd
} nd} | jdkrt
j
| }
t| j\} }
d |
t|
f }ttd d
t| d | W nF tk
rX t| jf||\} }| jdj|tj d}Y nX |r| j||\} }| jdk r| jrt ||| j |d }| jj
||g|| jd
} | ||fS )a Call wrapped function and cache result, or read cache if available.
This function returns the wrapped function output and some metadata.
Arguments:
----------
args, kwargs: list and dict
input arguments for wrapped function
shelving: bool
True when called via the call_and_shelve function.
Returns
-------
output: value or tuple or None
Output of the wrapped function.
If shelving is True and the call has been already cached,
output is None.
argument_hash: string
Hash of function arguments.
metadata: dict
Some metadata about wrapped function call (see _persist_input()).
NF )r]
z5Computing func {0}, argument hash {1} in location {2}r: T)rO rP )r` r, z%s cache loaded - %sr P _z*Exception while loading results for {}
{})_get_output_identifiers_check_previous_func_coderW
contains_itemry r r> r7 r& Zget_cached_func_inforL rS rO ra r printmaxr Exceptionr
traceback
format_excrt rX )r[ rR rq shelvingrM rN rP r` Z must_callr rF t0outtrQ r r r _cached_call sZ
"
zMemorizedFunc._cached_callc C sZ t | jdr>| jd kr&t| jj| _nt| jj| jkr>d | _| jd krTt| j| _| jS )N__code__)hasattrr> r idr r r )r[ r r r func_code_info& s
z MemorizedFunc.func_code_infoc O s6 | j ||dd\}}}t| j| j||| jd | jdS )a\ Call wrapped function, cache result and return a reference.
This method returns a reference to the cached result instead of the
result itself. The reference object is small and pickeable, allowing
to send or store it easily. Call .get() on reference object to get
result.
Returns
-------
cached_result: MemorizedResult or NotMemorizedResult
reference to the value returned by the wrapped function. The
class "NotMemorizedResult" is used when there is no cache
activated (e.g. location=None in Memory).
T)r r )rP r, rO )r rT rW r> ry rO )r[ rR rq r rN rP r r r rs = s
z MemorizedFunc.call_and_shelvec O s | j ||d S )Nr )r )r[ rR rq r r r rr Q s zMemorizedFunc.__call__c C s$ | j }| jj }d |d< d |d< |S )NrO r )r rg rh )r[ r ri r r r rj T s
zMemorizedFunc.__getstate__c O s | j ||\}}| jj||fS )a\ Check if function call is in the memory cache.
Does not call the function or do any work besides func inspection
and arg hashing.
Returns
-------
is_call_in_cache: bool
Whether or not the result of the function has been cached
for the input arguments that have been passed.
)r rW r )r[ rR rq rM rN r r r ru d s z!MemorizedFunc.check_call_in_cachec O s" t jt| j| j||| jd k dS )N)Z coerce_mmap)r hashr r> rx rX )r[ rR rq r r r _get_argument_hashw s z MemorizedFunc._get_argument_hashc O s t | j}| j||}||fS )z@Return the func identifier and input parameter hash of a result.)rC r> r )r[ rR rq rM r_ r r r r { s
z%MemorizedFunc._get_output_identifiersc C s( t t| jdd}t| jt | j|fS )z'Hash a function to key the online cacher N)r getattrr> r )r[ Z func_code_hr r r
_hash_func s zMemorizedFunc._hash_funcc C sx t | j}dt||f }| jj|g| d}t| jdoB| jjdk}|rt| j }y|t| j< W n t k
rr Y nX dS )z= Write the function code and the filename to a file.
z%s %i
%sFr zN)
rC r> r rW r{ r r r _FUNCTION_HASHESr4 )r[ r r rM Zis_named_callable func_hashr r r _write_func_code s
z MemorizedFunc._write_func_coder\ c C s y(| j tkr&| j }|t| j kr&dS W n tk
r< Y nX | j\}}}t| j }yt| jj|g\}}W n$ t t
fk
r | j || dS X ||krdS t | j ddd\} }
|| kodkn s|
dkr
|dksdj
|
||} n|
} tjtdj
| |d ||k r|d k rd} tjj|rt | j dd
\} }
t|jd }
t|&}|j |d |d |
d }W d Q R X d j|}|j |j k} n
|jd
} | rtjtd|
|||
||f |d | jdkrt | j dd
\} }
| jdj
|
| | j dd dS )z
stacklevel is the depth a which this function is called, to
issue useful warnings to the user.
TF) resolv_aliasZwin_charactersr zz
{0} ({1}:{2})z0Cannot detect name collisions for function '{0}')r] N)r r rD z r r r4 r rC r rW Zget_cached_func_codeIOErrorOSErrorr r r& r6 r7 r r0 r1 existsr r open_py_source readlinesr rstripr ry rd )r[ r] r r Z source_filer rM Z
old_func_codeZold_first_liner func_nameZfunc_descriptionZpossible_collisionZ num_linesfZon_disk_func_coder r r r sh
"
$
z'MemorizedFunc._check_previous_func_codeTc C sP t | j}| jdkr&|r&| jd| | jj|g | j\}}}| j|| dS )zEmpty the function's cache.r z(Clearing function cache identified by %sN)rC r> ry r7 rW Z
clear_pathr r )r[ r7 rM r r r r r r rd s
zMemorizedFunc.clearc O s t j }| j||\}}| jdkr4tt| j|| | j||}| jj||g|| jd t j | }| j|||}| jdkrt | j\} }
d|
t
|f } tt ddt | d | ||fS )zq Force the execution of the function with the given arguments and
persist the output values.
r )r, z%s - %sr r )
rL r ry r r r> rW Z dump_item_persist_inputr r r r ) r[ rR rq
start_timerM rN outputrU rP r rF r` r r r rt s
zMemorizedFunc.call ?c C s t j }t| j| j||}tdd |j D }||d}| j||\} }
| jj| |
g| t j | } | |krt j
d| dd |S )a% Save a small summary of the call using json format in the
output directory.
output_dir: string
directory where to write metadata.
duration: float
time taken by hashing input arguments, calling the wrapped
function and persisting its output.
args, kwargs: list and dict
input arguments for wrapped function
this_duration_limit: float
Max execution time for this function before issuing a warning.
c s s | ]\}}|t |fV qd S )N)repr)rE kvr r r - s z/MemorizedFunc._persist_input..)rU rI a~ Persisting input arguments took %.2fs to run.
If this happens often in your code, it can cause performance problems
(results will be correct in all cases).
The reason for this is probably some large input arguments for a wrapped
function (e.g. large strings).
THIS IS A JOBLIB ISSUE. If you can, kindly provide the joblib's team with an
example so that they can fix the problem. )r] ) rL r r> rx rz r3 r rW Zstore_metadatar6 r7 ) r[ rU rR rq Zthis_duration_limitr Z
argument_dictZ
input_reprrP rM rN Z
this_durationr r r r s
z MemorizedFunc._persist_inputc C s dj | jj| j| jjdS )Nz.{class_name}(func={func}, location={location}))re r> r: )r& r8 r r> rW r: )r[ r r r rf N s zMemorizedFunc.__repr__)r" NNFr N)F)r\ )T)r )r r r r! rV r rk r rs rr rj ru r r r r r rd rt r rf r r r r rv s s$ %
,
\
X
6rv c @ sZ e Zd ZdZdddZedd Zdd
d Zdd
dZdd Z dd Z
dd Z dd Z dS )Memorya A context object for caching a function's return value each time it
is called with the same input arguments.
All values are cached on the filesystem, in a deep directory
structure.
Read more in the :ref:`User Guide `.
Parameters
----------
location: str, pathlib.Path or None
The path of the base directory to use as a data store
or None. If None is given, no caching is done and
the Memory object is completely transparent. This option
replaces cachedir since version 0.12.
backend: str, optional
Type of store backend for reading/writing cache files.
Default: 'local'.
The 'local' backend is using regular filesystem operations to
manipulate data (open, mv, etc) in the backend.
cachedir: str or None, optional
.. deprecated: 0.12
'cachedir' has been deprecated in 0.12 and will be
removed in 0.14. Use the 'location' parameter instead.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
The memmapping mode used when loading from cache
numpy arrays. See numpy.load for the meaning of the
arguments.
compress: boolean, or integer, optional
Whether to zip the stored data on disk. If an integer is
given, it should be between 1 and 9, and sets the amount
of compression. Note that compressed arrays cannot be
read by memmapping.
verbose: int, optional
Verbosity flag, controls the debug messages that are issued
as functions are evaluated.
bytes_limit: int, optional
Limit in bytes of the size of the cache. By default, the size of
the cache is unlimited. When reducing the size of the cache,
``joblib`` keeps the most recently accessed items first.
**Note:** You need to call :meth:`joblib.Memory.reduce_size` to
actually reduce the cache size to be less than ``bytes_limit``.
backend_options: dict, optional
Contains a dictionnary of named parameters used to configure
the store backend.
Nr" Fr c C s t j| || _|| _tj | _|| _|| _|| _|d kr>i }|| _ |r^|d k r^t
j ddd |d k r|d k r~t dj
||t
j dj
|tdd |}|| _t|trtjj|d}t||| jtf ||d|d| _d S ) Nz&Compressed results cannot be memmappedr\ )r] zYou set both "location={0!r} and "cachedir={1!r}". 'cachedir' has been deprecated in version 0.12 and will be removed in version 0.14.
Please only set "location={0!r}"zThe 'cachedir' parameter has been deprecated in version 0.12 and will be removed in version 0.14.
You provided "cachedir={0!r}", use "location={0!r}" instead.joblib)rw rX )r, r- )r rV ry rX rL rO bytes_limitr* rw r- r6 r7 r% r& r^ r: r# r$ r0 r1 r r< rz rW ) r[ r: r* cachedirrX rw r, r r- r r r rV s<
zMemory.__init__c C s. t jdtdd | jd kr d S tjj| jdS )NzThe 'cachedir' attribute has been deprecated in version 0.12 and will be removed in version 0.14.
Use os.path.join(memory.location, 'joblib') attribute instead.r\ )r] r )r6 r7 r^ r: r0 r1 r )r[ r r r r s
zMemory.cachedirc
C sz |dkr t j| j|||dS | jdkr.t|S |dkr<| j}|dkrJ| j}t|trZ|j }t|| j| j
||| j || j dS )a{ Decorates the given function func to only compute its return
value for input arguments not cached on disk.
Parameters
----------
func: callable, optional
The function to be decorated
ignore: list of strings
A list of arguments name to ignore in the hashing
verbose: integer, optional
The verbosity mode of the function. By default that
of the memory object is used.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
The memmapping mode used when loading from cache
numpy arrays. See numpy.load for the meaning of the
arguments. By default that of the memory object is used.
Returns
-------
decorated_func: MemorizedFunc object
The returned object is a MemorizedFunc object, that is
callable (behaves like a function), but offers extra
methods for cache lookup and management. See the
documentation for :class:`joblib.memory.MemorizedFunc`.
N)rx r, rX F)r: r* rx rX rw r, rO )
r| partialcacherW rp ry rX r# rv r> r* rw rO )r[ r> rx r, rX r r r r s
z Memory.cacheTc C s& |r| j d | jdk r"| jj dS )z- Erase the complete cache directory.
z Flushing completely the cacheN)r7 rW rd )r[ r7 r r r rd s
z Memory.clearc C s& | j dk r"| jdk r"| jj| j dS )z@Remove cache elements to make cache size fit in ``bytes_limit``.N)r rW Zreduce_store_size)r[ r r r reduce_size s zMemory.reduce_sizec O s$ | j dkr|||S | j|||S )a Eval function func with arguments `*args` and `**kwargs`,
in the context of the memory.
This method works similarly to the builtin `apply`, except
that the function is called only if the cache is not
up to date.
N)rW r )r[ r> rR rq r r r eval s
z Memory.evalc C s$ dj | jj| jd krd n| jjdS )Nz!{class_name}(location={location}))re r: )r& r8 r rW r: )r[ r r r rf s zMemory.__repr__c C s | j j }d|d< |S )zf We don't store the timestamp when pickling, to avoid the hash
depending from it.
NrO )rg rh )r[ ri r r r rj s
zMemory.__getstate__)Nr" NNFr NN)NNNF)T)
r r r r! rV rk r r rd r r rf rj r r r r r X s 7
*
-
r )r N)NN)1r!
__future__r r0 rL r. r r r| r r6 r~ weakreftokenizer r rD r Z func_inspectr r r r r
loggerr r r
Z_store_backendsr r r r r9 r r( r+ r< rA rC rS WeakKeyDictionaryr rT objectrl rp rv r r r r r sH
(
g)& h