joblib API Reference
Full documentation for joblib (1.4.2)
joblib.Logger
Python Library Documentation: class Logger in module joblib.logger class Logger(builtins.object) | Logger(depth=3, name=None) | | Base class for logging messages. | | Methods defined here: | | __init__(self, depth=3, name=None) | Parameters | ---------- | depth: int, optional | The depth of objects printed. | name: str, optional | The namespace to log to. If None, defaults to joblib. | | debug(self, msg) | | format(self, obj, indent=0) | Return the formatted representation of the object. | | info(self, msg) | | warn(self, msg) | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
joblib.MemorizedResult
Python Library Documentation: class MemorizedResult in module joblib.memory
class MemorizedResult(joblib.logger.Logger)
| MemorizedResult(location, call_id, backend='local', mmap_mode=None, verbose=0, timestamp=None, metadata=None)
|
| Object representing a cached value.
|
| Attributes
| ----------
| location: str
| The location of joblib cache. Depends on the store backend used.
|
| func: function or str
| function whose output is cached. The string case is intended only for
| instantiation based on the output of repr() on another instance.
| (namely eval(repr(memorized_instance)) works).
|
| argument_hash: str
| hash of the function arguments.
|
| backend: str
| Type of store backend for reading/writing cache files.
| Default is 'local'.
|
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
| The memmapping mode used when loading from cache numpy arrays. See
| numpy.load for the meaning of the different values.
|
| verbose: int
| verbosity level (0 means no message).
|
| timestamp, metadata: string
| for internal use only.
|
| Method resolution order:
| MemorizedResult
| joblib.logger.Logger
| builtins.object
|
| Methods defined here:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, location, call_id, backend='local', mmap_mode=None, verbose=0, timestamp=None, metadata=None)
| Parameters
| ----------
| depth: int, optional
| The depth of objects printed.
| name: str, optional
| The namespace to log to. If None, defaults to joblib.
|
| __repr__(self)
| Return repr(self).
|
| clear(self)
| Clear value from cache
|
| get(self)
| Read value from cache and return it.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| args_id
|
| argument_hash
|
| func
|
| func_id
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.logger.Logger:
|
| debug(self, msg)
|
| format(self, obj, indent=0)
| Return the formatted representation of the object.
|
| info(self, msg)
|
| warn(self, msg)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from joblib.logger.Logger:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
joblib.Memory
Python Library Documentation: class Memory in module joblib.memory
class Memory(joblib.logger.Logger)
| Memory(location=None, backend='local', mmap_mode=None, compress=False, verbose=1, bytes_limit=None, backend_options=None)
|
| A context object for caching a function's return value each time it
| is called with the same input arguments.
|
| All values are cached on the filesystem, in a deep directory
| structure.
|
| Read more in the :ref:`User Guide <memory>`.
|
| Parameters
| ----------
| location: str, pathlib.Path or None
| The path of the base directory to use as a data store
| or None. If None is given, no caching is done and
| the Memory object is completely transparent. This option
| replaces cachedir since version 0.12.
|
| backend: str, optional
| Type of store backend for reading/writing cache files.
| Default: 'local'.
| The 'local' backend is using regular filesystem operations to
| manipulate data (open, mv, etc) in the backend.
|
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
| The memmapping mode used when loading from cache
| numpy arrays. See numpy.load for the meaning of the
| arguments.
|
| compress: boolean, or integer, optional
| Whether to zip the stored data on disk. If an integer is
| given, it should be between 1 and 9, and sets the amount
| of compression. Note that compressed arrays cannot be
| read by memmapping.
|
| verbose: int, optional
| Verbosity flag, controls the debug messages that are issued
| as functions are evaluated.
|
| bytes_limit: int | str, optional
| Limit in bytes of the size of the cache. By default, the size of
| the cache is unlimited. When reducing the size of the cache,
| ``joblib`` keeps the most recently accessed items first. If a
| str is passed, it is converted to a number of bytes using units
| { K | M | G} for kilo, mega, giga.
|
| **Note:** You need to call :meth:`joblib.Memory.reduce_size` to
| actually reduce the cache size to be less than ``bytes_limit``.
|
| **Note:** This argument has been deprecated. One should give the
| value of ``bytes_limit`` directly in
| :meth:`joblib.Memory.reduce_size`.
|
| backend_options: dict, optional
| Contains a dictionary of named parameters used to configure
| the store backend.
|
| Method resolution order:
| Memory
| joblib.logger.Logger
| builtins.object
|
| Methods defined here:
|
| __getstate__(self)
| We don't store the timestamp when pickling, to avoid the hash
| depending from it.
|
| __init__(self, location=None, backend='local', mmap_mode=None, compress=False, verbose=1, bytes_limit=None, backend_options=None)
| Parameters
| ----------
| depth: int, optional
| The depth of objects printed.
| name: str, optional
| The namespace to log to. If None, defaults to joblib.
|
| __repr__(self)
| Return repr(self).
|
| cache(self, func=None, ignore=None, verbose=None, mmap_mode=False, cache_validation_callback=None)
| Decorates the given function func to only compute its return
| value for input arguments not cached on disk.
|
| Parameters
| ----------
| func: callable, optional
| The function to be decorated
| ignore: list of strings
| A list of arguments name to ignore in the hashing
| verbose: integer, optional
| The verbosity mode of the function. By default that
| of the memory object is used.
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
| The memmapping mode used when loading from cache
| numpy arrays. See numpy.load for the meaning of the
| arguments. By default that of the memory object is used.
| cache_validation_callback: callable, optional
| Callable to validate whether or not the cache is valid. When
| the cached function is called with arguments for which a cache
| exists, this callable is called with the metadata of the cached
| result as its sole argument. If it returns True, then the
| cached result is returned, else the cache for these arguments
| is cleared and recomputed.
|
| Returns
| -------
| decorated_func: MemorizedFunc object
| The returned object is a MemorizedFunc object, that is
| callable (behaves like a function), but offers extra
| methods for cache lookup and management. See the
| documentation for :class:`joblib.memory.MemorizedFunc`.
|
| clear(self, warn=True)
| Erase the complete cache directory.
|
| eval(self, func, *args, **kwargs)
| Eval function func with arguments `*args` and `**kwargs`,
| in the context of the memory.
|
| This method works similarly to the builtin `apply`, except
| that the function is called only if the cache is not
| up to date.
|
| reduce_size(self, bytes_limit=None, items_limit=None, age_limit=None)
| Remove cache elements to make the cache fit its limits.
|
| The limitation can impose that the cache size fits in ``bytes_limit``,
| that the number of cache items is no more than ``items_limit``, and
| that all files in cache are not older than ``age_limit``.
|
| Parameters
| ----------
| bytes_limit: int | str, optional
| Limit in bytes of the size of the cache. By default, the size of
| the cache is unlimited. When reducing the size of the cache,
| ``joblib`` keeps the most recently accessed items first. If a
| str is passed, it is converted to a number of bytes using units
| { K | M | G} for kilo, mega, giga.
|
| items_limit: int, optional
| Number of items to limit the cache to. By default, the number of
| items in the cache is unlimited. When reducing the size of the
| cache, ``joblib`` keeps the most recently accessed items first.
|
| age_limit: datetime.timedelta, optional
| Maximum age of items to limit the cache to. When reducing the size
| of the cache, any items last accessed more than the given length of
| time ago are deleted.
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.logger.Logger:
|
| debug(self, msg)
|
| format(self, obj, indent=0)
| Return the formatted representation of the object.
|
| info(self, msg)
|
| warn(self, msg)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from joblib.logger.Logger:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
joblib.Parallel
Python Library Documentation: class Parallel in module joblib.parallel
class Parallel(joblib.logger.Logger)
| Parallel(n_jobs=default(None), backend=default(None), return_as='list', verbose=default(0), timeout=None, pre_dispatch='2 * n_jobs', batch_size='auto', temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None))
|
| Helper class for readable parallel mapping.
|
| Read more in the :ref:`User Guide <parallel>`.
|
| Parameters
| ----------
| n_jobs: int, default=None
| The maximum number of concurrently running jobs, such as the number
| of Python worker processes when ``backend="loky"`` or the size of
| the thread-pool when ``backend="threading"``.
| This argument is converted to an integer, rounded below for float.
| If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
| ``n_cpus`` is obtained with :func:`~cpu_count`.
| For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
| using ``n_jobs=-2`` will result in all CPUs but one being used.
| This argument can also go above ``n_cpus``, which will cause
| oversubscription. In some cases, slight oversubscription can be
| beneficial, e.g., for tasks with large I/O operations.
| If 1 is given, no parallel computing code is used at all, and the
| behavior amounts to a simple python `for` loop. This mode is not
| compatible with ``timeout``.
| None is a marker for 'unset' that will be interpreted as n_jobs=1
| unless the call is performed under a :func:`~parallel_config`
| context manager that sets another value for ``n_jobs``.
| If n_jobs = 0 then a ValueError is raised.
| backend: str, ParallelBackendBase instance or None, default='loky'
| Specify the parallelization backend implementation.
| Supported backends are:
|
| - "loky" used by default, can induce some
| communication and memory overhead when exchanging input and
| output data with the worker Python processes. On some rare
| systems (such as Pyiodide), the loky backend may not be
| available.
| - "multiprocessing" previous process-based backend based on
| `multiprocessing.Pool`. Less robust than `loky`.
| - "threading" is a very low-overhead backend but it suffers
| from the Python Global Interpreter Lock if the called function
| relies a lot on Python objects. "threading" is mostly useful
| when the execution bottleneck is a compiled extension that
| explicitly releases the GIL (for instance a Cython loop wrapped
| in a "with nogil" block or an expensive call to a library such
| as NumPy).
| - finally, you can register backends by calling
| :func:`~register_parallel_backend`. This will allow you to
| implement a backend of your liking.
|
| It is not recommended to hard-code the backend name in a call to
| :class:`~Parallel` in a library. Instead it is recommended to set
| soft hints (prefer) or hard constraints (require) so as to make it
| possible for library users to change the backend from the outside
| using the :func:`~parallel_config` context manager.
| return_as: str in {'list', 'generator', 'generator_unordered'}, default='list'
| If 'list', calls to this instance will return a list, only when
| all results have been processed and retrieved.
| If 'generator', it will return a generator that yields the results
| as soon as they are available, in the order the tasks have been
| submitted with.
| If 'generator_unordered', the generator will immediately yield
| available results independently of the submission order. The output
| order is not deterministic in this case because it depends on the
| concurrency of the workers.
| prefer: str in {'processes', 'threads'} or None, default=None
| Soft hint to choose the default backend if no specific backend
| was selected with the :func:`~parallel_config` context manager.
| The default process-based backend is 'loky' and the default
| thread-based backend is 'threading'. Ignored if the ``backend``
| parameter is specified.
| require: 'sharedmem' or None, default=None
| Hard constraint to select the backend. If set to 'sharedmem',
| the selected backend will be single-host and thread-based even
| if the user asked for a non-thread based backend with
| :func:`~joblib.parallel_config`.
| verbose: int, default=0
| The verbosity level: if non zero, progress messages are
| printed. Above 50, the output is sent to stdout.
| The frequency of the messages increases with the verbosity level.
| If it more than 10, all iterations are reported.
| timeout: float or None, default=None
| Timeout limit for each task to complete. If any task takes longer
| a TimeOutError will be raised. Only applied when n_jobs != 1
| pre_dispatch: {'all', integer, or expression, as in '3*n_jobs'}, default='2*n_jobs'
| The number of batches (of tasks) to be pre-dispatched.
| Default is '2*n_jobs'. When batch_size="auto" this is reasonable
| default and the workers should never starve. Note that only basic
| arithmetics are allowed here and no modules can be used in this
| expression.
| batch_size: int or 'auto', default='auto'
| The number of atomic tasks to dispatch at once to each
| worker. When individual evaluations are very fast, dispatching
| calls to workers can be slower than sequential computation because
| of the overhead. Batching fast computations together can mitigate
| this.
| The ``'auto'`` strategy keeps track of the time it takes for a
| batch to complete, and dynamically adjusts the batch size to keep
| the time on the order of half a second, using a heuristic. The
| initial batch size is 1.
| ``batch_size="auto"`` with ``backend="threading"`` will dispatch
| batches of a single task at a time as the threading backend has
| very little overhead and using larger batch size has not proved to
| bring any gain in that case.
| temp_folder: str or None, default=None
| Folder to be used by the pool for memmapping large arrays
| for sharing memory with worker processes. If None, this will try in
| order:
|
| - a folder pointed by the JOBLIB_TEMP_FOLDER environment
| variable,
| - /dev/shm if the folder exists and is writable: this is a
| RAM disk filesystem available by default on modern Linux
| distributions,
| - the default system temporary folder that can be
| overridden with TMP, TMPDIR or TEMP environment
| variables, typically /tmp under Unix operating systems.
|
| Only active when ``backend="loky"`` or ``"multiprocessing"``.
| max_nbytes int, str, or None, optional, default='1M'
| Threshold on the size of arrays passed to the workers that
| triggers automated memory mapping in temp_folder. Can be an int
| in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
| Use None to disable memmapping of large arrays.
| Only active when ``backend="loky"`` or ``"multiprocessing"``.
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
| Memmapping mode for numpy arrays passed to workers. None will
| disable memmapping, other modes defined in the numpy.memmap doc:
| https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
| Also, see 'max_nbytes' parameter documentation for more details.
|
| Notes
| -----
|
| This object uses workers to compute in parallel the application of a
| function to many different arguments. The main functionality it brings
| in addition to using the raw multiprocessing or concurrent.futures API
| are (see examples for details):
|
| * More readable code, in particular since it avoids
| constructing list of arguments.
|
| * Easier debugging:
| - informative tracebacks even when the error happens on
| the client side
| - using 'n_jobs=1' enables to turn off parallel computing
| for debugging without changing the codepath
| - early capture of pickling errors
|
| * An optional progress meter.
|
| * Interruption of multiprocesses jobs with 'Ctrl-C'
|
| * Flexible pickling control for the communication to and from
| the worker processes.
|
| * Ability to use shared memory efficiently with worker
| processes for large numpy-based datastructures.
|
| Note that the intended usage is to run one call at a time. Multiple
| calls to the same Parallel object will result in a ``RuntimeError``
|
| Examples
| --------
|
| A simple example:
|
| >>> from math import sqrt
| >>> from joblib import Parallel, delayed
| >>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
| [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
|
| Reshaping the output when the function has several return
| values:
|
| >>> from math import modf
| >>> from joblib import Parallel, delayed
| >>> r = Parallel(n_jobs=1)(delayed(modf)(i/2.) for i in range(10))
| >>> res, i = zip(*r)
| >>> res
| (0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5)
| >>> i
| (0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 4.0, 4.0)
|
| The progress meter: the higher the value of `verbose`, the more
| messages:
|
| >>> from time import sleep
| >>> from joblib import Parallel, delayed
| >>> r = Parallel(n_jobs=2, verbose=10)(
| ... delayed(sleep)(.2) for _ in range(10)) #doctest: +SKIP
| [Parallel(n_jobs=2)]: Done 1 tasks | elapsed: 0.6s
| [Parallel(n_jobs=2)]: Done 4 tasks | elapsed: 0.8s
| [Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 1.4s finished
|
| Traceback example, note how the line of the error is indicated
| as well as the values of the parameter passed to the function that
| triggered the exception, even though the traceback happens in the
| child process:
|
| >>> from heapq import nlargest
| >>> from joblib import Parallel, delayed
| >>> Parallel(n_jobs=2)(
| ... delayed(nlargest)(2, n) for n in (range(4), 'abcde', 3))
| ... # doctest: +SKIP
| -----------------------------------------------------------------------
| Sub-process traceback:
| -----------------------------------------------------------------------
| TypeError Mon Nov 12 11:37:46 2012
| PID: 12934 Python 2.7.3: /usr/bin/python
| ........................................................................
| /usr/lib/python2.7/heapq.pyc in nlargest(n=2, iterable=3, key=None)
| 419 if n >= size:
| 420 return sorted(iterable, key=key, reverse=True)[:n]
| 421
| 422 # When key is none, use simpler decoration
| 423 if key is None:
| --> 424 it = izip(iterable, count(0,-1)) # decorate
| 425 result = _nlargest(n, it)
| 426 return map(itemgetter(0), result) # undecorate
| 427
| 428 # General case, slowest method
| TypeError: izip argument #1 must support iteration
| _______________________________________________________________________
|
|
| Using pre_dispatch in a producer/consumer situation, where the
| data is generated on the fly. Note how the producer is first
| called 3 times before the parallel loop is initiated, and then
| called to generate new data on the fly:
|
| >>> from math import sqrt
| >>> from joblib import Parallel, delayed
| >>> def producer():
| ... for i in range(6):
| ... print('Produced %s' % i)
| ... yield i
| >>> out = Parallel(n_jobs=2, verbose=100, pre_dispatch='1.5*n_jobs')(
| ... delayed(sqrt)(i) for i in producer()) #doctest: +SKIP
| Produced 0
| Produced 1
| Produced 2
| [Parallel(n_jobs=2)]: Done 1 jobs | elapsed: 0.0s
| Produced 3
| [Parallel(n_jobs=2)]: Done 2 jobs | elapsed: 0.0s
| Produced 4
| [Parallel(n_jobs=2)]: Done 3 jobs | elapsed: 0.0s
| Produced 5
| [Parallel(n_jobs=2)]: Done 4 jobs | elapsed: 0.0s
| [Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s remaining: 0.0s
| [Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s finished
|
| Method resolution order:
| Parallel
| joblib.logger.Logger
| builtins.object
|
| Methods defined here:
|
| __call__(self, iterable)
| Main function to dispatch parallel tasks.
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_value, traceback)
|
| __init__(self, n_jobs=default(None), backend=default(None), return_as='list', verbose=default(0), timeout=None, pre_dispatch='2 * n_jobs', batch_size='auto', temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None))
| Parameters
| ----------
| depth: int, optional
| The depth of objects printed.
| name: str, optional
| The namespace to log to. If None, defaults to joblib.
|
| __repr__(self)
| Return repr(self).
|
| dispatch_next(self)
| Dispatch more data for parallel processing
|
| This method is meant to be called concurrently by the multiprocessing
| callback. We rely on the thread-safety of dispatch_one_batch to protect
| against concurrent consumption of the unprotected iterator.
|
| dispatch_one_batch(self, iterator)
| Prefetch the tasks for the next batch and dispatch them.
|
| The effective size of the batch is computed here.
| If there are no more jobs to dispatch, return False, else return True.
|
| The iterator consumption and dispatching is protected by the same
| lock so calling this function should be thread safe.
|
| print_progress(self)
| Display the process of the parallel execution only a fraction
| of time, controlled by self.verbose.
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.logger.Logger:
|
| debug(self, msg)
|
| format(self, obj, indent=0)
| Return the formatted representation of the object.
|
| info(self, msg)
|
| warn(self, msg)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from joblib.logger.Logger:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
joblib.PrintTime
Python Library Documentation: class PrintTime in module joblib.logger class PrintTime(builtins.object) | PrintTime(logfile=None, logdir=None) | | Print and log messages while keeping track of time. | | Methods defined here: | | __call__(self, msg='', total=False) | Print the time elapsed between the last call and the current | call, with an optional message. | | __init__(self, logfile=None, logdir=None) | Initialize self. See help(type(self)) for accurate signature. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
joblib.backports
Python Library Documentation: module joblib.backports in joblib
NAME
joblib.backports - Backports of fixes for joblib dependencies
CLASSES
builtins.object
Version
LooseVersion
class LooseVersion(Version)
| LooseVersion(vstring=None)
|
| Backport from deprecated distutils
|
| We maintain this backport to avoid introducing a new dependency on
| `packaging`.
|
| We might rexplore this choice in the future if all major Python projects
| introduce a dependency on packaging anyway.
|
| Method resolution order:
| LooseVersion
| Version
| builtins.object
|
| Methods defined here:
|
| __init__(self, vstring=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| parse(self, vstring)
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| component_re = re.compile('(\\d+ | [a-z]+ | \\.)', re.VERBOSE)
|
| ----------------------------------------------------------------------
| Methods inherited from Version:
|
| __eq__(self, other)
| Return self==value.
|
| __ge__(self, other)
| Return self>=value.
|
| __gt__(self, other)
| Return self>value.
|
| __le__(self, other)
| Return self<=value.
|
| __lt__(self, other)
| Return self<value.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from Version:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from Version:
|
| __hash__ = None
class Version(builtins.object)
| Version(vstring=None)
|
| Backport from deprecated distutils
|
| We maintain this backport to avoid introducing a new dependency on
| `packaging`.
|
| We might rexplore this choice in the future if all major Python projects
| introduce a dependency on packaging anyway.
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __ge__(self, other)
| Return self>=value.
|
| __gt__(self, other)
| Return self>value.
|
| __init__(self, vstring=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __le__(self, other)
| Return self<=value.
|
| __lt__(self, other)
| Return self<value.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __hash__ = None
FUNCTIONS
concurrency_safe_rename(src, dst)
Renames ``src`` into ``dst`` overwriting ``dst`` if it exists.
On Windows os.replace can yield permission errors if executed by two
different processes.
make_memmap(filename, dtype='uint8', mode='r+', offset=0, shape=None, order='C', unlink_on_gc_collect=False)
Custom memmap constructor compatible with numpy.memmap.
This function:
- is a backport the numpy memmap offset fix (See
https://github.com/numpy/numpy/pull/8443 for more details.
The numpy fix is available starting numpy 1.13)
- adds ``unlink_on_gc_collect``, which specifies explicitly whether
the process re-constructing the memmap owns a reference to the
underlying file. If set to True, it adds a finalizer to the
newly-created memmap that sends a maybe_unlink request for the
memmaped file to resource_tracker.
replace(src, dst, *, src_dir_fd=None, dst_dir_fd=None)
Rename a file or directory, overwriting the destination.
If either src_dir_fd or dst_dir_fd is not None, it should be a file
descriptor open to a directory, and the respective path string (src or dst)
should be relative; the path will then be relative to that directory.
src_dir_fd and dst_dir_fd, may not be implemented on your platform.
If they are unavailable, using them will raise a NotImplementedError.
DATA
access_denied_errors = (5, 13)
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\backports.py
joblib.compressor
Python Library Documentation: module joblib.compressor in joblib
NAME
joblib.compressor - Classes and functions for managing compressors.
CLASSES
builtins.object
CompressorWrapper
BZ2CompressorWrapper
GzipCompressorWrapper
LZ4CompressorWrapper
LZMACompressorWrapper
XZCompressorWrapper
ZlibCompressorWrapper
io.BufferedIOBase(_io._BufferedIOBase, io.IOBase)
BinaryZlibFile
BinaryGzipFile
class BZ2CompressorWrapper(CompressorWrapper)
| Method resolution order:
| BZ2CompressorWrapper
| CompressorWrapper
| builtins.object
|
| Methods defined here:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| compressor_file(self, fileobj, compresslevel=None)
| Returns an instance of a compressor file object.
|
| decompressor_file(self, fileobj)
| Returns an instance of a decompressor file object.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| extension = '.bz2'
|
| prefix = b'BZ'
|
| ----------------------------------------------------------------------
| Data descriptors inherited from CompressorWrapper:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class BinaryGzipFile(BinaryZlibFile)
| BinaryGzipFile(filename, mode='rb', compresslevel=3)
|
| A file object providing transparent gzip (de)compression.
|
| If filename is a str or bytes object, it gives the name
| of the file to be opened. Otherwise, it should be a file object,
| which will be used to read or write the compressed data.
|
| mode can be 'rb' for reading (default) or 'wb' for (over)writing
|
| If mode is 'wb', compresslevel can be a number between 1
| and 9 specifying the level of compression: 1 produces the least
| compression, and 9 produces the most compression. 3 is the default.
|
| Method resolution order:
| BinaryGzipFile
| BinaryZlibFile
| io.BufferedIOBase
| _io._BufferedIOBase
| io.IOBase
| _io._IOBase
| builtins.object
|
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| wbits = 31
|
| ----------------------------------------------------------------------
| Methods inherited from BinaryZlibFile:
|
| __init__(self, filename, mode='rb', compresslevel=3)
| Initialize self. See help(type(self)) for accurate signature.
|
| close(self)
| Flush and close the file.
|
| May be called more than once without error. Once the file is
| closed, any other operation on it will raise a ValueError.
|
| fileno(self)
| Return the file descriptor for the underlying file.
|
| read(self, size=-1)
| Read up to size uncompressed bytes from the file.
|
| If size is negative or omitted, read until EOF is reached.
| Returns b'' if the file is already at EOF.
|
| readable(self)
| Return whether the file was opened for reading.
|
| readinto(self, b)
| Read up to len(b) bytes into b.
|
| Returns the number of bytes read (0 for EOF).
|
| seek(self, offset, whence=0)
| Change the file position.
|
| The new position is specified by offset, relative to the
| position indicated by whence. Values for whence are:
|
| 0: start of stream (default); offset must not be negative
| 1: current stream position
| 2: end of stream; offset must not be positive
|
| Returns the new file position.
|
| Note that seeking is emulated, so depending on the parameters,
| this operation may be extremely slow.
|
| seekable(self)
| Return whether the file supports seeking.
|
| tell(self)
| Return the current file position.
|
| writable(self)
| Return whether the file was opened for writing.
|
| write(self, data)
| Write a byte string to the file.
|
| Returns the number of uncompressed bytes written, which is
| always len(data). Note that due to buffering, the file on disk
| may not reflect the data written until close() is called.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from BinaryZlibFile:
|
| closed
| True if this file is closed.
|
| ----------------------------------------------------------------------
| Methods inherited from _io._BufferedIOBase:
|
| detach(self, /)
| Disconnect this buffer from its underlying raw stream and return it.
|
| After the raw stream has been detached, the buffer is in an unusable
| state.
|
| read1(self, size=-1, /)
| Read and return up to size bytes, with at most one read() call to the underlying raw stream.
|
| Return an empty bytes object on EOF.
| A short result does not imply that EOF is imminent.
|
| readinto1(self, buffer, /)
|
| ----------------------------------------------------------------------
| Methods inherited from _io._IOBase:
|
| __del__(...)
|
| __enter__(...)
|
| __exit__(...)
|
| __iter__(self, /)
| Implement iter(self).
|
| __next__(self, /)
| Implement next(self).
|
| flush(self, /)
| Flush write buffers, if applicable.
|
| This is not implemented for read-only and non-blocking streams.
|
| isatty(self, /)
| Return whether this is an 'interactive' stream.
|
| Return False if it can't be determined.
|
| readline(self, size=-1, /)
| Read and return a line from the stream.
|
| If size is specified, at most size bytes will be read.
|
| The line terminator is always b'\n' for binary files; for text
| files, the newlines argument to open can be used to select the line
| terminator(s) recognized.
|
| readlines(self, hint=-1, /)
| Return a list of lines from the stream.
|
| hint can be specified to control the number of lines read: no more
| lines will be read if the total size (in bytes/characters) of all
| lines so far exceeds hint.
|
| truncate(self, size=None, /)
| Truncate file to size bytes.
|
| File pointer is left unchanged. Size defaults to the current IO position
| as reported by tell(). Return the new size.
|
| writelines(self, lines, /)
| Write a list of lines to stream.
|
| Line separators are not added, so it is usual for each of the
| lines provided to have a line separator at the end.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _io._IOBase:
|
| __dict__
class BinaryZlibFile(io.BufferedIOBase)
| BinaryZlibFile(filename, mode='rb', compresslevel=3)
|
| A file object providing transparent zlib (de)compression.
|
| TODO python2_drop: is it still needed since we dropped Python 2 support A
| BinaryZlibFile can act as a wrapper for an existing file object, or refer
| directly to a named file on disk.
|
| Note that BinaryZlibFile provides only a *binary* file interface: data read
| is returned as bytes, and data to be written should be given as bytes.
|
| This object is an adaptation of the BZ2File object and is compatible with
| versions of python >= 2.7.
|
| If filename is a str or bytes object, it gives the name
| of the file to be opened. Otherwise, it should be a file object,
| which will be used to read or write the compressed data.
|
| mode can be 'rb' for reading (default) or 'wb' for (over)writing
|
| If mode is 'wb', compresslevel can be a number between 1
| and 9 specifying the level of compression: 1 produces the least
| compression, and 9 produces the most compression. 3 is the default.
|
| Method resolution order:
| BinaryZlibFile
| io.BufferedIOBase
| _io._BufferedIOBase
| io.IOBase
| _io._IOBase
| builtins.object
|
| Methods defined here:
|
| __init__(self, filename, mode='rb', compresslevel=3)
| Initialize self. See help(type(self)) for accurate signature.
|
| close(self)
| Flush and close the file.
|
| May be called more than once without error. Once the file is
| closed, any other operation on it will raise a ValueError.
|
| fileno(self)
| Return the file descriptor for the underlying file.
|
| read(self, size=-1)
| Read up to size uncompressed bytes from the file.
|
| If size is negative or omitted, read until EOF is reached.
| Returns b'' if the file is already at EOF.
|
| readable(self)
| Return whether the file was opened for reading.
|
| readinto(self, b)
| Read up to len(b) bytes into b.
|
| Returns the number of bytes read (0 for EOF).
|
| seek(self, offset, whence=0)
| Change the file position.
|
| The new position is specified by offset, relative to the
| position indicated by whence. Values for whence are:
|
| 0: start of stream (default); offset must not be negative
| 1: current stream position
| 2: end of stream; offset must not be positive
|
| Returns the new file position.
|
| Note that seeking is emulated, so depending on the parameters,
| this operation may be extremely slow.
|
| seekable(self)
| Return whether the file supports seeking.
|
| tell(self)
| Return the current file position.
|
| writable(self)
| Return whether the file was opened for writing.
|
| write(self, data)
| Write a byte string to the file.
|
| Returns the number of uncompressed bytes written, which is
| always len(data). Note that due to buffering, the file on disk
| may not reflect the data written until close() is called.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| closed
| True if this file is closed.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| wbits = 15
|
| ----------------------------------------------------------------------
| Methods inherited from _io._BufferedIOBase:
|
| detach(self, /)
| Disconnect this buffer from its underlying raw stream and return it.
|
| After the raw stream has been detached, the buffer is in an unusable
| state.
|
| read1(self, size=-1, /)
| Read and return up to size bytes, with at most one read() call to the underlying raw stream.
|
| Return an empty bytes object on EOF.
| A short result does not imply that EOF is imminent.
|
| readinto1(self, buffer, /)
|
| ----------------------------------------------------------------------
| Methods inherited from _io._IOBase:
|
| __del__(...)
|
| __enter__(...)
|
| __exit__(...)
|
| __iter__(self, /)
| Implement iter(self).
|
| __next__(self, /)
| Implement next(self).
|
| flush(self, /)
| Flush write buffers, if applicable.
|
| This is not implemented for read-only and non-blocking streams.
|
| isatty(self, /)
| Return whether this is an 'interactive' stream.
|
| Return False if it can't be determined.
|
| readline(self, size=-1, /)
| Read and return a line from the stream.
|
| If size is specified, at most size bytes will be read.
|
| The line terminator is always b'\n' for binary files; for text
| files, the newlines argument to open can be used to select the line
| terminator(s) recognized.
|
| readlines(self, hint=-1, /)
| Return a list of lines from the stream.
|
| hint can be specified to control the number of lines read: no more
| lines will be read if the total size (in bytes/characters) of all
| lines so far exceeds hint.
|
| truncate(self, size=None, /)
| Truncate file to size bytes.
|
| File pointer is left unchanged. Size defaults to the current IO position
| as reported by tell(). Return the new size.
|
| writelines(self, lines, /)
| Write a list of lines to stream.
|
| Line separators are not added, so it is usual for each of the
| lines provided to have a line separator at the end.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _io._IOBase:
|
| __dict__
class CompressorWrapper(builtins.object)
| CompressorWrapper(obj, prefix=b'', extension='')
|
| A wrapper around a compressor file object.
|
| Attributes
| ----------
| obj: a file-like object
| The object must implement the buffer interface and will be used
| internally to compress/decompress the data.
| prefix: bytestring
| A bytestring corresponding to the magic number that identifies the
| file format associated to the compressor.
| extension: str
| The file extension used to automatically select this compressor during
| a dump to a file.
|
| Methods defined here:
|
| __init__(self, obj, prefix=b'', extension='')
| Initialize self. See help(type(self)) for accurate signature.
|
| compressor_file(self, fileobj, compresslevel=None)
| Returns an instance of a compressor file object.
|
| decompressor_file(self, fileobj)
| Returns an instance of a decompressor file object.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class GzipCompressorWrapper(CompressorWrapper)
| Method resolution order:
| GzipCompressorWrapper
| CompressorWrapper
| builtins.object
|
| Methods defined here:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from CompressorWrapper:
|
| compressor_file(self, fileobj, compresslevel=None)
| Returns an instance of a compressor file object.
|
| decompressor_file(self, fileobj)
| Returns an instance of a decompressor file object.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from CompressorWrapper:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class LZ4CompressorWrapper(CompressorWrapper)
| Method resolution order:
| LZ4CompressorWrapper
| CompressorWrapper
| builtins.object
|
| Methods defined here:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| compressor_file(self, fileobj, compresslevel=None)
| Returns an instance of a compressor file object.
|
| decompressor_file(self, fileobj)
| Returns an instance of a decompressor file object.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| extension = '.lz4'
|
| prefix = b'\x04"M\x18'
|
| ----------------------------------------------------------------------
| Data descriptors inherited from CompressorWrapper:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class LZMACompressorWrapper(CompressorWrapper)
| Method resolution order:
| LZMACompressorWrapper
| CompressorWrapper
| builtins.object
|
| Methods defined here:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| compressor_file(self, fileobj, compresslevel=None)
| Returns an instance of a compressor file object.
|
| decompressor_file(self, fileobj)
| Returns an instance of a decompressor file object.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| extension = '.lzma'
|
| prefix = b']\x00'
|
| ----------------------------------------------------------------------
| Data descriptors inherited from CompressorWrapper:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class XZCompressorWrapper(LZMACompressorWrapper)
| Method resolution order:
| XZCompressorWrapper
| LZMACompressorWrapper
| CompressorWrapper
| builtins.object
|
| Data and other attributes defined here:
|
| extension = '.xz'
|
| prefix = b'\xfd7zXZ'
|
| ----------------------------------------------------------------------
| Methods inherited from LZMACompressorWrapper:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| compressor_file(self, fileobj, compresslevel=None)
| Returns an instance of a compressor file object.
|
| decompressor_file(self, fileobj)
| Returns an instance of a decompressor file object.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from CompressorWrapper:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class ZlibCompressorWrapper(CompressorWrapper)
| Method resolution order:
| ZlibCompressorWrapper
| CompressorWrapper
| builtins.object
|
| Methods defined here:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from CompressorWrapper:
|
| compressor_file(self, fileobj, compresslevel=None)
| Returns an instance of a compressor file object.
|
| decompressor_file(self, fileobj)
| Returns an instance of a decompressor file object.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from CompressorWrapper:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
register_compressor(compressor_name, compressor, force=False)
Register a new compressor.
Parameters
----------
compressor_name: str.
The name of the compressor.
compressor: CompressorWrapper
An instance of a 'CompressorWrapper'.
DATA
LZ4_NOT_INSTALLED_ERROR = 'LZ4 is not installed. Install it with pip: ...
lz4 = None
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\compressor.py
joblib.cpu_count
Python Library Documentation: function cpu_count in module joblib.parallel
cpu_count(only_physical_cores=False)
Return the number of CPUs.
This delegates to loky.cpu_count that takes into account additional
constraints such as Linux CFS scheduler quotas (typically set by container
runtimes such as docker) and CPU affinity (for instance using the taskset
command on Linux).
If only_physical_cores is True, do not take hyperthreading / SMT logical
cores into account.
joblib.delayed
Python Library Documentation: function delayed in module joblib.parallel
delayed(function)
Decorator used to capture the arguments of a function.
joblib.disk
Python Library Documentation: module joblib.disk in joblib
NAME
joblib.disk - Disk management utilities.
FUNCTIONS
delete_folder(folder_path, onerror=None, allow_non_empty=True)
Utility function to cleanup a temporary folder if it still exists.
disk_used(path)
Return the disk usage in a directory.
memstr_to_bytes(text)
Convert a memory text to its value in bytes.
mkdirp(d)
Ensure directory d exists (like mkdir -p on Unix)
No guarantee that the directory is writable.
rm_subdirs(path, onerror=None)
Remove all subdirectories in this path.
The directory indicated by `path` is left in place, and its subdirectories
are erased.
If onerror is set, it is called to handle the error with arguments (func,
path, exc_info) where func is os.listdir, os.remove, or os.rmdir;
path is the argument to that function that caused it to fail; and
exc_info is a tuple returned by sys.exc_info(). If onerror is None,
an exception is raised.
DATA
RM_SUBDIRS_N_RETRY = 10
RM_SUBDIRS_RETRY_TIME = 0.1
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\disk.py
joblib.dump
Python Library Documentation: function dump in module joblib.numpy_pickle
dump(value, filename, compress=0, protocol=None, cache_size=None)
Persist an arbitrary Python object into one file.
Read more in the :ref:`User Guide <persistence>`.
Parameters
----------
value: any Python object
The object to store to disk.
filename: str, pathlib.Path, or file object.
The file object or path of the file in which it is to be stored.
The compression method corresponding to one of the supported filename
extensions ('.z', '.gz', '.bz2', '.xz' or '.lzma') will be used
automatically.
compress: int from 0 to 9 or bool or 2-tuple, optional
Optional compression level for the data. 0 or False is no compression.
Higher value means more compression, but also slower read and
write times. Using a value of 3 is often a good compromise.
See the notes for more details.
If compress is True, the compression level used is 3.
If compress is a 2-tuple, the first element must correspond to a string
between supported compressors (e.g 'zlib', 'gzip', 'bz2', 'lzma'
'xz'), the second element must be an integer from 0 to 9, corresponding
to the compression level.
protocol: int, optional
Pickle protocol, see pickle.dump documentation for more details.
cache_size: positive int, optional
This option is deprecated in 0.10 and has no effect.
Returns
-------
filenames: list of strings
The list of file names in which the data is stored. If
compress is false, each array is stored in a different file.
See Also
--------
joblib.load : corresponding loader
Notes
-----
Memmapping on load cannot be used for compressed files. Thus
using compression can significantly slow down loading. In
addition, compressed files take up extra memory during
dump and load.
joblib.effective_n_jobs
Python Library Documentation: function effective_n_jobs in module joblib.parallel
effective_n_jobs(n_jobs=-1)
Determine the number of jobs that can actually run in parallel
n_jobs is the number of workers requested by the callers. Passing n_jobs=-1
means requesting all available workers for instance matching the number of
CPU cores on the worker host(s).
This method should return a guesstimate of the number of workers that can
actually perform work concurrently with the currently enabled default
backend. The primary use case is to make it possible for the caller to know
in how many chunks to slice the work.
In general working on larger data chunks is more efficient (less scheduling
overhead and better use of CPU cache prefetching heuristics) as long as all
the workers have enough work to do.
Warning: this function is experimental and subject to change in a future
version of joblib.
.. versionadded:: 0.10
joblib.executor
Python Library Documentation: module joblib.executor in joblib
NAME
joblib.executor - Utility function to construct a loky.ReusableExecutor with custom pickler.
DESCRIPTION
This module provides efficient ways of working with data stored in
shared memory with numpy.memmap arrays without inducing any memory
copy between the parent and child processes.
CLASSES
joblib.externals.loky.reusable_executor._ReusablePoolExecutor(joblib.externals.loky.process_executor.ProcessPoolExecutor)
MemmappingExecutor
class MemmappingExecutor(joblib.externals.loky.reusable_executor._ReusablePoolExecutor)
| MemmappingExecutor(submit_resize_lock, max_workers=None, context=None, timeout=None, executor_id=0, job_reducers=None, result_reducers=None, initializer=None, initargs=(), env=None)
|
| Method resolution order:
| MemmappingExecutor
| joblib.externals.loky.reusable_executor._ReusablePoolExecutor
| joblib.externals.loky.process_executor.ProcessPoolExecutor
| concurrent.futures._base.Executor
| builtins.object
|
| Methods defined here:
|
| terminate(self, kill_workers=False)
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| get_memmapping_executor(n_jobs, timeout=300, initializer=None, initargs=(), env=None, temp_folder=None, context_id=None, **backend_args)
| Factory for ReusableExecutor with automatic memmapping for large
| numpy arrays.
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.externals.loky.reusable_executor._ReusablePoolExecutor:
|
| __init__(self, submit_resize_lock, max_workers=None, context=None, timeout=None, executor_id=0, job_reducers=None, result_reducers=None, initializer=None, initargs=(), env=None)
| Initializes a new ProcessPoolExecutor instance.
|
| Args:
| max_workers: int, optional (default: cpu_count())
| The maximum number of processes that can be used to execute the
| given calls. If None or not given then as many worker processes
| will be created as the number of CPUs the current process
| can use.
| job_reducers, result_reducers: dict(type: reducer_func)
| Custom reducer for pickling the jobs and the results from the
| Executor. If only `job_reducers` is provided, `result_reducer`
| will use the same reducers
| timeout: int, optional (default: None)
| Idle workers exit after timeout seconds. If a new job is
| submitted after the timeout, the executor will start enough
| new Python processes to make sure the pool of workers is full.
| context: A multiprocessing context to launch the workers. This
| object should provide SimpleQueue, Queue and Process.
| initializer: An callable used to initialize worker processes.
| initargs: A tuple of arguments to pass to the initializer.
| env: A dict of environment variable to overwrite in the child
| process. The environment variables are set before any module is
| loaded. Note that this only works with the loky context.
|
| submit(self, fn, *args, **kwargs)
| Submits a callable to be executed with the given arguments.
|
| Schedules the callable to be executed as fn(*args, **kwargs) and returns
| a Future instance representing the execution of the callable.
|
| Returns:
| A Future representing the given call.
|
| ----------------------------------------------------------------------
| Class methods inherited from joblib.externals.loky.reusable_executor._ReusablePoolExecutor:
|
| get_reusable_executor(max_workers=None, context=None, timeout=10, kill_workers=False, reuse='auto', job_reducers=None, result_reducers=None, initializer=None, initargs=(), env=None)
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.externals.loky.process_executor.ProcessPoolExecutor:
|
| map(self, fn, *iterables, **kwargs)
| Returns an iterator equivalent to map(fn, iter).
|
| Args:
| fn: A callable that will take as many arguments as there are
| passed iterables.
| timeout: The maximum number of seconds to wait. If None, then there
| is no limit on the wait time.
| chunksize: If greater than one, the iterables will be chopped into
| chunks of size chunksize and submitted to the process pool.
| If set to one, the items in the list will be sent one at a
| time.
|
| Returns:
| An iterator equivalent to: map(func, *iterables) but the calls may
| be evaluated out-of-order.
|
| Raises:
| TimeoutError: If the entire result iterator could not be generated
| before the given timeout.
| Exception: If fn(*args) raises for any values.
|
| shutdown(self, wait=True, kill_workers=False)
| Clean-up the resources associated with the Executor.
|
| It is safe to call this method several times. Otherwise, no other
| methods can be called after this one.
|
| Args:
| wait: If True then shutdown will not return until all running
| futures have finished executing and the resources used by the
| executor have been reclaimed.
| cancel_futures: If True then shutdown will cancel all pending
| futures. Futures that are completed or running will not be
| cancelled.
|
| ----------------------------------------------------------------------
| Methods inherited from concurrent.futures._base.Executor:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from concurrent.futures._base.Executor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
get_memmapping_executor(n_jobs, **kwargs)
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\executor.py
joblib.expires_after
Python Library Documentation: function expires_after in module joblib.memory
expires_after(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0)
Helper cache_validation_callback to force recompute after a duration.
Parameters
----------
days, seconds, microseconds, milliseconds, minutes, hours, weeks: numbers
argument passed to a timedelta.
joblib.externals
Python Library Documentation: package joblib.externals in joblib
NAME
joblib.externals
PACKAGE CONTENTS
cloudpickle (package)
loky (package)
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\externals\__init__.py
joblib.func_inspect
Python Library Documentation: module joblib.func_inspect in joblib
NAME
joblib.func_inspect - My own variation on function-specific inspect-like features.
CLASSES
builtins.tuple(builtins.object)
FullArgSpec
full_argspec_type = class FullArgSpec(builtins.tuple)
| full_argspec_type(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations)
|
| FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations)
|
| Method resolution order:
| FullArgSpec
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __getnewargs__(self) from collections.FullArgSpec
| Return self as a plain tuple. Used by copy and pickle.
|
| __repr__(self) from collections.FullArgSpec
| Return a nicely formatted representation string
|
| _asdict(self) from collections.FullArgSpec
| Return a new dict which maps field names to their values.
|
| _replace(self, /, **kwds) from collections.FullArgSpec
| Return a new FullArgSpec object replacing specified fields with new values
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| _make(iterable) from collections.FullArgSpec
| Make a new FullArgSpec object from a sequence or iterable
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(_cls, args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations) from namedtuple_FullArgSpec.FullArgSpec
| Create new instance of FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| args
| Alias for field number 0
|
| varargs
| Alias for field number 1
|
| varkw
| Alias for field number 2
|
| defaults
| Alias for field number 3
|
| kwonlyargs
| Alias for field number 4
|
| kwonlydefaults
| Alias for field number 5
|
| annotations
| Alias for field number 6
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('args', 'varargs', 'varkw', 'defaults', 'kwonlyargs'...
|
| _field_defaults = {}
|
| _fields = ('args', 'varargs', 'varkw', 'defaults', 'kwonlyargs', 'kwon...
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
FUNCTIONS
filter_args(func, ignore_lst, args=(), kwargs={})
Filters the given args and kwargs using a list of arguments to
ignore, and a function specification.
Parameters
----------
func: callable
Function giving the argument specification
ignore_lst: list of strings
List of arguments to ignore (either a name of an argument
in the function spec, or '*', or '**')
*args: list
Positional arguments passed to the function.
**kwargs: dict
Keyword arguments passed to the function
Returns
-------
filtered_args: list
List of filtered positional and keyword arguments.
format_call(func, args, kwargs, object_name='Memory')
Returns a nicely formatted statement displaying the function
call with the given arguments.
format_signature(func, *args, **kwargs)
get_func_code(func)
Attempts to retrieve a reliable function code hash.
The reason we don't use inspect.getsource is that it caches the
source, whereas we want this to be modified on the fly when the
function is modified.
Returns
-------
func_code: string
The function code
source_file: string
The path to the file in which the function is defined.
first_line: int
The first line of the code in the source file.
Notes
------
This function does a bit more magic than inspect, and is thus
more robust.
get_func_name(func, resolv_alias=True, win_characters=True)
Return the function import path (as a list of module names), and
a name for the function.
Parameters
----------
func: callable
The func to inspect
resolv_alias: boolean, optional
If true, possible local aliases are indicated.
win_characters: boolean, optional
If true, substitute special characters using urllib.quote
This is useful in Windows, as it cannot encode some filenames
DATA
full_argspec_fields = 'args varargs varkw defaults kwonlyargs kwonlyde...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\func_inspect.py
joblib.hash
Python Library Documentation: function hash in module joblib.hashing
hash(obj, hash_name='md5', coerce_mmap=False)
Quick calculation of a hash to identify uniquely Python objects
containing numpy arrays.
Parameters
----------
hash_name: 'md5' or 'sha1'
Hashing algorithm used. sha1 is supposedly safer, but md5 is
faster.
coerce_mmap: boolean
Make no difference between np.memmap and np.ndarray
joblib.hashing
Python Library Documentation: module joblib.hashing in joblib
NAME
joblib.hashing
DESCRIPTION
Fast cryptographic hash of Python objects, with a special case for fast
hashing of numpy arrays.
CLASSES
pickle._Pickler(builtins.object)
Hasher
NumpyHasher
class Hasher(pickle._Pickler)
| Hasher(hash_name='md5')
|
| A subclass of pickler, to do cryptographic hashing, rather than
| pickling.
|
| Method resolution order:
| Hasher
| pickle._Pickler
| builtins.object
|
| Methods defined here:
|
| __init__(self, hash_name='md5')
| This takes a binary file for writing a pickle data stream.
|
| The optional *protocol* argument tells the pickler to use the
| given protocol; supported protocols are 0, 1, 2, 3, 4 and 5.
| The default protocol is 4. It was introduced in Python 3.4, and
| is incompatible with previous versions.
|
| Specifying a negative protocol version selects the highest
| protocol version supported. The higher the protocol used, the
| more recent the version of Python needed to read the pickle
| produced.
|
| The *file* argument must have a write() method that accepts a
| single bytes argument. It can thus be a file object opened for
| binary writing, an io.BytesIO instance, or any other custom
| object that meets this interface.
|
| If *fix_imports* is True and *protocol* is less than 3, pickle
| will try to map the new Python 3 names to the old module names
| used in Python 2, so that the pickle data stream is readable
| with Python 2.
|
| If *buffer_callback* is None (the default), buffer views are
| serialized into *file* as part of the pickle stream.
|
| If *buffer_callback* is not None, then it can be called any number
| of times with a buffer view. If the callback returns a false value
| (such as None), the given buffer is out-of-band; otherwise the
| buffer is serialized in-band, i.e. inside the pickle stream.
|
| It is an error if *buffer_callback* is not None and *protocol*
| is None or smaller than 5.
|
| hash(self, obj, return_digest=True)
|
| memoize(self, obj)
| Store an object in the memo.
|
| save(self, obj)
|
| save_global(self, obj, name=None, pack=<built-in function pack>)
| # The dispatch table of the pickler is not accessible in Python
| # 3, as these lines are only bugware for IPython, we skip them.
|
| save_set(self, set_items)
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| dispatch = {<class 'NoneType'>: <function _Pickler.save_none>, <class ...
|
| ----------------------------------------------------------------------
| Methods inherited from pickle._Pickler:
|
| clear_memo(self)
| Clears the pickler's "memo".
|
| The memo is the data structure that remembers which objects the
| pickler has already seen, so that shared or recursive objects
| are pickled by reference and not by value. This method is
| useful when re-using picklers.
|
| dump(self, obj)
| Write a pickled representation of obj to the open file.
|
| get(self, i)
| # Return a GET (BINGET, LONG_BINGET) opcode string, with argument i.
|
| persistent_id(self, obj)
|
| put(self, idx)
| # Return a PUT (BINPUT, LONG_BINPUT) opcode string, with argument i.
|
| save_bool(self, obj)
|
| save_bytearray(self, obj)
|
| save_bytes(self, obj)
|
| save_dict(self, obj)
|
| save_float(self, obj)
|
| save_frozenset(self, obj)
|
| save_list(self, obj)
|
| save_long(self, obj)
|
| save_none(self, obj)
|
| save_pers(self, pid)
|
| save_picklebuffer(self, obj)
|
| save_reduce(self, func, args, state=None, listitems=None, dictitems=None, state_setter=None, *, obj=None)
|
| save_str(self, obj)
|
| save_tuple(self, obj)
|
| save_type(self, obj)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from pickle._Pickler:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class NumpyHasher(Hasher)
| NumpyHasher(hash_name='md5', coerce_mmap=False)
|
| Special case the hasher for when numpy is loaded.
|
| Method resolution order:
| NumpyHasher
| Hasher
| pickle._Pickler
| builtins.object
|
| Methods defined here:
|
| __init__(self, hash_name='md5', coerce_mmap=False)
| Parameters
| ----------
| hash_name: string
| The hash algorithm to be used
| coerce_mmap: boolean
| Make no difference between np.memmap and np.ndarray
| objects.
|
| save(self, obj)
| Subclass the save method, to hash ndarray subclass, rather
| than pickling them. Off course, this is a total abuse of
| the Pickler class.
|
| ----------------------------------------------------------------------
| Methods inherited from Hasher:
|
| hash(self, obj, return_digest=True)
|
| memoize(self, obj)
| Store an object in the memo.
|
| save_global(self, obj, name=None, pack=<built-in function pack>)
| # The dispatch table of the pickler is not accessible in Python
| # 3, as these lines are only bugware for IPython, we skip them.
|
| save_set(self, set_items)
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from Hasher:
|
| dispatch = {<class 'NoneType'>: <function _Pickler.save_none>, <class ...
|
| ----------------------------------------------------------------------
| Methods inherited from pickle._Pickler:
|
| clear_memo(self)
| Clears the pickler's "memo".
|
| The memo is the data structure that remembers which objects the
| pickler has already seen, so that shared or recursive objects
| are pickled by reference and not by value. This method is
| useful when re-using picklers.
|
| dump(self, obj)
| Write a pickled representation of obj to the open file.
|
| get(self, i)
| # Return a GET (BINGET, LONG_BINGET) opcode string, with argument i.
|
| persistent_id(self, obj)
|
| put(self, idx)
| # Return a PUT (BINPUT, LONG_BINPUT) opcode string, with argument i.
|
| save_bool(self, obj)
|
| save_bytearray(self, obj)
|
| save_bytes(self, obj)
|
| save_dict(self, obj)
|
| save_float(self, obj)
|
| save_frozenset(self, obj)
|
| save_list(self, obj)
|
| save_long(self, obj)
|
| save_none(self, obj)
|
| save_pers(self, pid)
|
| save_picklebuffer(self, obj)
|
| save_reduce(self, func, args, state=None, listitems=None, dictitems=None, state_setter=None, *, obj=None)
|
| save_str(self, obj)
|
| save_tuple(self, obj)
|
| save_type(self, obj)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from pickle._Pickler:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
hash(obj, hash_name='md5', coerce_mmap=False)
Quick calculation of a hash to identify uniquely Python objects
containing numpy arrays.
Parameters
----------
hash_name: 'md5' or 'sha1'
Hashing algorithm used. sha1 is supposedly safer, but md5 is
faster.
coerce_mmap: boolean
Make no difference between np.memmap and np.ndarray
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\hashing.py
joblib.load
Python Library Documentation: function load in module joblib.numpy_pickle
load(filename, mmap_mode=None)
Reconstruct a Python object from a file persisted with joblib.dump.
Read more in the :ref:`User Guide <persistence>`.
WARNING: joblib.load relies on the pickle module and can therefore
execute arbitrary Python code. It should therefore never be used
to load files from untrusted sources.
Parameters
----------
filename: str, pathlib.Path, or file object.
The file object or path of the file from which to load the object
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
If not None, the arrays are memory-mapped from the disk. This
mode has no effect for compressed files. Note that in this
case the reconstructed object might no longer match exactly
the originally pickled object.
Returns
-------
result: any Python object
The object stored in the file.
See Also
--------
joblib.dump : function to save an object
Notes
-----
This function can load numpy array files saved separately during the
dump. If the mmap_mode argument is given, it is passed to np.load and
arrays are loaded as memmaps. As a consequence, the reconstructed
object might not match the original pickled object. Note that if the
file was saved with compression, the arrays cannot be memmapped.
joblib.logger
Python Library Documentation: module joblib.logger in joblib
NAME
joblib.logger - Helpers for logging.
DESCRIPTION
This module needs much love to become useful.
CLASSES
builtins.object
Logger
PrintTime
class Logger(builtins.object)
| Logger(depth=3, name=None)
|
| Base class for logging messages.
|
| Methods defined here:
|
| __init__(self, depth=3, name=None)
| Parameters
| ----------
| depth: int, optional
| The depth of objects printed.
| name: str, optional
| The namespace to log to. If None, defaults to joblib.
|
| debug(self, msg)
|
| format(self, obj, indent=0)
| Return the formatted representation of the object.
|
| info(self, msg)
|
| warn(self, msg)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class PrintTime(builtins.object)
| PrintTime(logfile=None, logdir=None)
|
| Print and log messages while keeping track of time.
|
| Methods defined here:
|
| __call__(self, msg='', total=False)
| Print the time elapsed between the last call and the current
| call, with an optional message.
|
| __init__(self, logfile=None, logdir=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
format_time(t)
pformat(obj, indent=0, depth=3)
short_format_time(t)
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\logger.py
joblib.memory
Python Library Documentation: module joblib.memory in joblib
NAME
joblib.memory
DESCRIPTION
A context object for caching a function's return value each time it
is called with the same input arguments.
CLASSES
builtins.UserWarning(builtins.Warning)
JobLibCollisionWarning
builtins.object
NotMemorizedFunc
AsyncNotMemorizedFunc
NotMemorizedResult
joblib.logger.Logger(builtins.object)
MemorizedFunc
AsyncMemorizedFunc
MemorizedResult
Memory
class AsyncMemorizedFunc(MemorizedFunc)
| AsyncMemorizedFunc(func, location, backend='local', ignore=None, mmap_mode=None, compress=False, verbose=1, timestamp=None, cache_validation_callback=None)
|
| ###############################################################################
| # class `AsyncMemorizedFunc`
| ###############################################################################
|
| Method resolution order:
| AsyncMemorizedFunc
| MemorizedFunc
| joblib.logger.Logger
| builtins.object
|
| Methods defined here:
|
| async __call__(self, *args, **kwargs)
| Call self as a function.
|
| async call(self, *args, **kwargs)
| Force the execution of the function with the given arguments.
|
| The output values will be persisted, i.e., the cache will be updated
| with any new values.
|
| Parameters
| ----------
| *args: arguments
| The arguments.
| **kwargs: keyword arguments
| Keyword arguments.
|
| Returns
| -------
| output : object
| The output of the function call.
| metadata : dict
| The metadata associated with the call.
|
| async call_and_shelve(self, *args, **kwargs)
| Call wrapped function, cache result and return a reference.
|
| This method returns a reference to the cached result instead of the
| result itself. The reference object is small and pickeable, allowing
| to send or store it easily. Call .get() on reference object to get
| result.
|
| Returns
| -------
| cached_result: MemorizedResult or NotMemorizedResult
| reference to the value returned by the wrapped function. The
| class "NotMemorizedResult" is used when there is no cache
| activated (e.g. location=None in Memory).
|
| ----------------------------------------------------------------------
| Methods inherited from MemorizedFunc:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, func, location, backend='local', ignore=None, mmap_mode=None, compress=False, verbose=1, timestamp=None, cache_validation_callback=None)
| Parameters
| ----------
| depth: int, optional
| The depth of objects printed.
| name: str, optional
| The namespace to log to. If None, defaults to joblib.
|
| __repr__(self)
| Return repr(self).
|
| check_call_in_cache(self, *args, **kwargs)
| Check if function call is in the memory cache.
|
| Does not call the function or do any work besides func inspection
| and arg hashing.
|
| Returns
| -------
| is_call_in_cache: bool
| Whether or not the result of the function has been cached
| for the input arguments that have been passed.
|
| clear(self, warn=True)
| Empty the function's cache.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from MemorizedFunc:
|
| func_code_info
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.logger.Logger:
|
| debug(self, msg)
|
| format(self, obj, indent=0)
| Return the formatted representation of the object.
|
| info(self, msg)
|
| warn(self, msg)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from joblib.logger.Logger:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class AsyncNotMemorizedFunc(NotMemorizedFunc)
| AsyncNotMemorizedFunc(func)
|
| ###############################################################################
| # class `AsyncNotMemorizedFunc`
| ###############################################################################
|
| Method resolution order:
| AsyncNotMemorizedFunc
| NotMemorizedFunc
| builtins.object
|
| Methods defined here:
|
| async call_and_shelve(self, *args, **kwargs)
|
| ----------------------------------------------------------------------
| Methods inherited from NotMemorizedFunc:
|
| __call__(self, *args, **kwargs)
| Call self as a function.
|
| __init__(self, func)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| call(self, *args, **kwargs)
|
| check_call_in_cache(self, *args, **kwargs)
|
| clear(self, warn=True)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from NotMemorizedFunc:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class JobLibCollisionWarning(builtins.UserWarning)
| Warn that there might be a collision between names of functions.
|
| Method resolution order:
| JobLibCollisionWarning
| builtins.UserWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.UserWarning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.UserWarning:
|
| __new__(*args, **kwargs) class method of builtins.UserWarning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class MemorizedFunc(joblib.logger.Logger)
| MemorizedFunc(func, location, backend='local', ignore=None, mmap_mode=None, compress=False, verbose=1, timestamp=None, cache_validation_callback=None)
|
| Callable object decorating a function for caching its return value
| each time it is called.
|
| Methods are provided to inspect the cache or clean it.
|
| Attributes
| ----------
| func: callable
| The original, undecorated, function.
|
| location: string
| The location of joblib cache. Depends on the store backend used.
|
| backend: str
| Type of store backend for reading/writing cache files.
| Default is 'local', in which case the location is the path to a
| disk storage.
|
| ignore: list or None
| List of variable names to ignore when choosing whether to
| recompute.
|
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
| The memmapping mode used when loading from cache
| numpy arrays. See numpy.load for the meaning of the different
| values.
|
| compress: boolean, or integer
| Whether to zip the stored data on disk. If an integer is
| given, it should be between 1 and 9, and sets the amount
| of compression. Note that compressed arrays cannot be
| read by memmapping.
|
| verbose: int, optional
| The verbosity flag, controls messages that are issued as
| the function is evaluated.
|
| cache_validation_callback: callable, optional
| Callable to check if a result in cache is valid or is to be recomputed.
| When the function is called with arguments for which a cache exists,
| the callback is called with the cache entry's metadata as its sole
| argument. If it returns True, the cached result is returned, else the
| cache for these arguments is cleared and the result is recomputed.
|
| Method resolution order:
| MemorizedFunc
| joblib.logger.Logger
| builtins.object
|
| Methods defined here:
|
| __call__(self, *args, **kwargs)
| Call self as a function.
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, func, location, backend='local', ignore=None, mmap_mode=None, compress=False, verbose=1, timestamp=None, cache_validation_callback=None)
| Parameters
| ----------
| depth: int, optional
| The depth of objects printed.
| name: str, optional
| The namespace to log to. If None, defaults to joblib.
|
| __repr__(self)
| Return repr(self).
|
| call(self, *args, **kwargs)
| Force the execution of the function with the given arguments.
|
| The output values will be persisted, i.e., the cache will be updated
| with any new values.
|
| Parameters
| ----------
| *args: arguments
| The arguments.
| **kwargs: keyword arguments
| Keyword arguments.
|
| Returns
| -------
| output : object
| The output of the function call.
| metadata : dict
| The metadata associated with the call.
|
| call_and_shelve(self, *args, **kwargs)
| Call wrapped function, cache result and return a reference.
|
| This method returns a reference to the cached result instead of the
| result itself. The reference object is small and pickeable, allowing
| to send or store it easily. Call .get() on reference object to get
| result.
|
| Returns
| -------
| cached_result: MemorizedResult or NotMemorizedResult
| reference to the value returned by the wrapped function. The
| class "NotMemorizedResult" is used when there is no cache
| activated (e.g. location=None in Memory).
|
| check_call_in_cache(self, *args, **kwargs)
| Check if function call is in the memory cache.
|
| Does not call the function or do any work besides func inspection
| and arg hashing.
|
| Returns
| -------
| is_call_in_cache: bool
| Whether or not the result of the function has been cached
| for the input arguments that have been passed.
|
| clear(self, warn=True)
| Empty the function's cache.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| func_code_info
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.logger.Logger:
|
| debug(self, msg)
|
| format(self, obj, indent=0)
| Return the formatted representation of the object.
|
| info(self, msg)
|
| warn(self, msg)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from joblib.logger.Logger:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class MemorizedResult(joblib.logger.Logger)
| MemorizedResult(location, call_id, backend='local', mmap_mode=None, verbose=0, timestamp=None, metadata=None)
|
| Object representing a cached value.
|
| Attributes
| ----------
| location: str
| The location of joblib cache. Depends on the store backend used.
|
| func: function or str
| function whose output is cached. The string case is intended only for
| instantiation based on the output of repr() on another instance.
| (namely eval(repr(memorized_instance)) works).
|
| argument_hash: str
| hash of the function arguments.
|
| backend: str
| Type of store backend for reading/writing cache files.
| Default is 'local'.
|
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
| The memmapping mode used when loading from cache numpy arrays. See
| numpy.load for the meaning of the different values.
|
| verbose: int
| verbosity level (0 means no message).
|
| timestamp, metadata: string
| for internal use only.
|
| Method resolution order:
| MemorizedResult
| joblib.logger.Logger
| builtins.object
|
| Methods defined here:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, location, call_id, backend='local', mmap_mode=None, verbose=0, timestamp=None, metadata=None)
| Parameters
| ----------
| depth: int, optional
| The depth of objects printed.
| name: str, optional
| The namespace to log to. If None, defaults to joblib.
|
| __repr__(self)
| Return repr(self).
|
| clear(self)
| Clear value from cache
|
| get(self)
| Read value from cache and return it.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| args_id
|
| argument_hash
|
| func
|
| func_id
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.logger.Logger:
|
| debug(self, msg)
|
| format(self, obj, indent=0)
| Return the formatted representation of the object.
|
| info(self, msg)
|
| warn(self, msg)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from joblib.logger.Logger:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Memory(joblib.logger.Logger)
| Memory(location=None, backend='local', mmap_mode=None, compress=False, verbose=1, bytes_limit=None, backend_options=None)
|
| A context object for caching a function's return value each time it
| is called with the same input arguments.
|
| All values are cached on the filesystem, in a deep directory
| structure.
|
| Read more in the :ref:`User Guide <memory>`.
|
| Parameters
| ----------
| location: str, pathlib.Path or None
| The path of the base directory to use as a data store
| or None. If None is given, no caching is done and
| the Memory object is completely transparent. This option
| replaces cachedir since version 0.12.
|
| backend: str, optional
| Type of store backend for reading/writing cache files.
| Default: 'local'.
| The 'local' backend is using regular filesystem operations to
| manipulate data (open, mv, etc) in the backend.
|
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
| The memmapping mode used when loading from cache
| numpy arrays. See numpy.load for the meaning of the
| arguments.
|
| compress: boolean, or integer, optional
| Whether to zip the stored data on disk. If an integer is
| given, it should be between 1 and 9, and sets the amount
| of compression. Note that compressed arrays cannot be
| read by memmapping.
|
| verbose: int, optional
| Verbosity flag, controls the debug messages that are issued
| as functions are evaluated.
|
| bytes_limit: int | str, optional
| Limit in bytes of the size of the cache. By default, the size of
| the cache is unlimited. When reducing the size of the cache,
| ``joblib`` keeps the most recently accessed items first. If a
| str is passed, it is converted to a number of bytes using units
| { K | M | G} for kilo, mega, giga.
|
| **Note:** You need to call :meth:`joblib.Memory.reduce_size` to
| actually reduce the cache size to be less than ``bytes_limit``.
|
| **Note:** This argument has been deprecated. One should give the
| value of ``bytes_limit`` directly in
| :meth:`joblib.Memory.reduce_size`.
|
| backend_options: dict, optional
| Contains a dictionary of named parameters used to configure
| the store backend.
|
| Method resolution order:
| Memory
| joblib.logger.Logger
| builtins.object
|
| Methods defined here:
|
| __getstate__(self)
| We don't store the timestamp when pickling, to avoid the hash
| depending from it.
|
| __init__(self, location=None, backend='local', mmap_mode=None, compress=False, verbose=1, bytes_limit=None, backend_options=None)
| Parameters
| ----------
| depth: int, optional
| The depth of objects printed.
| name: str, optional
| The namespace to log to. If None, defaults to joblib.
|
| __repr__(self)
| Return repr(self).
|
| cache(self, func=None, ignore=None, verbose=None, mmap_mode=False, cache_validation_callback=None)
| Decorates the given function func to only compute its return
| value for input arguments not cached on disk.
|
| Parameters
| ----------
| func: callable, optional
| The function to be decorated
| ignore: list of strings
| A list of arguments name to ignore in the hashing
| verbose: integer, optional
| The verbosity mode of the function. By default that
| of the memory object is used.
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
| The memmapping mode used when loading from cache
| numpy arrays. See numpy.load for the meaning of the
| arguments. By default that of the memory object is used.
| cache_validation_callback: callable, optional
| Callable to validate whether or not the cache is valid. When
| the cached function is called with arguments for which a cache
| exists, this callable is called with the metadata of the cached
| result as its sole argument. If it returns True, then the
| cached result is returned, else the cache for these arguments
| is cleared and recomputed.
|
| Returns
| -------
| decorated_func: MemorizedFunc object
| The returned object is a MemorizedFunc object, that is
| callable (behaves like a function), but offers extra
| methods for cache lookup and management. See the
| documentation for :class:`joblib.memory.MemorizedFunc`.
|
| clear(self, warn=True)
| Erase the complete cache directory.
|
| eval(self, func, *args, **kwargs)
| Eval function func with arguments `*args` and `**kwargs`,
| in the context of the memory.
|
| This method works similarly to the builtin `apply`, except
| that the function is called only if the cache is not
| up to date.
|
| reduce_size(self, bytes_limit=None, items_limit=None, age_limit=None)
| Remove cache elements to make the cache fit its limits.
|
| The limitation can impose that the cache size fits in ``bytes_limit``,
| that the number of cache items is no more than ``items_limit``, and
| that all files in cache are not older than ``age_limit``.
|
| Parameters
| ----------
| bytes_limit: int | str, optional
| Limit in bytes of the size of the cache. By default, the size of
| the cache is unlimited. When reducing the size of the cache,
| ``joblib`` keeps the most recently accessed items first. If a
| str is passed, it is converted to a number of bytes using units
| { K | M | G} for kilo, mega, giga.
|
| items_limit: int, optional
| Number of items to limit the cache to. By default, the number of
| items in the cache is unlimited. When reducing the size of the
| cache, ``joblib`` keeps the most recently accessed items first.
|
| age_limit: datetime.timedelta, optional
| Maximum age of items to limit the cache to. When reducing the size
| of the cache, any items last accessed more than the given length of
| time ago are deleted.
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.logger.Logger:
|
| debug(self, msg)
|
| format(self, obj, indent=0)
| Return the formatted representation of the object.
|
| info(self, msg)
|
| warn(self, msg)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from joblib.logger.Logger:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class NotMemorizedFunc(builtins.object)
| NotMemorizedFunc(func)
|
| No-op object decorating a function.
|
| This class replaces MemorizedFunc when there is no cache. It provides an
| identical API but does not write anything on disk.
|
| Attributes
| ----------
| func: callable
| Original undecorated function.
|
| Methods defined here:
|
| __call__(self, *args, **kwargs)
| Call self as a function.
|
| __init__(self, func)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| call(self, *args, **kwargs)
|
| call_and_shelve(self, *args, **kwargs)
|
| check_call_in_cache(self, *args, **kwargs)
|
| clear(self, warn=True)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class NotMemorizedResult(builtins.object)
| NotMemorizedResult(value)
|
| Class representing an arbitrary value.
|
| This class is a replacement for MemorizedResult when there is no cache.
|
| Methods defined here:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, value)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| __setstate__(self, state)
|
| clear(self)
|
| get(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| valid
|
| value
FUNCTIONS
expires_after(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0)
Helper cache_validation_callback to force recompute after a duration.
Parameters
----------
days, seconds, microseconds, milliseconds, minutes, hours, weeks: numbers
argument passed to a timedelta.
extract_first_line(func_code)
Extract the first line information from the function code
text if available.
register_store_backend(backend_name, backend)
Extend available store backends.
The Memory, MemorizeResult and MemorizeFunc objects are designed to be
agnostic to the type of store used behind. By default, the local file
system is used but this function gives the possibility to extend joblib's
memory pattern with other types of storage such as cloud storage (S3, GCS,
OpenStack, HadoopFS, etc) or blob DBs.
Parameters
----------
backend_name: str
The name identifying the store backend being registered. For example,
'local' is used with FileSystemStoreBackend.
backend: StoreBackendBase subclass
The name of a class that implements the StoreBackendBase interface.
DATA
FIRST_LINE_TEXT = '# first line:'
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\memory.py
joblib.numpy_pickle
Python Library Documentation: module joblib.numpy_pickle in joblib
NAME
joblib.numpy_pickle - Utilities for fast persistence of big data, with optional compression.
CLASSES
builtins.object
NumpyArrayWrapper
pickle._Pickler(builtins.object)
NumpyPickler
pickle._Unpickler(builtins.object)
NumpyUnpickler
class NumpyArrayWrapper(builtins.object)
| NumpyArrayWrapper(subclass, shape, order, dtype, allow_mmap=False, numpy_array_alignment_bytes=16)
|
| An object to be persisted instead of numpy arrays.
|
| This object is used to hack into the pickle machinery and read numpy
| array data from our custom persistence format.
| More precisely, this object is used for:
| * carrying the information of the persisted array: subclass, shape, order,
| dtype. Those ndarray metadata are used to correctly reconstruct the array
| with low level numpy functions.
| * determining if memmap is allowed on the array.
| * reading the array bytes from a file.
| * reading the array using memorymap from a file.
| * writing the array bytes to a file.
|
| Attributes
| ----------
| subclass: numpy.ndarray subclass
| Determine the subclass of the wrapped array.
| shape: numpy.ndarray shape
| Determine the shape of the wrapped array.
| order: {'C', 'F'}
| Determine the order of wrapped array data. 'C' is for C order, 'F' is
| for fortran order.
| dtype: numpy.ndarray dtype
| Determine the data type of the wrapped array.
| allow_mmap: bool
| Determine if memory mapping is allowed on the wrapped array.
| Default: False.
|
| Methods defined here:
|
| __init__(self, subclass, shape, order, dtype, allow_mmap=False, numpy_array_alignment_bytes=16)
| Constructor. Store the useful information for later.
|
| read(self, unpickler)
| Read the array corresponding to this wrapper.
|
| Use the unpickler to get all information to correctly read the array.
|
| Parameters
| ----------
| unpickler: NumpyUnpickler
|
| Returns
| -------
| array: numpy.ndarray
|
| read_array(self, unpickler)
| Read array from unpickler file handle.
|
| This function is an adaptation of the numpy read_array function
| available in version 1.10.1 in numpy/lib/format.py.
|
| read_mmap(self, unpickler)
| Read an array using numpy memmap.
|
| safe_get_numpy_array_alignment_bytes(self)
|
| write_array(self, array, pickler)
| Write array bytes to pickler file handle.
|
| This function is an adaptation of the numpy write_array function
| available in version 1.10.1 in numpy/lib/format.py.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class NumpyPickler(pickle._Pickler)
| NumpyPickler(fp, protocol=None)
|
| A pickler to persist big data efficiently.
|
| The main features of this object are:
| * persistence of numpy arrays in a single file.
| * optional compression with a special care on avoiding memory copies.
|
| Attributes
| ----------
| fp: file
| File object handle used for serializing the input object.
| protocol: int, optional
| Pickle protocol used. Default is pickle.DEFAULT_PROTOCOL.
|
| Method resolution order:
| NumpyPickler
| pickle._Pickler
| builtins.object
|
| Methods defined here:
|
| __init__(self, fp, protocol=None)
| This takes a binary file for writing a pickle data stream.
|
| The optional *protocol* argument tells the pickler to use the
| given protocol; supported protocols are 0, 1, 2, 3, 4 and 5.
| The default protocol is 4. It was introduced in Python 3.4, and
| is incompatible with previous versions.
|
| Specifying a negative protocol version selects the highest
| protocol version supported. The higher the protocol used, the
| more recent the version of Python needed to read the pickle
| produced.
|
| The *file* argument must have a write() method that accepts a
| single bytes argument. It can thus be a file object opened for
| binary writing, an io.BytesIO instance, or any other custom
| object that meets this interface.
|
| If *fix_imports* is True and *protocol* is less than 3, pickle
| will try to map the new Python 3 names to the old module names
| used in Python 2, so that the pickle data stream is readable
| with Python 2.
|
| If *buffer_callback* is None (the default), buffer views are
| serialized into *file* as part of the pickle stream.
|
| If *buffer_callback* is not None, then it can be called any number
| of times with a buffer view. If the callback returns a false value
| (such as None), the given buffer is out-of-band; otherwise the
| buffer is serialized in-band, i.e. inside the pickle stream.
|
| It is an error if *buffer_callback* is not None and *protocol*
| is None or smaller than 5.
|
| save(self, obj)
| Subclass the Pickler `save` method.
|
| This is a total abuse of the Pickler class in order to use the numpy
| persistence function `save` instead of the default pickle
| implementation. The numpy array is replaced by a custom wrapper in the
| pickle persistence stack and the serialized array is written right
| after in the file. Warning: the file produced does not follow the
| pickle format. As such it can not be read with `pickle.load`.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| dispatch = {<class 'NoneType'>: <function _Pickler.save_none>, <class ...
|
| ----------------------------------------------------------------------
| Methods inherited from pickle._Pickler:
|
| clear_memo(self)
| Clears the pickler's "memo".
|
| The memo is the data structure that remembers which objects the
| pickler has already seen, so that shared or recursive objects
| are pickled by reference and not by value. This method is
| useful when re-using picklers.
|
| dump(self, obj)
| Write a pickled representation of obj to the open file.
|
| get(self, i)
| # Return a GET (BINGET, LONG_BINGET) opcode string, with argument i.
|
| memoize(self, obj)
| Store an object in the memo.
|
| persistent_id(self, obj)
|
| put(self, idx)
| # Return a PUT (BINPUT, LONG_BINPUT) opcode string, with argument i.
|
| save_bool(self, obj)
|
| save_bytearray(self, obj)
|
| save_bytes(self, obj)
|
| save_dict(self, obj)
|
| save_float(self, obj)
|
| save_frozenset(self, obj)
|
| save_global(self, obj, name=None)
|
| save_list(self, obj)
|
| save_long(self, obj)
|
| save_none(self, obj)
|
| save_pers(self, pid)
|
| save_picklebuffer(self, obj)
|
| save_reduce(self, func, args, state=None, listitems=None, dictitems=None, state_setter=None, *, obj=None)
|
| save_set(self, obj)
|
| save_str(self, obj)
|
| save_tuple(self, obj)
|
| save_type(self, obj)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from pickle._Pickler:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class NumpyUnpickler(pickle._Unpickler)
| NumpyUnpickler(filename, file_handle, mmap_mode=None)
|
| A subclass of the Unpickler to unpickle our numpy pickles.
|
| Attributes
| ----------
| mmap_mode: str
| The memorymap mode to use for reading numpy arrays.
| file_handle: file_like
| File object to unpickle from.
| filename: str
| Name of the file to unpickle from. It should correspond to file_handle.
| This parameter is required when using mmap_mode.
| np: module
| Reference to numpy module if numpy is installed else None.
|
| Method resolution order:
| NumpyUnpickler
| pickle._Unpickler
| builtins.object
|
| Methods defined here:
|
| __init__(self, filename, file_handle, mmap_mode=None)
| This takes a binary file for reading a pickle data stream.
|
| The protocol version of the pickle is detected automatically, so
| no proto argument is needed.
|
| The argument *file* must have two methods, a read() method that
| takes an integer argument, and a readline() method that requires
| no arguments. Both methods should return bytes. Thus *file*
| can be a binary file object opened for reading, an io.BytesIO
| object, or any other custom object that meets this interface.
|
| The file-like object must have two methods, a read() method
| that takes an integer argument, and a readline() method that
| requires no arguments. Both methods should return bytes.
| Thus file-like object can be a binary file object opened for
| reading, a BytesIO object, or any other custom object that
| meets this interface.
|
| If *buffers* is not None, it should be an iterable of buffer-enabled
| objects that is consumed each time the pickle stream references
| an out-of-band buffer view. Such buffers have been given in order
| to the *buffer_callback* of a Pickler object.
|
| If *buffers* is None (the default), then the buffers are taken
| from the pickle stream, assuming they are serialized there.
| It is an error for *buffers* to be None if the pickle stream
| was produced with a non-None *buffer_callback*.
|
| Other optional arguments are *fix_imports*, *encoding* and
| *errors*, which are used to control compatibility support for
| pickle stream generated by Python 2. If *fix_imports* is True,
| pickle will try to map the old Python 2 names to the new names
| used in Python 3. The *encoding* and *errors* tell pickle how
| to decode 8-bit string instances pickled by Python 2; these
| default to 'ASCII' and 'strict', respectively. *encoding* can be
| 'bytes' to read these 8-bit string instances as bytes objects.
|
| load_build(self)
| Called to set the state of a newly created object.
|
| We capture it to replace our place-holder objects, NDArrayWrapper or
| NumpyArrayWrapper, by the array we are interested in. We
| replace them directly in the stack of pickler.
| NDArrayWrapper is used for backward compatibility with joblib <= 0.9.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| dispatch = {40: <function _Unpickler.load_mark>, 41: <function _Unpick...
|
| ----------------------------------------------------------------------
| Methods inherited from pickle._Unpickler:
|
| find_class(self, module, name)
|
| get_extension(self, code)
|
| load(self)
| Read a pickled object representation from the open file.
|
| Return the reconstituted object hierarchy specified in the file.
|
| load_additems(self)
|
| load_append(self)
|
| load_appends(self)
|
| load_binbytes(self)
|
| load_binbytes8(self)
|
| load_binfloat(self)
|
| load_binget(self)
|
| load_binint(self)
|
| load_binint1(self)
|
| load_binint2(self)
|
| load_binpersid(self)
|
| load_binput(self)
|
| load_binstring(self)
|
| load_binunicode(self)
|
| load_binunicode8(self)
|
| load_bytearray8(self)
|
| load_dict(self)
|
| load_dup(self)
|
| load_empty_dictionary(self)
|
| load_empty_list(self)
|
| load_empty_set(self)
|
| load_empty_tuple(self)
|
| load_ext1(self)
|
| load_ext2(self)
|
| load_ext4(self)
|
| load_false(self)
|
| load_float(self)
|
| load_frame(self)
|
| load_frozenset(self)
|
| load_get(self)
|
| load_global(self)
|
| load_inst(self)
|
| load_int(self)
|
| load_list(self)
|
| load_long(self)
|
| load_long1(self)
|
| load_long4(self)
|
| load_long_binget(self)
|
| load_long_binput(self)
|
| load_mark(self)
|
| load_memoize(self)
|
| load_newobj(self)
|
| load_newobj_ex(self)
|
| load_next_buffer(self)
|
| load_none(self)
|
| load_obj(self)
|
| load_persid(self)
|
| load_pop(self)
|
| load_pop_mark(self)
|
| load_proto(self)
|
| load_put(self)
|
| load_readonly_buffer(self)
|
| load_reduce(self)
|
| load_setitem(self)
|
| load_setitems(self)
|
| load_short_binbytes(self)
|
| load_short_binstring(self)
|
| load_short_binunicode(self)
|
| load_stack_global(self)
|
| load_stop(self)
|
| load_string(self)
|
| load_true(self)
|
| load_tuple(self)
|
| load_tuple1(self)
|
| load_tuple2(self)
|
| load_tuple3(self)
|
| load_unicode(self)
|
| persistent_load(self, pid)
|
| pop_mark(self)
| # Return a list of items pushed in the stack after last MARK instruction.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from pickle._Unpickler:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
dump(value, filename, compress=0, protocol=None, cache_size=None)
Persist an arbitrary Python object into one file.
Read more in the :ref:`User Guide <persistence>`.
Parameters
----------
value: any Python object
The object to store to disk.
filename: str, pathlib.Path, or file object.
The file object or path of the file in which it is to be stored.
The compression method corresponding to one of the supported filename
extensions ('.z', '.gz', '.bz2', '.xz' or '.lzma') will be used
automatically.
compress: int from 0 to 9 or bool or 2-tuple, optional
Optional compression level for the data. 0 or False is no compression.
Higher value means more compression, but also slower read and
write times. Using a value of 3 is often a good compromise.
See the notes for more details.
If compress is True, the compression level used is 3.
If compress is a 2-tuple, the first element must correspond to a string
between supported compressors (e.g 'zlib', 'gzip', 'bz2', 'lzma'
'xz'), the second element must be an integer from 0 to 9, corresponding
to the compression level.
protocol: int, optional
Pickle protocol, see pickle.dump documentation for more details.
cache_size: positive int, optional
This option is deprecated in 0.10 and has no effect.
Returns
-------
filenames: list of strings
The list of file names in which the data is stored. If
compress is false, each array is stored in a different file.
See Also
--------
joblib.load : corresponding loader
Notes
-----
Memmapping on load cannot be used for compressed files. Thus
using compression can significantly slow down loading. In
addition, compressed files take up extra memory during
dump and load.
load(filename, mmap_mode=None)
Reconstruct a Python object from a file persisted with joblib.dump.
Read more in the :ref:`User Guide <persistence>`.
WARNING: joblib.load relies on the pickle module and can therefore
execute arbitrary Python code. It should therefore never be used
to load files from untrusted sources.
Parameters
----------
filename: str, pathlib.Path, or file object.
The file object or path of the file from which to load the object
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
If not None, the arrays are memory-mapped from the disk. This
mode has no effect for compressed files. Note that in this
case the reconstructed object might no longer match exactly
the originally pickled object.
Returns
-------
result: any Python object
The object stored in the file.
See Also
--------
joblib.dump : function to save an object
Notes
-----
This function can load numpy array files saved separately during the
dump. If the mmap_mode argument is given, it is passed to np.load and
arrays are loaded as memmaps. As a consequence, the reconstructed
object might not match the original pickled object. Note that if the
file was saved with compression, the arrays cannot be memmapped.
load_temporary_memmap(filename, mmap_mode, unlink_on_gc_collect)
DATA
BUFFER_SIZE = 262144
LZ4_NOT_INSTALLED_ERROR = 'LZ4 is not installed. Install it with pip: ...
NUMPY_ARRAY_ALIGNMENT_BYTES = 16
lz4 = None
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\numpy_pickle.py
joblib.numpy_pickle_compat
Python Library Documentation: module joblib.numpy_pickle_compat in joblib
NAME
joblib.numpy_pickle_compat - Numpy pickle compatibility functions.
CLASSES
builtins.object
NDArrayWrapper
ZNDArrayWrapper
pickle._Unpickler(builtins.object)
ZipNumpyUnpickler
class NDArrayWrapper(builtins.object)
| NDArrayWrapper(filename, subclass, allow_mmap=True)
|
| An object to be persisted instead of numpy arrays.
|
| The only thing this object does, is to carry the filename in which
| the array has been persisted, and the array subclass.
|
| Methods defined here:
|
| __init__(self, filename, subclass, allow_mmap=True)
| Constructor. Store the useful information for later.
|
| read(self, unpickler)
| Reconstruct the array.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class ZNDArrayWrapper(NDArrayWrapper)
| ZNDArrayWrapper(filename, init_args, state)
|
| An object to be persisted instead of numpy arrays.
|
| This object store the Zfile filename in which
| the data array has been persisted, and the meta information to
| retrieve it.
| The reason that we store the raw buffer data of the array and
| the meta information, rather than array representation routine
| (tobytes) is that it enables us to use completely the strided
| model to avoid memory copies (a and a.T store as fast). In
| addition saving the heavy information separately can avoid
| creating large temporary buffers when unpickling data with
| large arrays.
|
| Method resolution order:
| ZNDArrayWrapper
| NDArrayWrapper
| builtins.object
|
| Methods defined here:
|
| __init__(self, filename, init_args, state)
| Constructor. Store the useful information for later.
|
| read(self, unpickler)
| Reconstruct the array from the meta-information and the z-file.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from NDArrayWrapper:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class ZipNumpyUnpickler(pickle._Unpickler)
| ZipNumpyUnpickler(filename, file_handle, mmap_mode=None)
|
| A subclass of the Unpickler to unpickle our numpy pickles.
|
| Method resolution order:
| ZipNumpyUnpickler
| pickle._Unpickler
| builtins.object
|
| Methods defined here:
|
| __init__(self, filename, file_handle, mmap_mode=None)
| Constructor.
|
| load_build(self)
| Set the state of a newly created object.
|
| We capture it to replace our place-holder objects,
| NDArrayWrapper, by the array we are interested in. We
| replace them directly in the stack of pickler.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| dispatch = {40: <function _Unpickler.load_mark>, 41: <function _Unpick...
|
| ----------------------------------------------------------------------
| Methods inherited from pickle._Unpickler:
|
| find_class(self, module, name)
|
| get_extension(self, code)
|
| load(self)
| Read a pickled object representation from the open file.
|
| Return the reconstituted object hierarchy specified in the file.
|
| load_additems(self)
|
| load_append(self)
|
| load_appends(self)
|
| load_binbytes(self)
|
| load_binbytes8(self)
|
| load_binfloat(self)
|
| load_binget(self)
|
| load_binint(self)
|
| load_binint1(self)
|
| load_binint2(self)
|
| load_binpersid(self)
|
| load_binput(self)
|
| load_binstring(self)
|
| load_binunicode(self)
|
| load_binunicode8(self)
|
| load_bytearray8(self)
|
| load_dict(self)
|
| load_dup(self)
|
| load_empty_dictionary(self)
|
| load_empty_list(self)
|
| load_empty_set(self)
|
| load_empty_tuple(self)
|
| load_ext1(self)
|
| load_ext2(self)
|
| load_ext4(self)
|
| load_false(self)
|
| load_float(self)
|
| load_frame(self)
|
| load_frozenset(self)
|
| load_get(self)
|
| load_global(self)
|
| load_inst(self)
|
| load_int(self)
|
| load_list(self)
|
| load_long(self)
|
| load_long1(self)
|
| load_long4(self)
|
| load_long_binget(self)
|
| load_long_binput(self)
|
| load_mark(self)
|
| load_memoize(self)
|
| load_newobj(self)
|
| load_newobj_ex(self)
|
| load_next_buffer(self)
|
| load_none(self)
|
| load_obj(self)
|
| load_persid(self)
|
| load_pop(self)
|
| load_pop_mark(self)
|
| load_proto(self)
|
| load_put(self)
|
| load_readonly_buffer(self)
|
| load_reduce(self)
|
| load_setitem(self)
|
| load_setitems(self)
|
| load_short_binbytes(self)
|
| load_short_binstring(self)
|
| load_short_binunicode(self)
|
| load_stack_global(self)
|
| load_stop(self)
|
| load_string(self)
|
| load_true(self)
|
| load_tuple(self)
|
| load_tuple1(self)
|
| load_tuple2(self)
|
| load_tuple3(self)
|
| load_unicode(self)
|
| persistent_load(self, pid)
|
| pop_mark(self)
| # Return a list of items pushed in the stack after last MARK instruction.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from pickle._Unpickler:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
asbytes(s)
hex_str(an_int)
Convert an int to an hexadecimal string.
load_compatibility(filename)
Reconstruct a Python object from a file persisted with joblib.dump.
This function ensures the compatibility with joblib old persistence format
(<= 0.9.3).
Parameters
----------
filename: string
The name of the file from which to load the object
Returns
-------
result: any Python object
The object stored in the file.
See Also
--------
joblib.dump : function to save an object
Notes
-----
This function can load numpy array files saved separately during the
dump.
read_zfile(file_handle)
Read the z-file and return the content as a string.
Z-files are raw data compressed with zlib used internally by joblib
for persistence. Backward compatibility is not guaranteed. Do not
use for external purposes.
write_zfile(file_handle, data, compress=1)
Write the data in the given file as a Z-file.
Z-files are raw data compressed with zlib used internally by joblib
for persistence. Backward compatibility is not guaranteed. Do not
use for external purposes.
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\numpy_pickle_compat.py
joblib.numpy_pickle_utils
Python Library Documentation: module joblib.numpy_pickle_utils in joblib
NAME
joblib.numpy_pickle_utils - Utilities for fast persistence of big data, with optional compression.
DATA
BUFFER_SIZE = 262144
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\numpy_pickle_utils.py
joblib.os
Python Library Documentation: module os
NAME
os - OS routines for NT or Posix depending on what system we're on.
MODULE REFERENCE
https://docs.python.org/3.12/library/os.html
The following documentation is automatically generated from the Python
source files. It may be incomplete, incorrect or include features that
are considered implementation detail and may vary between Python
implementations. When in doubt, consult the module reference at the
location listed above.
DESCRIPTION
This exports:
- all functions from posix or nt, e.g. unlink, stat, etc.
- os.path is either posixpath or ntpath
- os.name is either 'posix' or 'nt'
- os.curdir is a string representing the current directory (always '.')
- os.pardir is a string representing the parent directory (always '..')
- os.sep is the (or a most common) pathname separator ('/' or '\\')
- os.extsep is the extension separator (always '.')
- os.altsep is the alternate pathname separator (None or '/')
- os.pathsep is the component separator used in $PATH etc
- os.linesep is the line separator in text files ('\r' or '\n' or '\r\n')
- os.defpath is the default search path for executables
- os.devnull is the file path of the null device ('/dev/null', etc.)
Programs that import and use 'os' stand a better chance of being
portable between different platforms. Of course, they must then
only use functions that are defined by all platforms (e.g., unlink
and opendir), and leave all pathname manipulation to os.path
(e.g., split and join).
CLASSES
builtins.Exception(builtins.BaseException)
builtins.OSError
builtins.object
nt.DirEntry
builtins.tuple(builtins.object)
nt.times_result
nt.uname_result
stat_result
statvfs_result
terminal_size
class DirEntry(builtins.object)
| Methods defined here:
|
| __fspath__(self, /)
| Returns the path for the entry.
|
| __repr__(self, /)
| Return repr(self).
|
| inode(self, /)
| Return inode of the entry; cached per entry.
|
| is_dir(self, /, *, follow_symlinks=True)
| Return True if the entry is a directory; cached per entry.
|
| is_file(self, /, *, follow_symlinks=True)
| Return True if the entry is a file; cached per entry.
|
| is_junction(self, /)
| Return True if the entry is a junction; cached per entry.
|
| is_symlink(self, /)
| Return True if the entry is a symbolic link; cached per entry.
|
| stat(self, /, *, follow_symlinks=True)
| Return stat_result object for the entry; cached per entry.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| name
| the entry's base filename, relative to scandir() "path" argument
|
| path
| the entry's full path name; equivalent to os.path.join(scandir_path, entry.name)
error = class OSError(Exception)
| Base class for I/O related errors.
|
| Method resolution order:
| OSError
| Exception
| BaseException
| object
|
| Built-in subclasses:
| BlockingIOError
| ChildProcessError
| ConnectionError
| FileExistsError
| ... and 7 other subclasses
|
| Methods defined here:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __reduce__(...)
| Helper for pickle.
|
| __str__(self, /)
| Return str(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| characters_written
|
| errno
| POSIX exception code
|
| filename
| exception filename
|
| filename2
| second exception filename
|
| strerror
| exception strerror
|
| winerror
| Win32 exception code
|
| ----------------------------------------------------------------------
| Methods inherited from BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class stat_result(builtins.tuple)
| stat_result(iterable=(), /)
|
| stat_result: Result from stat, fstat, or lstat.
|
| This object may be accessed either as a tuple of
| (mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime)
| or via the attributes st_mode, st_ino, st_dev, st_nlink, st_uid, and so on.
|
| Posix/windows: If your platform supports st_blksize, st_blocks, st_rdev,
| or st_flags, they are available as attributes only.
|
| See os.stat for more information.
|
| Method resolution order:
| stat_result
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| st_atime
| time of last access
|
| st_atime_ns
| time of last access in nanoseconds
|
| st_birthtime
| time of creation
|
| st_birthtime_ns
| time of creation in nanoseconds
|
| st_ctime
| time of last change
|
| st_ctime_ns
| time of last change in nanoseconds
|
| st_dev
| device
|
| st_file_attributes
| Windows file attribute bits
|
| st_gid
| group ID of owner
|
| st_ino
| inode
|
| st_mode
| protection bits
|
| st_mtime
| time of last modification
|
| st_mtime_ns
| time of last modification in nanoseconds
|
| st_nlink
| number of hard links
|
| st_reparse_tag
| Windows reparse tag
|
| st_size
| total size, in bytes
|
| st_uid
| user ID of owner
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('st_mode', 'st_ino', 'st_dev', 'st_nlink', 'st_uid',...
|
| n_fields = 20
|
| n_sequence_fields = 10
|
| n_unnamed_fields = 3
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class statvfs_result(builtins.tuple)
| statvfs_result(iterable=(), /)
|
| statvfs_result: Result from statvfs or fstatvfs.
|
| This object may be accessed either as a tuple of
| (bsize, frsize, blocks, bfree, bavail, files, ffree, favail, flag, namemax),
| or via the attributes f_bsize, f_frsize, f_blocks, f_bfree, and so on.
|
| See os.statvfs for more information.
|
| Method resolution order:
| statvfs_result
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| f_bavail
|
| f_bfree
|
| f_blocks
|
| f_bsize
|
| f_favail
|
| f_ffree
|
| f_files
|
| f_flag
|
| f_frsize
|
| f_fsid
|
| f_namemax
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('f_bsize', 'f_frsize', 'f_blocks', 'f_bfree', 'f_bav...
|
| n_fields = 11
|
| n_sequence_fields = 10
|
| n_unnamed_fields = 0
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class terminal_size(builtins.tuple)
| terminal_size(iterable=(), /)
|
| A tuple of (columns, lines) for holding terminal window size
|
| Method resolution order:
| terminal_size
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| columns
| width of the terminal window in characters
|
| lines
| height of the terminal window in characters
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('columns', 'lines')
|
| n_fields = 2
|
| n_sequence_fields = 2
|
| n_unnamed_fields = 0
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class times_result(builtins.tuple)
| times_result(iterable=(), /)
|
| times_result: Result from os.times().
|
| This object may be accessed either as a tuple of
| (user, system, children_user, children_system, elapsed),
| or via the attributes user, system, children_user, children_system,
| and elapsed.
|
| See os.times for more information.
|
| Method resolution order:
| times_result
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| children_system
| system time of children
|
| children_user
| user time of children
|
| elapsed
| elapsed time since an arbitrary point in the past
|
| system
| system time
|
| user
| user time
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('user', 'system', 'children_user', 'children_system'...
|
| n_fields = 5
|
| n_sequence_fields = 5
|
| n_unnamed_fields = 0
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class uname_result(builtins.tuple)
| uname_result(iterable=(), /)
|
| uname_result: Result from os.uname().
|
| This object may be accessed either as a tuple of
| (sysname, nodename, release, version, machine),
| or via the attributes sysname, nodename, release, version, and machine.
|
| See os.uname for more information.
|
| Method resolution order:
| uname_result
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| machine
| hardware identifier
|
| nodename
| name of machine on network (implementation-defined)
|
| release
| operating system release
|
| sysname
| operating system name
|
| version
| operating system version
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('sysname', 'nodename', 'release', 'version', 'machin...
|
| n_fields = 5
|
| n_sequence_fields = 5
|
| n_unnamed_fields = 0
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
FUNCTIONS
_exit(status)
Exit to the system with specified status, without normal exit processing.
abort()
Abort the interpreter immediately.
This function 'dumps core' or otherwise fails in the hardest way possible
on the hosting operating system. This function never returns.
access(path, mode, *, dir_fd=None, effective_ids=False, follow_symlinks=True)
Use the real uid/gid to test for access to a path.
path
Path to be tested; can be string, bytes, or a path-like object.
mode
Operating-system mode bitfield. Can be F_OK to test existence,
or the inclusive-OR of R_OK, W_OK, and X_OK.
dir_fd
If not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that
directory.
effective_ids
If True, access will use the effective uid/gid instead of
the real uid/gid.
follow_symlinks
If False, and the last element of the path is a symbolic link,
access will examine the symbolic link itself instead of the file
the link points to.
dir_fd, effective_ids, and follow_symlinks may not be implemented
on your platform. If they are unavailable, using them will raise a
NotImplementedError.
Note that most operations will use the effective uid/gid, therefore this
routine can be used in a suid/sgid environment to test if the invoking user
has the specified access to the path.
chdir(path)
Change the current working directory to the specified path.
path may always be specified as a string.
On some platforms, path may also be specified as an open file descriptor.
If this functionality is unavailable, using it raises an exception.
chmod(path, mode, *, dir_fd=None, follow_symlinks=True)
Change the access permissions of a file.
path
Path to be modified. May always be specified as a str, bytes, or a path-like object.
On some platforms, path may also be specified as an open file descriptor.
If this functionality is unavailable, using it raises an exception.
mode
Operating-system mode bitfield.
Be careful when using number literals for *mode*. The conventional UNIX notation for
numeric modes uses an octal base, which needs to be indicated with a ``0o`` prefix in
Python.
dir_fd
If not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that
directory.
follow_symlinks
If False, and the last element of the path is a symbolic link,
chmod will modify the symbolic link itself instead of the file
the link points to.
It is an error to use dir_fd or follow_symlinks when specifying path as
an open file descriptor.
dir_fd and follow_symlinks may not be implemented on your platform.
If they are unavailable, using them will raise a NotImplementedError.
close(fd)
Close a file descriptor.
closerange(fd_low, fd_high, /)
Closes all file descriptors in [fd_low, fd_high), ignoring errors.
cpu_count()
Return the number of CPUs in the system; return None if indeterminable.
This number is not equivalent to the number of CPUs the current process can
use. The number of usable CPUs can be obtained with
``len(os.sched_getaffinity(0))``
device_encoding(fd)
Return a string describing the encoding of a terminal's file descriptor.
The file descriptor must be attached to a terminal.
If the device is not a terminal, return None.
dup(fd, /)
Return a duplicate of a file descriptor.
dup2(fd, fd2, inheritable=True)
Duplicate file descriptor.
execl(file, *args)
execl(file, *args)
Execute the executable file with argument list args, replacing the
current process.
execle(file, *args)
execle(file, *args, env)
Execute the executable file with argument list args and
environment env, replacing the current process.
execlp(file, *args)
execlp(file, *args)
Execute the executable file (which is searched for along $PATH)
with argument list args, replacing the current process.
execlpe(file, *args)
execlpe(file, *args, env)
Execute the executable file (which is searched for along $PATH)
with argument list args and environment env, replacing the current
process.
execv(path, argv, /)
Execute an executable path with arguments, replacing current process.
path
Path of executable file.
argv
Tuple or list of strings.
execve(path, argv, env)
Execute an executable path with arguments, replacing current process.
path
Path of executable file.
argv
Tuple or list of strings.
env
Dictionary of strings mapping to strings.
execvp(file, args)
execvp(file, args)
Execute the executable file (which is searched for along $PATH)
with argument list args, replacing the current process.
args may be a list or tuple of strings.
execvpe(file, args, env)
execvpe(file, args, env)
Execute the executable file (which is searched for along $PATH)
with argument list args and environment env, replacing the
current process.
args may be a list or tuple of strings.
fdopen(fd, mode='r', buffering=-1, encoding=None, *args, **kwargs)
fsdecode(filename)
Decode filename (an os.PathLike, bytes, or str) from the filesystem
encoding with 'surrogateescape' error handler, return str unchanged. On
Windows, use 'strict' error handler if the file system encoding is
'mbcs' (which is the default encoding).
fsencode(filename)
Encode filename (an os.PathLike, bytes, or str) to the filesystem
encoding with 'surrogateescape' error handler, return bytes unchanged.
On Windows, use 'strict' error handler if the file system encoding is
'mbcs' (which is the default encoding).
fspath(path)
Return the file system path representation of the object.
If the object is str or bytes, then allow it to pass through as-is. If the
object defines __fspath__(), then return the result of that method. All other
types raise a TypeError.
fstat(fd)
Perform a stat system call on the given file descriptor.
Like stat(), but for an open file descriptor.
Equivalent to os.stat(fd).
fsync(fd)
Force write of fd to disk.
ftruncate(fd, length, /)
Truncate a file, specified by file descriptor, to a specific length.
get_blocking(fd, /)
Get the blocking mode of the file descriptor.
Return False if the O_NONBLOCK flag is set, True if the flag is cleared.
get_exec_path(env=None)
Returns the sequence of directories that will be searched for the
named executable (similar to a shell) when launching a process.
*env* must be an environment variable dict or None. If *env* is None,
os.environ will be used.
get_handle_inheritable(handle, /)
Get the close-on-exe flag of the specified file descriptor.
get_inheritable(fd, /)
Get the close-on-exe flag of the specified file descriptor.
get_terminal_size(...)
Return the size of the terminal window as (columns, lines).
The optional argument fd (default standard output) specifies
which file descriptor should be queried.
If the file descriptor is not connected to a terminal, an OSError
is thrown.
This function will only be defined if an implementation is
available for this system.
shutil.get_terminal_size is the high-level function which should
normally be used, os.get_terminal_size is the low-level implementation.
getcwd()
Return a unicode string representing the current working directory.
getcwdb()
Return a bytes string representing the current working directory.
getenv(key, default=None)
Get an environment variable, return None if it doesn't exist.
The optional second argument can specify an alternate default.
key, default and the result are str.
getlogin()
Return the actual login name.
getpid()
Return the current process id.
getppid()
Return the parent's process id.
If the parent process has already exited, Windows machines will still
return its id; others systems will return the id of the 'init' process (1).
isatty(fd, /)
Return True if the fd is connected to a terminal.
Return True if the file descriptor is an open file descriptor
connected to the slave end of a terminal.
kill(pid, signal, /)
Kill a process with a signal.
link(src, dst, *, src_dir_fd=None, dst_dir_fd=None, follow_symlinks=True)
Create a hard link to a file.
If either src_dir_fd or dst_dir_fd is not None, it should be a file
descriptor open to a directory, and the respective path string (src or dst)
should be relative; the path will then be relative to that directory.
If follow_symlinks is False, and the last element of src is a symbolic
link, link will create a link to the symbolic link itself instead of the
file the link points to.
src_dir_fd, dst_dir_fd, and follow_symlinks may not be implemented on your
platform. If they are unavailable, using them will raise a
NotImplementedError.
listdir(path=None)
Return a list containing the names of the files in the directory.
path can be specified as either str, bytes, or a path-like object. If path is bytes,
the filenames returned will also be bytes; in all other circumstances
the filenames returned will be str.
If path is None, uses the path='.'.
On some platforms, path may also be specified as an open file descriptor;\
the file descriptor must refer to a directory.
If this functionality is unavailable, using it raises NotImplementedError.
The list is in arbitrary order. It does not include the special
entries '.' and '..' even if they are present in the directory.
listdrives()
Return a list containing the names of drives in the system.
A drive name typically looks like 'C:\\'.
listmounts(volume)
Return a list containing mount points for a particular volume.
'volume' should be a GUID path as returned from os.listvolumes.
listvolumes()
Return a list containing the volumes in the system.
Volumes are typically represented as a GUID path.
lseek(fd, position, whence, /)
Set the position of a file descriptor. Return the new position.
fd
An open file descriptor, as returned by os.open().
position
Position, interpreted relative to 'whence'.
whence
The relative position to seek from. Valid values are:
- SEEK_SET: seek from the start of the file.
- SEEK_CUR: seek from the current file position.
- SEEK_END: seek from the end of the file.
The return value is the number of bytes relative to the beginning of the file.
lstat(path, *, dir_fd=None)
Perform a stat system call on the given path, without following symbolic links.
Like stat(), but do not follow symbolic links.
Equivalent to stat(path, follow_symlinks=False).
makedirs(name, mode=511, exist_ok=False)
makedirs(name [, mode=0o777][, exist_ok=False])
Super-mkdir; create a leaf directory and all intermediate ones. Works like
mkdir, except that any intermediate path segment (not just the rightmost)
will be created if it does not exist. If the target directory already
exists, raise an OSError if exist_ok is False. Otherwise no exception is
raised. This is recursive.
mkdir(path, mode=511, *, dir_fd=None)
Create a directory.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
The mode argument is ignored on Windows. Where it is used, the current umask
value is first masked out.
open(path, flags, mode=511, *, dir_fd=None)
Open a file for low level IO. Returns a file descriptor (integer).
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
pipe()
Create a pipe.
Returns a tuple of two file descriptors:
(read_fd, write_fd)
popen(cmd, mode='r', buffering=-1)
putenv(name, value, /)
Change or add an environment variable.
read(fd, length, /)
Read from a file descriptor. Returns a bytes object.
readlink(path, *, dir_fd=None)
Return a string representing the path to which the symbolic link points.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform. If it is unavailable,
using it will raise a NotImplementedError.
remove(path, *, dir_fd=None)
Remove a file (same as unlink()).
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
removedirs(name)
removedirs(name)
Super-rmdir; remove a leaf directory and all empty intermediate
ones. Works like rmdir except that, if the leaf directory is
successfully removed, directories corresponding to rightmost path
segments will be pruned away until either the whole path is
consumed or an error occurs. Errors during this latter phase are
ignored -- they generally mean that a directory was not empty.
rename(src, dst, *, src_dir_fd=None, dst_dir_fd=None)
Rename a file or directory.
If either src_dir_fd or dst_dir_fd is not None, it should be a file
descriptor open to a directory, and the respective path string (src or dst)
should be relative; the path will then be relative to that directory.
src_dir_fd and dst_dir_fd, may not be implemented on your platform.
If they are unavailable, using them will raise a NotImplementedError.
renames(old, new)
renames(old, new)
Super-rename; create directories as necessary and delete any left
empty. Works like rename, except creation of any intermediate
directories needed to make the new pathname good is attempted
first. After the rename, directories corresponding to rightmost
path segments of the old name will be pruned until either the
whole path is consumed or a nonempty directory is found.
Note: this function can fail with the new directory structure made
if you lack permissions needed to unlink the leaf directory or
file.
replace(src, dst, *, src_dir_fd=None, dst_dir_fd=None)
Rename a file or directory, overwriting the destination.
If either src_dir_fd or dst_dir_fd is not None, it should be a file
descriptor open to a directory, and the respective path string (src or dst)
should be relative; the path will then be relative to that directory.
src_dir_fd and dst_dir_fd, may not be implemented on your platform.
If they are unavailable, using them will raise a NotImplementedError.
rmdir(path, *, dir_fd=None)
Remove a directory.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
scandir(path=None)
Return an iterator of DirEntry objects for given path.
path can be specified as either str, bytes, or a path-like object. If path
is bytes, the names of yielded DirEntry objects will also be bytes; in
all other circumstances they will be str.
If path is None, uses the path='.'.
set_blocking(fd, blocking, /)
Set the blocking mode of the specified file descriptor.
Set the O_NONBLOCK flag if blocking is False,
clear the O_NONBLOCK flag otherwise.
set_handle_inheritable(handle, inheritable, /)
Set the inheritable flag of the specified handle.
set_inheritable(fd, inheritable, /)
Set the inheritable flag of the specified file descriptor.
spawnl(mode, file, *args)
spawnl(mode, file, *args) -> integer
Execute file with arguments from args in a subprocess.
If mode == P_NOWAIT return the pid of the process.
If mode == P_WAIT return the process's exit code if it exits normally;
otherwise return -SIG, where SIG is the signal that killed it.
spawnle(mode, file, *args)
spawnle(mode, file, *args, env) -> integer
Execute file with arguments from args in a subprocess with the
supplied environment.
If mode == P_NOWAIT return the pid of the process.
If mode == P_WAIT return the process's exit code if it exits normally;
otherwise return -SIG, where SIG is the signal that killed it.
spawnv(mode, path, argv, /)
Execute the program specified by path in a new process.
mode
Mode of process creation.
path
Path of executable file.
argv
Tuple or list of strings.
spawnve(mode, path, argv, env, /)
Execute the program specified by path in a new process.
mode
Mode of process creation.
path
Path of executable file.
argv
Tuple or list of strings.
env
Dictionary of strings mapping to strings.
startfile(...)
Start a file with its associated application.
When "operation" is not specified or "open", this acts like
double-clicking the file in Explorer, or giving the file name as an
argument to the DOS "start" command: the file is opened with whatever
application (if any) its extension is associated.
When another "operation" is given, it specifies what should be done with
the file. A typical operation is "print".
"arguments" is passed to the application, but should be omitted if the
file is a document.
"cwd" is the working directory for the operation. If "filepath" is
relative, it will be resolved against this directory. This argument
should usually be an absolute path.
"show_cmd" can be used to override the recommended visibility option.
See the Windows ShellExecute documentation for values.
startfile returns as soon as the associated application is launched.
There is no option to wait for the application to close, and no way
to retrieve the application's exit status.
The filepath is relative to the current directory. If you want to use
an absolute path, make sure the first character is not a slash ("/");
the underlying Win32 ShellExecute function doesn't work if it is.
stat(path, *, dir_fd=None, follow_symlinks=True)
Perform a stat system call on the given path.
path
Path to be examined; can be string, bytes, a path-like object or
open-file-descriptor int.
dir_fd
If not None, it should be a file descriptor open to a directory,
and path should be a relative string; path will then be relative to
that directory.
follow_symlinks
If False, and the last element of the path is a symbolic link,
stat will examine the symbolic link itself instead of the file
the link points to.
dir_fd and follow_symlinks may not be implemented
on your platform. If they are unavailable, using them will raise a
NotImplementedError.
It's an error to use dir_fd or follow_symlinks when specifying path as
an open file descriptor.
strerror(code, /)
Translate an error code to a message string.
symlink(src, dst, target_is_directory=False, *, dir_fd=None)
Create a symbolic link pointing to src named dst.
target_is_directory is required on Windows if the target is to be
interpreted as a directory. (On Windows, symlink requires
Windows 6.0 or greater, and raises a NotImplementedError otherwise.)
target_is_directory is ignored on non-Windows platforms.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
system(command)
Execute the command in a subshell.
times()
Return a collection containing process timing information.
The object returned behaves like a named tuple with these fields:
(utime, stime, cutime, cstime, elapsed_time)
All fields are floating-point numbers.
truncate(path, length)
Truncate a file, specified by path, to a specific length.
On some platforms, path may also be specified as an open file descriptor.
If this functionality is unavailable, using it raises an exception.
umask(mask, /)
Set the current numeric umask and return the previous umask.
unlink(path, *, dir_fd=None)
Remove a file (same as remove()).
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
unsetenv(name, /)
Delete an environment variable.
urandom(size, /)
Return a bytes object containing random bytes suitable for cryptographic use.
utime(...)
Set the access and modified time of path.
path may always be specified as a string.
On some platforms, path may also be specified as an open file descriptor.
If this functionality is unavailable, using it raises an exception.
If times is not None, it must be a tuple (atime, mtime);
atime and mtime should be expressed as float seconds since the epoch.
If ns is specified, it must be a tuple (atime_ns, mtime_ns);
atime_ns and mtime_ns should be expressed as integer nanoseconds
since the epoch.
If times is None and ns is unspecified, utime uses the current time.
Specifying tuples for both times and ns is an error.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
If follow_symlinks is False, and the last element of the path is a symbolic
link, utime will modify the symbolic link itself instead of the file the
link points to.
It is an error to use dir_fd or follow_symlinks when specifying path
as an open file descriptor.
dir_fd and follow_symlinks may not be available on your platform.
If they are unavailable, using them will raise a NotImplementedError.
waitpid(pid, options, /)
Wait for completion of a given process.
Returns a tuple of information regarding the process:
(pid, status << 8)
The options argument is ignored on Windows.
waitstatus_to_exitcode(status)
Convert a wait status to an exit code.
On Unix:
* If WIFEXITED(status) is true, return WEXITSTATUS(status).
* If WIFSIGNALED(status) is true, return -WTERMSIG(status).
* Otherwise, raise a ValueError.
On Windows, return status shifted right by 8 bits.
On Unix, if the process is being traced or if waitpid() was called with
WUNTRACED option, the caller must first check if WIFSTOPPED(status) is true.
This function must not be called if WIFSTOPPED(status) is true.
walk(top, topdown=True, onerror=None, followlinks=False)
Directory tree generator.
For each directory in the directory tree rooted at top (including top
itself, but excluding '.' and '..'), yields a 3-tuple
dirpath, dirnames, filenames
dirpath is a string, the path to the directory. dirnames is a list of
the names of the subdirectories in dirpath (including symlinks to directories,
and excluding '.' and '..').
filenames is a list of the names of the non-directory files in dirpath.
Note that the names in the lists are just names, with no path components.
To get a full path (which begins with top) to a file or directory in
dirpath, do os.path.join(dirpath, name).
If optional arg 'topdown' is true or not specified, the triple for a
directory is generated before the triples for any of its subdirectories
(directories are generated top down). If topdown is false, the triple
for a directory is generated after the triples for all of its
subdirectories (directories are generated bottom up).
When topdown is true, the caller can modify the dirnames list in-place
(e.g., via del or slice assignment), and walk will only recurse into the
subdirectories whose names remain in dirnames; this can be used to prune the
search, or to impose a specific order of visiting. Modifying dirnames when
topdown is false has no effect on the behavior of os.walk(), since the
directories in dirnames have already been generated by the time dirnames
itself is generated. No matter the value of topdown, the list of
subdirectories is retrieved before the tuples for the directory and its
subdirectories are generated.
By default errors from the os.scandir() call are ignored. If
optional arg 'onerror' is specified, it should be a function; it
will be called with one argument, an OSError instance. It can
report the error to continue with the walk, or raise the exception
to abort the walk. Note that the filename is available as the
filename attribute of the exception object.
By default, os.walk does not follow symbolic links to subdirectories on
systems that support them. In order to get this functionality, set the
optional argument 'followlinks' to true.
Caution: if you pass a relative pathname for top, don't change the
current working directory between resumptions of walk. walk never
changes the current directory, and assumes that the client doesn't
either.
Example:
import os
from os.path import join, getsize
for root, dirs, files in os.walk('python/Lib/email'):
print(root, "consumes ")
print(sum(getsize(join(root, name)) for name in files), end=" ")
print("bytes in", len(files), "non-directory files")
if 'CVS' in dirs:
dirs.remove('CVS') # don't visit CVS directories
write(fd, data, /)
Write a bytes object to a file descriptor.
DATA
EX_OK = 0
F_OK = 0
O_APPEND = 8
O_BINARY = 32768
O_CREAT = 256
O_EXCL = 1024
O_NOINHERIT = 128
O_RANDOM = 16
O_RDONLY = 0
O_RDWR = 2
O_SEQUENTIAL = 32
O_SHORT_LIVED = 4096
O_TEMPORARY = 64
O_TEXT = 16384
O_TRUNC = 512
O_WRONLY = 1
P_DETACH = 4
P_NOWAIT = 1
P_NOWAITO = 3
P_OVERLAY = 2
P_WAIT = 0
R_OK = 4
SEEK_CUR = 1
SEEK_END = 2
SEEK_SET = 0
TMP_MAX = 2147483647
W_OK = 2
X_OK = 1
__all__ = ['altsep', 'curdir', 'pardir', 'sep', 'pathsep', 'linesep', ...
altsep = '/'
curdir = '.'
defpath = r'.;C:\bin'
devnull = 'nul'
environ = environ({'ALLUSERSPROFILE': 'C:\\ProgramData', '...DE_INJECT...
extsep = '.'
linesep = '\r\n'
name = 'nt'
pardir = '..'
pathsep = ';'
sep = r'\'
supports_bytes_environ = False
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\os.py
joblib.parallel
Python Library Documentation: module joblib.parallel in joblib
NAME
joblib.parallel - Helpers for embarrassingly parallel code.
CLASSES
builtins.object
BatchCompletionCallBack
BatchedCalls
parallel_config
parallel_backend
joblib.logger.Logger(builtins.object)
Parallel
class BatchCompletionCallBack(builtins.object)
| BatchCompletionCallBack(dispatch_timestamp, batch_size, parallel)
|
| Callback to keep track of completed results and schedule the next tasks.
|
| This callable is executed by the parent process whenever a worker process
| has completed a batch of tasks.
|
| It is used for progress reporting, to update estimate of the batch
| processing duration and to schedule the next batch of tasks to be
| processed.
|
| It is assumed that this callback will always be triggered by the backend
| right after the end of a task, in case of success as well as in case of
| failure.
|
| Methods defined here:
|
| __call__(self, out)
| Function called by the callback thread after a job is completed.
|
| __init__(self, dispatch_timestamp, batch_size, parallel)
| Initialize self. See help(type(self)) for accurate signature.
|
| get_result(self, timeout)
| Returns the raw result of the task that was submitted.
|
| If the task raised an exception rather than returning, this same
| exception will be raised instead.
|
| If the backend supports the retrieval callback, it is assumed that this
| method is only called after the result has been registered. It is
| ensured by checking that `self.status(timeout)` does not return
| TASK_PENDING. In this case, `get_result` directly returns the
| registered result (or raise the registered exception).
|
| For other backends, there are no such assumptions, but `get_result`
| still needs to synchronously retrieve the result before it can
| return it or raise. It will block at most `self.timeout` seconds
| waiting for retrieval to complete, after that it raises a TimeoutError.
|
| get_status(self, timeout)
| Get the status of the task.
|
| This function also checks if the timeout has been reached and register
| the TimeoutError outcome when it is the case.
|
| register_job(self, job)
| Register the object returned by `apply_async`.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class BatchedCalls(builtins.object)
| BatchedCalls(iterator_slice, backend_and_jobs, reducer_callback=None, pickle_cache=None)
|
| Wrap a sequence of (func, args, kwargs) tuples as a single callable
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __init__(self, iterator_slice, backend_and_jobs, reducer_callback=None, pickle_cache=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __len__(self)
|
| __reduce__(self)
| Helper for pickle.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Parallel(joblib.logger.Logger)
| Parallel(n_jobs=default(None), backend=default(None), return_as='list', verbose=default(0), timeout=None, pre_dispatch='2 * n_jobs', batch_size='auto', temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None))
|
| Helper class for readable parallel mapping.
|
| Read more in the :ref:`User Guide <parallel>`.
|
| Parameters
| ----------
| n_jobs: int, default=None
| The maximum number of concurrently running jobs, such as the number
| of Python worker processes when ``backend="loky"`` or the size of
| the thread-pool when ``backend="threading"``.
| This argument is converted to an integer, rounded below for float.
| If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
| ``n_cpus`` is obtained with :func:`~cpu_count`.
| For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
| using ``n_jobs=-2`` will result in all CPUs but one being used.
| This argument can also go above ``n_cpus``, which will cause
| oversubscription. In some cases, slight oversubscription can be
| beneficial, e.g., for tasks with large I/O operations.
| If 1 is given, no parallel computing code is used at all, and the
| behavior amounts to a simple python `for` loop. This mode is not
| compatible with ``timeout``.
| None is a marker for 'unset' that will be interpreted as n_jobs=1
| unless the call is performed under a :func:`~parallel_config`
| context manager that sets another value for ``n_jobs``.
| If n_jobs = 0 then a ValueError is raised.
| backend: str, ParallelBackendBase instance or None, default='loky'
| Specify the parallelization backend implementation.
| Supported backends are:
|
| - "loky" used by default, can induce some
| communication and memory overhead when exchanging input and
| output data with the worker Python processes. On some rare
| systems (such as Pyiodide), the loky backend may not be
| available.
| - "multiprocessing" previous process-based backend based on
| `multiprocessing.Pool`. Less robust than `loky`.
| - "threading" is a very low-overhead backend but it suffers
| from the Python Global Interpreter Lock if the called function
| relies a lot on Python objects. "threading" is mostly useful
| when the execution bottleneck is a compiled extension that
| explicitly releases the GIL (for instance a Cython loop wrapped
| in a "with nogil" block or an expensive call to a library such
| as NumPy).
| - finally, you can register backends by calling
| :func:`~register_parallel_backend`. This will allow you to
| implement a backend of your liking.
|
| It is not recommended to hard-code the backend name in a call to
| :class:`~Parallel` in a library. Instead it is recommended to set
| soft hints (prefer) or hard constraints (require) so as to make it
| possible for library users to change the backend from the outside
| using the :func:`~parallel_config` context manager.
| return_as: str in {'list', 'generator', 'generator_unordered'}, default='list'
| If 'list', calls to this instance will return a list, only when
| all results have been processed and retrieved.
| If 'generator', it will return a generator that yields the results
| as soon as they are available, in the order the tasks have been
| submitted with.
| If 'generator_unordered', the generator will immediately yield
| available results independently of the submission order. The output
| order is not deterministic in this case because it depends on the
| concurrency of the workers.
| prefer: str in {'processes', 'threads'} or None, default=None
| Soft hint to choose the default backend if no specific backend
| was selected with the :func:`~parallel_config` context manager.
| The default process-based backend is 'loky' and the default
| thread-based backend is 'threading'. Ignored if the ``backend``
| parameter is specified.
| require: 'sharedmem' or None, default=None
| Hard constraint to select the backend. If set to 'sharedmem',
| the selected backend will be single-host and thread-based even
| if the user asked for a non-thread based backend with
| :func:`~joblib.parallel_config`.
| verbose: int, default=0
| The verbosity level: if non zero, progress messages are
| printed. Above 50, the output is sent to stdout.
| The frequency of the messages increases with the verbosity level.
| If it more than 10, all iterations are reported.
| timeout: float or None, default=None
| Timeout limit for each task to complete. If any task takes longer
| a TimeOutError will be raised. Only applied when n_jobs != 1
| pre_dispatch: {'all', integer, or expression, as in '3*n_jobs'}, default='2*n_jobs'
| The number of batches (of tasks) to be pre-dispatched.
| Default is '2*n_jobs'. When batch_size="auto" this is reasonable
| default and the workers should never starve. Note that only basic
| arithmetics are allowed here and no modules can be used in this
| expression.
| batch_size: int or 'auto', default='auto'
| The number of atomic tasks to dispatch at once to each
| worker. When individual evaluations are very fast, dispatching
| calls to workers can be slower than sequential computation because
| of the overhead. Batching fast computations together can mitigate
| this.
| The ``'auto'`` strategy keeps track of the time it takes for a
| batch to complete, and dynamically adjusts the batch size to keep
| the time on the order of half a second, using a heuristic. The
| initial batch size is 1.
| ``batch_size="auto"`` with ``backend="threading"`` will dispatch
| batches of a single task at a time as the threading backend has
| very little overhead and using larger batch size has not proved to
| bring any gain in that case.
| temp_folder: str or None, default=None
| Folder to be used by the pool for memmapping large arrays
| for sharing memory with worker processes. If None, this will try in
| order:
|
| - a folder pointed by the JOBLIB_TEMP_FOLDER environment
| variable,
| - /dev/shm if the folder exists and is writable: this is a
| RAM disk filesystem available by default on modern Linux
| distributions,
| - the default system temporary folder that can be
| overridden with TMP, TMPDIR or TEMP environment
| variables, typically /tmp under Unix operating systems.
|
| Only active when ``backend="loky"`` or ``"multiprocessing"``.
| max_nbytes int, str, or None, optional, default='1M'
| Threshold on the size of arrays passed to the workers that
| triggers automated memory mapping in temp_folder. Can be an int
| in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
| Use None to disable memmapping of large arrays.
| Only active when ``backend="loky"`` or ``"multiprocessing"``.
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
| Memmapping mode for numpy arrays passed to workers. None will
| disable memmapping, other modes defined in the numpy.memmap doc:
| https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
| Also, see 'max_nbytes' parameter documentation for more details.
|
| Notes
| -----
|
| This object uses workers to compute in parallel the application of a
| function to many different arguments. The main functionality it brings
| in addition to using the raw multiprocessing or concurrent.futures API
| are (see examples for details):
|
| * More readable code, in particular since it avoids
| constructing list of arguments.
|
| * Easier debugging:
| - informative tracebacks even when the error happens on
| the client side
| - using 'n_jobs=1' enables to turn off parallel computing
| for debugging without changing the codepath
| - early capture of pickling errors
|
| * An optional progress meter.
|
| * Interruption of multiprocesses jobs with 'Ctrl-C'
|
| * Flexible pickling control for the communication to and from
| the worker processes.
|
| * Ability to use shared memory efficiently with worker
| processes for large numpy-based datastructures.
|
| Note that the intended usage is to run one call at a time. Multiple
| calls to the same Parallel object will result in a ``RuntimeError``
|
| Examples
| --------
|
| A simple example:
|
| >>> from math import sqrt
| >>> from joblib import Parallel, delayed
| >>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
| [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
|
| Reshaping the output when the function has several return
| values:
|
| >>> from math import modf
| >>> from joblib import Parallel, delayed
| >>> r = Parallel(n_jobs=1)(delayed(modf)(i/2.) for i in range(10))
| >>> res, i = zip(*r)
| >>> res
| (0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5)
| >>> i
| (0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 4.0, 4.0)
|
| The progress meter: the higher the value of `verbose`, the more
| messages:
|
| >>> from time import sleep
| >>> from joblib import Parallel, delayed
| >>> r = Parallel(n_jobs=2, verbose=10)(
| ... delayed(sleep)(.2) for _ in range(10)) #doctest: +SKIP
| [Parallel(n_jobs=2)]: Done 1 tasks | elapsed: 0.6s
| [Parallel(n_jobs=2)]: Done 4 tasks | elapsed: 0.8s
| [Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 1.4s finished
|
| Traceback example, note how the line of the error is indicated
| as well as the values of the parameter passed to the function that
| triggered the exception, even though the traceback happens in the
| child process:
|
| >>> from heapq import nlargest
| >>> from joblib import Parallel, delayed
| >>> Parallel(n_jobs=2)(
| ... delayed(nlargest)(2, n) for n in (range(4), 'abcde', 3))
| ... # doctest: +SKIP
| -----------------------------------------------------------------------
| Sub-process traceback:
| -----------------------------------------------------------------------
| TypeError Mon Nov 12 11:37:46 2012
| PID: 12934 Python 2.7.3: /usr/bin/python
| ........................................................................
| /usr/lib/python2.7/heapq.pyc in nlargest(n=2, iterable=3, key=None)
| 419 if n >= size:
| 420 return sorted(iterable, key=key, reverse=True)[:n]
| 421
| 422 # When key is none, use simpler decoration
| 423 if key is None:
| --> 424 it = izip(iterable, count(0,-1)) # decorate
| 425 result = _nlargest(n, it)
| 426 return map(itemgetter(0), result) # undecorate
| 427
| 428 # General case, slowest method
| TypeError: izip argument #1 must support iteration
| _______________________________________________________________________
|
|
| Using pre_dispatch in a producer/consumer situation, where the
| data is generated on the fly. Note how the producer is first
| called 3 times before the parallel loop is initiated, and then
| called to generate new data on the fly:
|
| >>> from math import sqrt
| >>> from joblib import Parallel, delayed
| >>> def producer():
| ... for i in range(6):
| ... print('Produced %s' % i)
| ... yield i
| >>> out = Parallel(n_jobs=2, verbose=100, pre_dispatch='1.5*n_jobs')(
| ... delayed(sqrt)(i) for i in producer()) #doctest: +SKIP
| Produced 0
| Produced 1
| Produced 2
| [Parallel(n_jobs=2)]: Done 1 jobs | elapsed: 0.0s
| Produced 3
| [Parallel(n_jobs=2)]: Done 2 jobs | elapsed: 0.0s
| Produced 4
| [Parallel(n_jobs=2)]: Done 3 jobs | elapsed: 0.0s
| Produced 5
| [Parallel(n_jobs=2)]: Done 4 jobs | elapsed: 0.0s
| [Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s remaining: 0.0s
| [Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s finished
|
| Method resolution order:
| Parallel
| joblib.logger.Logger
| builtins.object
|
| Methods defined here:
|
| __call__(self, iterable)
| Main function to dispatch parallel tasks.
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_value, traceback)
|
| __init__(self, n_jobs=default(None), backend=default(None), return_as='list', verbose=default(0), timeout=None, pre_dispatch='2 * n_jobs', batch_size='auto', temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None))
| Parameters
| ----------
| depth: int, optional
| The depth of objects printed.
| name: str, optional
| The namespace to log to. If None, defaults to joblib.
|
| __repr__(self)
| Return repr(self).
|
| dispatch_next(self)
| Dispatch more data for parallel processing
|
| This method is meant to be called concurrently by the multiprocessing
| callback. We rely on the thread-safety of dispatch_one_batch to protect
| against concurrent consumption of the unprotected iterator.
|
| dispatch_one_batch(self, iterator)
| Prefetch the tasks for the next batch and dispatch them.
|
| The effective size of the batch is computed here.
| If there are no more jobs to dispatch, return False, else return True.
|
| The iterator consumption and dispatching is protected by the same
| lock so calling this function should be thread safe.
|
| print_progress(self)
| Display the process of the parallel execution only a fraction
| of time, controlled by self.verbose.
|
| ----------------------------------------------------------------------
| Methods inherited from joblib.logger.Logger:
|
| debug(self, msg)
|
| format(self, obj, indent=0)
| Return the formatted representation of the object.
|
| info(self, msg)
|
| warn(self, msg)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from joblib.logger.Logger:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class parallel_backend(parallel_config)
| parallel_backend(backend, n_jobs=-1, inner_max_num_threads=None, **backend_params)
|
| Change the default backend used by Parallel inside a with block.
|
| .. warning::
| It is advised to use the :class:`~joblib.parallel_config` context
| manager instead, which allows more fine-grained control over the
| backend configuration.
|
| If ``backend`` is a string it must match a previously registered
| implementation using the :func:`~register_parallel_backend` function.
|
| By default the following backends are available:
|
| - 'loky': single-host, process-based parallelism (used by default),
| - 'threading': single-host, thread-based parallelism,
| - 'multiprocessing': legacy single-host, process-based parallelism.
|
| 'loky' is recommended to run functions that manipulate Python objects.
| 'threading' is a low-overhead alternative that is most efficient for
| functions that release the Global Interpreter Lock: e.g. I/O-bound code or
| CPU-bound code in a few calls to native code that explicitly releases the
| GIL. Note that on some rare systems (such as Pyodide),
| multiprocessing and loky may not be available, in which case joblib
| defaults to threading.
|
| You can also use the `Dask <https://docs.dask.org/en/stable/>`_ joblib
| backend to distribute work across machines. This works well with
| scikit-learn estimators with the ``n_jobs`` parameter, for example::
|
| >>> import joblib # doctest: +SKIP
| >>> from sklearn.model_selection import GridSearchCV # doctest: +SKIP
| >>> from dask.distributed import Client, LocalCluster # doctest: +SKIP
|
| >>> # create a local Dask cluster
| >>> cluster = LocalCluster() # doctest: +SKIP
| >>> client = Client(cluster) # doctest: +SKIP
| >>> grid_search = GridSearchCV(estimator, param_grid, n_jobs=-1)
| ... # doctest: +SKIP
| >>> with joblib.parallel_backend("dask", scatter=[X, y]): # doctest: +SKIP
| ... grid_search.fit(X, y)
|
| It is also possible to use the distributed 'ray' backend for distributing
| the workload to a cluster of nodes. To use the 'ray' joblib backend add
| the following lines::
|
| >>> from ray.util.joblib import register_ray # doctest: +SKIP
| >>> register_ray() # doctest: +SKIP
| >>> with parallel_backend("ray"): # doctest: +SKIP
| ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
| [-1, -2, -3, -4, -5]
|
| Alternatively the backend can be passed directly as an instance.
|
| By default all available workers will be used (``n_jobs=-1``) unless the
| caller passes an explicit value for the ``n_jobs`` parameter.
|
| This is an alternative to passing a ``backend='backend_name'`` argument to
| the :class:`~Parallel` class constructor. It is particularly useful when
| calling into library code that uses joblib internally but does not expose
| the backend argument in its own API.
|
| >>> from operator import neg
| >>> with parallel_backend('threading'):
| ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
| ...
| [-1, -2, -3, -4, -5]
|
| Joblib also tries to limit the oversubscription by limiting the number of
| threads usable in some third-party library threadpools like OpenBLAS, MKL
| or OpenMP. The default limit in each worker is set to
| ``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
| overwritten with the ``inner_max_num_threads`` argument which will be used
| to set this limit in the child processes.
|
| .. versionadded:: 0.10
|
| See Also
| --------
| joblib.parallel_config: context manager to change the backend
| configuration.
|
| Method resolution order:
| parallel_backend
| parallel_config
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
|
| __init__(self, backend, n_jobs=-1, inner_max_num_threads=None, **backend_params)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from parallel_config:
|
| __exit__(self, type, value, traceback)
|
| unregister(self)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from parallel_config:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class parallel_config(builtins.object)
| parallel_config(backend=default(None), *, n_jobs=default(None), verbose=default(0), temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None), inner_max_num_threads=None, **backend_params)
|
| Set the default backend or configuration for :class:`~joblib.Parallel`.
|
| This is an alternative to directly passing keyword arguments to the
| :class:`~joblib.Parallel` class constructor. It is particularly useful when
| calling into library code that uses joblib internally but does not expose
| the various parallel configuration arguments in its own API.
|
| Parameters
| ----------
| backend: str or ParallelBackendBase instance, default=None
| If ``backend`` is a string it must match a previously registered
| implementation using the :func:`~register_parallel_backend` function.
|
| By default the following backends are available:
|
| - 'loky': single-host, process-based parallelism (used by default),
| - 'threading': single-host, thread-based parallelism,
| - 'multiprocessing': legacy single-host, process-based parallelism.
|
| 'loky' is recommended to run functions that manipulate Python objects.
| 'threading' is a low-overhead alternative that is most efficient for
| functions that release the Global Interpreter Lock: e.g. I/O-bound
| code or CPU-bound code in a few calls to native code that explicitly
| releases the GIL. Note that on some rare systems (such as pyodide),
| multiprocessing and loky may not be available, in which case joblib
| defaults to threading.
|
| In addition, if the ``dask`` and ``distributed`` Python packages are
| installed, it is possible to use the 'dask' backend for better
| scheduling of nested parallel calls without over-subscription and
| potentially distribute parallel calls over a networked cluster of
| several hosts.
|
| It is also possible to use the distributed 'ray' backend for
| distributing the workload to a cluster of nodes. See more details
| in the Examples section below.
|
| Alternatively the backend can be passed directly as an instance.
|
| n_jobs: int, default=None
| The maximum number of concurrently running jobs, such as the number
| of Python worker processes when ``backend="loky"`` or the size of the
| thread-pool when ``backend="threading"``.
| This argument is converted to an integer, rounded below for float.
| If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
| ``n_cpus`` is obtained with :func:`~cpu_count`.
| For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
| using ``n_jobs=-2`` will result in all CPUs but one being used.
| This argument can also go above ``n_cpus``, which will cause
| oversubscription. In some cases, slight oversubscription can be
| beneficial, e.g., for tasks with large I/O operations.
| If 1 is given, no parallel computing code is used at all, and the
| behavior amounts to a simple python `for` loop. This mode is not
| compatible with `timeout`.
| None is a marker for 'unset' that will be interpreted as n_jobs=1
| unless the call is performed under a :func:`~parallel_config`
| context manager that sets another value for ``n_jobs``.
| If n_jobs = 0 then a ValueError is raised.
|
| verbose: int, default=0
| The verbosity level: if non zero, progress messages are
| printed. Above 50, the output is sent to stdout.
| The frequency of the messages increases with the verbosity level.
| If it more than 10, all iterations are reported.
|
| temp_folder: str or None, default=None
| Folder to be used by the pool for memmapping large arrays
| for sharing memory with worker processes. If None, this will try in
| order:
|
| - a folder pointed by the ``JOBLIB_TEMP_FOLDER`` environment
| variable,
| - ``/dev/shm`` if the folder exists and is writable: this is a
| RAM disk filesystem available by default on modern Linux
| distributions,
| - the default system temporary folder that can be
| overridden with ``TMP``, ``TMPDIR`` or ``TEMP`` environment
| variables, typically ``/tmp`` under Unix operating systems.
|
| max_nbytes int, str, or None, optional, default='1M'
| Threshold on the size of arrays passed to the workers that
| triggers automated memory mapping in temp_folder. Can be an int
| in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
| Use None to disable memmapping of large arrays.
|
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
| Memmapping mode for numpy arrays passed to workers. None will
| disable memmapping, other modes defined in the numpy.memmap doc:
| https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
| Also, see 'max_nbytes' parameter documentation for more details.
|
| prefer: str in {'processes', 'threads'} or None, default=None
| Soft hint to choose the default backend.
| The default process-based backend is 'loky' and the default
| thread-based backend is 'threading'. Ignored if the ``backend``
| parameter is specified.
|
| require: 'sharedmem' or None, default=None
| Hard constraint to select the backend. If set to 'sharedmem',
| the selected backend will be single-host and thread-based.
|
| inner_max_num_threads: int, default=None
| If not None, overwrites the limit set on the number of threads
| usable in some third-party library threadpools like OpenBLAS,
| MKL or OpenMP. This is only used with the ``loky`` backend.
|
| backend_params: dict
| Additional parameters to pass to the backend constructor when
| backend is a string.
|
| Notes
| -----
| Joblib tries to limit the oversubscription by limiting the number of
| threads usable in some third-party library threadpools like OpenBLAS, MKL
| or OpenMP. The default limit in each worker is set to
| ``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
| overwritten with the ``inner_max_num_threads`` argument which will be used
| to set this limit in the child processes.
|
| .. versionadded:: 1.3
|
| Examples
| --------
| >>> from operator import neg
| >>> with parallel_config(backend='threading'):
| ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
| ...
| [-1, -2, -3, -4, -5]
|
| To use the 'ray' joblib backend add the following lines:
|
| >>> from ray.util.joblib import register_ray # doctest: +SKIP
| >>> register_ray() # doctest: +SKIP
| >>> with parallel_config(backend="ray"): # doctest: +SKIP
| ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
| [-1, -2, -3, -4, -5]
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, type, value, traceback)
|
| __init__(self, backend=default(None), *, n_jobs=default(None), verbose=default(0), temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None), inner_max_num_threads=None, **backend_params)
| Initialize self. See help(type(self)) for accurate signature.
|
| unregister(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
cpu_count(only_physical_cores=False)
Return the number of CPUs.
This delegates to loky.cpu_count that takes into account additional
constraints such as Linux CFS scheduler quotas (typically set by container
runtimes such as docker) and CPU affinity (for instance using the taskset
command on Linux).
If only_physical_cores is True, do not take hyperthreading / SMT logical
cores into account.
delayed(function)
Decorator used to capture the arguments of a function.
effective_n_jobs(n_jobs=-1)
Determine the number of jobs that can actually run in parallel
n_jobs is the number of workers requested by the callers. Passing n_jobs=-1
means requesting all available workers for instance matching the number of
CPU cores on the worker host(s).
This method should return a guesstimate of the number of workers that can
actually perform work concurrently with the currently enabled default
backend. The primary use case is to make it possible for the caller to know
in how many chunks to slice the work.
In general working on larger data chunks is more efficient (less scheduling
overhead and better use of CPU cache prefetching heuristics) as long as all
the workers have enough work to do.
Warning: this function is experimental and subject to change in a future
version of joblib.
.. versionadded:: 0.10
get_active_backend(prefer=default(None), require=default(None), verbose=default(0))
Return the active default backend
register_parallel_backend(name, factory, make_default=False)
Register a new Parallel backend factory.
The new backend can then be selected by passing its name as the backend
argument to the :class:`~Parallel` class. Moreover, the default backend can
be overwritten globally by setting make_default=True.
The factory can be any callable that takes no argument and return an
instance of ``ParallelBackendBase``.
Warning: this function is experimental and subject to change in a future
version of joblib.
.. versionadded:: 0.10
sqrt(x, /)
Return the square root of x.
DATA
BACKENDS = {'loky': <class 'joblib._parallel_backends.LokyBackend'>, '...
DEFAULT_BACKEND = 'loky'
DEFAULT_MP_CONTEXT = None
DEFAULT_THREAD_BACKEND = 'threading'
EXTERNAL_BACKENDS = {'dask': <function _register_dask>}
IS_PYPY = False
MAYBE_AVAILABLE_BACKENDS = {'loky', 'multiprocessing'}
TASK_DONE = 'Done'
TASK_ERROR = 'Error'
TASK_PENDING = 'Pending'
VALID_BACKEND_CONSTRAINTS = ('sharedmem', None)
VALID_BACKEND_HINTS = ('processes', 'threads', None)
default_parallel_config = {'backend': default(None), 'max_nbytes': def...
method = None
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\parallel.py
joblib.parallel_backend
Python Library Documentation: class parallel_backend in module joblib.parallel
class parallel_backend(parallel_config)
| parallel_backend(backend, n_jobs=-1, inner_max_num_threads=None, **backend_params)
|
| Change the default backend used by Parallel inside a with block.
|
| .. warning::
| It is advised to use the :class:`~joblib.parallel_config` context
| manager instead, which allows more fine-grained control over the
| backend configuration.
|
| If ``backend`` is a string it must match a previously registered
| implementation using the :func:`~register_parallel_backend` function.
|
| By default the following backends are available:
|
| - 'loky': single-host, process-based parallelism (used by default),
| - 'threading': single-host, thread-based parallelism,
| - 'multiprocessing': legacy single-host, process-based parallelism.
|
| 'loky' is recommended to run functions that manipulate Python objects.
| 'threading' is a low-overhead alternative that is most efficient for
| functions that release the Global Interpreter Lock: e.g. I/O-bound code or
| CPU-bound code in a few calls to native code that explicitly releases the
| GIL. Note that on some rare systems (such as Pyodide),
| multiprocessing and loky may not be available, in which case joblib
| defaults to threading.
|
| You can also use the `Dask <https://docs.dask.org/en/stable/>`_ joblib
| backend to distribute work across machines. This works well with
| scikit-learn estimators with the ``n_jobs`` parameter, for example::
|
| >>> import joblib # doctest: +SKIP
| >>> from sklearn.model_selection import GridSearchCV # doctest: +SKIP
| >>> from dask.distributed import Client, LocalCluster # doctest: +SKIP
|
| >>> # create a local Dask cluster
| >>> cluster = LocalCluster() # doctest: +SKIP
| >>> client = Client(cluster) # doctest: +SKIP
| >>> grid_search = GridSearchCV(estimator, param_grid, n_jobs=-1)
| ... # doctest: +SKIP
| >>> with joblib.parallel_backend("dask", scatter=[X, y]): # doctest: +SKIP
| ... grid_search.fit(X, y)
|
| It is also possible to use the distributed 'ray' backend for distributing
| the workload to a cluster of nodes. To use the 'ray' joblib backend add
| the following lines::
|
| >>> from ray.util.joblib import register_ray # doctest: +SKIP
| >>> register_ray() # doctest: +SKIP
| >>> with parallel_backend("ray"): # doctest: +SKIP
| ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
| [-1, -2, -3, -4, -5]
|
| Alternatively the backend can be passed directly as an instance.
|
| By default all available workers will be used (``n_jobs=-1``) unless the
| caller passes an explicit value for the ``n_jobs`` parameter.
|
| This is an alternative to passing a ``backend='backend_name'`` argument to
| the :class:`~Parallel` class constructor. It is particularly useful when
| calling into library code that uses joblib internally but does not expose
| the backend argument in its own API.
|
| >>> from operator import neg
| >>> with parallel_backend('threading'):
| ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
| ...
| [-1, -2, -3, -4, -5]
|
| Joblib also tries to limit the oversubscription by limiting the number of
| threads usable in some third-party library threadpools like OpenBLAS, MKL
| or OpenMP. The default limit in each worker is set to
| ``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
| overwritten with the ``inner_max_num_threads`` argument which will be used
| to set this limit in the child processes.
|
| .. versionadded:: 0.10
|
| See Also
| --------
| joblib.parallel_config: context manager to change the backend
| configuration.
|
| Method resolution order:
| parallel_backend
| parallel_config
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
|
| __init__(self, backend, n_jobs=-1, inner_max_num_threads=None, **backend_params)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from parallel_config:
|
| __exit__(self, type, value, traceback)
|
| unregister(self)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from parallel_config:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
joblib.parallel_config
Python Library Documentation: class parallel_config in module joblib.parallel
class parallel_config(builtins.object)
| parallel_config(backend=default(None), *, n_jobs=default(None), verbose=default(0), temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None), inner_max_num_threads=None, **backend_params)
|
| Set the default backend or configuration for :class:`~joblib.Parallel`.
|
| This is an alternative to directly passing keyword arguments to the
| :class:`~joblib.Parallel` class constructor. It is particularly useful when
| calling into library code that uses joblib internally but does not expose
| the various parallel configuration arguments in its own API.
|
| Parameters
| ----------
| backend: str or ParallelBackendBase instance, default=None
| If ``backend`` is a string it must match a previously registered
| implementation using the :func:`~register_parallel_backend` function.
|
| By default the following backends are available:
|
| - 'loky': single-host, process-based parallelism (used by default),
| - 'threading': single-host, thread-based parallelism,
| - 'multiprocessing': legacy single-host, process-based parallelism.
|
| 'loky' is recommended to run functions that manipulate Python objects.
| 'threading' is a low-overhead alternative that is most efficient for
| functions that release the Global Interpreter Lock: e.g. I/O-bound
| code or CPU-bound code in a few calls to native code that explicitly
| releases the GIL. Note that on some rare systems (such as pyodide),
| multiprocessing and loky may not be available, in which case joblib
| defaults to threading.
|
| In addition, if the ``dask`` and ``distributed`` Python packages are
| installed, it is possible to use the 'dask' backend for better
| scheduling of nested parallel calls without over-subscription and
| potentially distribute parallel calls over a networked cluster of
| several hosts.
|
| It is also possible to use the distributed 'ray' backend for
| distributing the workload to a cluster of nodes. See more details
| in the Examples section below.
|
| Alternatively the backend can be passed directly as an instance.
|
| n_jobs: int, default=None
| The maximum number of concurrently running jobs, such as the number
| of Python worker processes when ``backend="loky"`` or the size of the
| thread-pool when ``backend="threading"``.
| This argument is converted to an integer, rounded below for float.
| If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
| ``n_cpus`` is obtained with :func:`~cpu_count`.
| For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
| using ``n_jobs=-2`` will result in all CPUs but one being used.
| This argument can also go above ``n_cpus``, which will cause
| oversubscription. In some cases, slight oversubscription can be
| beneficial, e.g., for tasks with large I/O operations.
| If 1 is given, no parallel computing code is used at all, and the
| behavior amounts to a simple python `for` loop. This mode is not
| compatible with `timeout`.
| None is a marker for 'unset' that will be interpreted as n_jobs=1
| unless the call is performed under a :func:`~parallel_config`
| context manager that sets another value for ``n_jobs``.
| If n_jobs = 0 then a ValueError is raised.
|
| verbose: int, default=0
| The verbosity level: if non zero, progress messages are
| printed. Above 50, the output is sent to stdout.
| The frequency of the messages increases with the verbosity level.
| If it more than 10, all iterations are reported.
|
| temp_folder: str or None, default=None
| Folder to be used by the pool for memmapping large arrays
| for sharing memory with worker processes. If None, this will try in
| order:
|
| - a folder pointed by the ``JOBLIB_TEMP_FOLDER`` environment
| variable,
| - ``/dev/shm`` if the folder exists and is writable: this is a
| RAM disk filesystem available by default on modern Linux
| distributions,
| - the default system temporary folder that can be
| overridden with ``TMP``, ``TMPDIR`` or ``TEMP`` environment
| variables, typically ``/tmp`` under Unix operating systems.
|
| max_nbytes int, str, or None, optional, default='1M'
| Threshold on the size of arrays passed to the workers that
| triggers automated memory mapping in temp_folder. Can be an int
| in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
| Use None to disable memmapping of large arrays.
|
| mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
| Memmapping mode for numpy arrays passed to workers. None will
| disable memmapping, other modes defined in the numpy.memmap doc:
| https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
| Also, see 'max_nbytes' parameter documentation for more details.
|
| prefer: str in {'processes', 'threads'} or None, default=None
| Soft hint to choose the default backend.
| The default process-based backend is 'loky' and the default
| thread-based backend is 'threading'. Ignored if the ``backend``
| parameter is specified.
|
| require: 'sharedmem' or None, default=None
| Hard constraint to select the backend. If set to 'sharedmem',
| the selected backend will be single-host and thread-based.
|
| inner_max_num_threads: int, default=None
| If not None, overwrites the limit set on the number of threads
| usable in some third-party library threadpools like OpenBLAS,
| MKL or OpenMP. This is only used with the ``loky`` backend.
|
| backend_params: dict
| Additional parameters to pass to the backend constructor when
| backend is a string.
|
| Notes
| -----
| Joblib tries to limit the oversubscription by limiting the number of
| threads usable in some third-party library threadpools like OpenBLAS, MKL
| or OpenMP. The default limit in each worker is set to
| ``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
| overwritten with the ``inner_max_num_threads`` argument which will be used
| to set this limit in the child processes.
|
| .. versionadded:: 1.3
|
| Examples
| --------
| >>> from operator import neg
| >>> with parallel_config(backend='threading'):
| ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
| ...
| [-1, -2, -3, -4, -5]
|
| To use the 'ray' joblib backend add the following lines:
|
| >>> from ray.util.joblib import register_ray # doctest: +SKIP
| >>> register_ray() # doctest: +SKIP
| >>> with parallel_config(backend="ray"): # doctest: +SKIP
| ... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
| [-1, -2, -3, -4, -5]
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, type, value, traceback)
|
| __init__(self, backend=default(None), *, n_jobs=default(None), verbose=default(0), temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None), inner_max_num_threads=None, **backend_params)
| Initialize self. See help(type(self)) for accurate signature.
|
| unregister(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
joblib.pool
Python Library Documentation: module joblib.pool in joblib
NAME
joblib.pool - Custom implementation of multiprocessing.Pool with custom pickler.
DESCRIPTION
This module provides efficient ways of working with data stored in
shared memory with numpy.memmap arrays without inducing any memory
copy between the parent and child processes.
This module should not be imported if multiprocessing is not
available as it implements subclasses of multiprocessing Pool
that uses a custom alternative to SimpleQueue.
CLASSES
_pickle.Pickler(builtins.object)
CustomizablePickler
builtins.object
CustomizablePicklingQueue
multiprocessing.pool.Pool(builtins.object)
PicklingPool
MemmappingPool
class CustomizablePickler(_pickle.Pickler)
| CustomizablePickler(writer, reducers=None, protocol=5)
|
| Pickler that accepts custom reducers.
|
| TODO python2_drop : can this be simplified ?
|
| HIGHEST_PROTOCOL is selected by default as this pickler is used
| to pickle ephemeral datastructures for interprocess communication
| hence no backward compatibility is required.
|
| `reducers` is expected to be a dictionary with key/values
| being `(type, callable)` pairs where `callable` is a function that
| give an instance of `type` will return a tuple `(constructor,
| tuple_of_objects)` to rebuild an instance out of the pickled
| `tuple_of_objects` as would return a `__reduce__` method. See the
| standard library documentation on pickling for more details.
|
| Method resolution order:
| CustomizablePickler
| _pickle.Pickler
| builtins.object
|
| Methods defined here:
|
| __init__(self, writer, reducers=None, protocol=5)
| Initialize self. See help(type(self)) for accurate signature.
|
| register(self, type, reduce_func)
| Attach a reducer function to a given type in the dispatch table.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from _pickle.Pickler:
|
| __sizeof__(self, /)
| Returns size in memory, in bytes.
|
| clear_memo(self, /)
| Clears the pickler's "memo".
|
| The memo is the data structure that remembers which objects the
| pickler has already seen, so that shared or recursive objects are
| pickled by reference and not by value. This method is useful when
| re-using picklers.
|
| dump(self, obj, /)
| Write a pickled representation of the given object to the open file.
|
| ----------------------------------------------------------------------
| Static methods inherited from _pickle.Pickler:
|
| __new__(*args, **kwargs) class method of _pickle.Pickler
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _pickle.Pickler:
|
| bin
|
| dispatch_table
|
| fast
|
| memo
|
| persistent_id
class CustomizablePicklingQueue(builtins.object)
| CustomizablePicklingQueue(context, reducers=None)
|
| Locked Pipe implementation that uses a customizable pickler.
|
| This class is an alternative to the multiprocessing implementation
| of SimpleQueue in order to make it possible to pass custom
| pickling reducers, for instance to avoid memory copy when passing
| memory mapped datastructures.
|
| `reducers` is expected to be a dict with key / values being
| `(type, callable)` pairs where `callable` is a function that, given an
| instance of `type`, will return a tuple `(constructor, tuple_of_objects)`
| to rebuild an instance out of the pickled `tuple_of_objects` as would
| return a `__reduce__` method.
|
| See the standard library documentation on pickling for more details.
|
| Methods defined here:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, context, reducers=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setstate__(self, state)
|
| empty(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class MemmappingPool(PicklingPool)
| MemmappingPool(processes=None, temp_folder=None, max_nbytes=1000000.0, mmap_mode='r', forward_reducers=None, backward_reducers=None, verbose=0, context_id=None, prewarm=False, **kwargs)
|
| Process pool that shares large arrays to avoid memory copy.
|
| This drop-in replacement for `multiprocessing.pool.Pool` makes
| it possible to work efficiently with shared memory in a numpy
| context.
|
| Existing instances of numpy.memmap are preserved: the child
| suprocesses will have access to the same shared memory in the
| original mode except for the 'w+' mode that is automatically
| transformed as 'r+' to avoid zeroing the original data upon
| instantiation.
|
| Furthermore large arrays from the parent process are automatically
| dumped to a temporary folder on the filesystem such as child
| processes to access their content via memmapping (file system
| backed shared memory).
|
| Note: it is important to call the terminate method to collect
| the temporary folder used by the pool.
|
| Parameters
| ----------
| processes: int, optional
| Number of worker processes running concurrently in the pool.
| initializer: callable, optional
| Callable executed on worker process creation.
| initargs: tuple, optional
| Arguments passed to the initializer callable.
| temp_folder: (str, callable) optional
| If str:
| Folder to be used by the pool for memmapping large arrays
| for sharing memory with worker processes. If None, this will try in
| order:
| - a folder pointed by the JOBLIB_TEMP_FOLDER environment variable,
| - /dev/shm if the folder exists and is writable: this is a RAMdisk
| filesystem available by default on modern Linux distributions,
| - the default system temporary folder that can be overridden
| with TMP, TMPDIR or TEMP environment variables, typically /tmp
| under Unix operating systems.
| if callable:
| An callable in charge of dynamically resolving a temporary folder
| for memmapping large arrays.
| max_nbytes int or None, optional, 1e6 by default
| Threshold on the size of arrays passed to the workers that
| triggers automated memory mapping in temp_folder.
| Use None to disable memmapping of large arrays.
| mmap_mode: {'r+', 'r', 'w+', 'c'}
| Memmapping mode for numpy arrays passed to workers.
| See 'max_nbytes' parameter documentation for more details.
| forward_reducers: dictionary, optional
| Reducers used to pickle objects passed from main process to worker
| processes: see below.
| backward_reducers: dictionary, optional
| Reducers used to pickle return values from workers back to the
| main process.
| verbose: int, optional
| Make it possible to monitor how the communication of numpy arrays
| with the subprocess is handled (pickling or memmapping)
| prewarm: bool or str, optional, "auto" by default.
| If True, force a read on newly memmapped array to make sure that OS
| pre-cache it in memory. This can be useful to avoid concurrent disk
| access when the same data array is passed to different worker
| processes. If "auto" (by default), prewarm is set to True, unless the
| Linux shared memory partition /dev/shm is available and used as temp
| folder.
|
| `forward_reducers` and `backward_reducers` are expected to be
| dictionaries with key/values being `(type, callable)` pairs where
| `callable` is a function that give an instance of `type` will return
| a tuple `(constructor, tuple_of_objects)` to rebuild an instance out
| of the pickled `tuple_of_objects` as would return a `__reduce__`
| method. See the standard library documentation on pickling for more
| details.
|
| Method resolution order:
| MemmappingPool
| PicklingPool
| multiprocessing.pool.Pool
| builtins.object
|
| Methods defined here:
|
| __init__(self, processes=None, temp_folder=None, max_nbytes=1000000.0, mmap_mode='r', forward_reducers=None, backward_reducers=None, verbose=0, context_id=None, prewarm=False, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| terminate(self)
|
| ----------------------------------------------------------------------
| Methods inherited from multiprocessing.pool.Pool:
|
| __del__(self, _warn=<built-in function warn>, RUN='RUN')
| # Copy globals as function locals to make sure that they are available
| # during Python shutdown when the Pool is destroyed.
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self)
| Return repr(self).
|
| apply(self, func, args=(), kwds={})
| Equivalent of `func(*args, **kwds)`.
| Pool must be running.
|
| apply_async(self, func, args=(), kwds={}, callback=None, error_callback=None)
| Asynchronous version of `apply()` method.
|
| close(self)
|
| imap(self, func, iterable, chunksize=1)
| Equivalent of `map()` -- can be MUCH slower than `Pool.map()`.
|
| imap_unordered(self, func, iterable, chunksize=1)
| Like `imap()` method but ordering of results is arbitrary.
|
| join(self)
|
| map(self, func, iterable, chunksize=None)
| Apply `func` to each element in `iterable`, collecting the results
| in a list that is returned.
|
| map_async(self, func, iterable, chunksize=None, callback=None, error_callback=None)
| Asynchronous version of `map()` method.
|
| starmap(self, func, iterable, chunksize=None)
| Like `map()` method but the elements of the `iterable` are expected to
| be iterables as well and will be unpacked as arguments. Hence
| `func` and (a, b) becomes func(a, b).
|
| starmap_async(self, func, iterable, chunksize=None, callback=None, error_callback=None)
| Asynchronous version of `starmap()` method.
|
| ----------------------------------------------------------------------
| Static methods inherited from multiprocessing.pool.Pool:
|
| Process(ctx, *args, **kwds)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from multiprocessing.pool.Pool:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class PicklingPool(multiprocessing.pool.Pool)
| PicklingPool(processes=None, forward_reducers=None, backward_reducers=None, **kwargs)
|
| Pool implementation with customizable pickling reducers.
|
| This is useful to control how data is shipped between processes
| and makes it possible to use shared memory without useless
| copies induces by the default pickling methods of the original
| objects passed as arguments to dispatch.
|
| `forward_reducers` and `backward_reducers` are expected to be
| dictionaries with key/values being `(type, callable)` pairs where
| `callable` is a function that, given an instance of `type`, will return a
| tuple `(constructor, tuple_of_objects)` to rebuild an instance out of the
| pickled `tuple_of_objects` as would return a `__reduce__` method.
| See the standard library documentation about pickling for more details.
|
| Method resolution order:
| PicklingPool
| multiprocessing.pool.Pool
| builtins.object
|
| Methods defined here:
|
| __init__(self, processes=None, forward_reducers=None, backward_reducers=None, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from multiprocessing.pool.Pool:
|
| __del__(self, _warn=<built-in function warn>, RUN='RUN')
| # Copy globals as function locals to make sure that they are available
| # during Python shutdown when the Pool is destroyed.
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self)
| Return repr(self).
|
| apply(self, func, args=(), kwds={})
| Equivalent of `func(*args, **kwds)`.
| Pool must be running.
|
| apply_async(self, func, args=(), kwds={}, callback=None, error_callback=None)
| Asynchronous version of `apply()` method.
|
| close(self)
|
| imap(self, func, iterable, chunksize=1)
| Equivalent of `map()` -- can be MUCH slower than `Pool.map()`.
|
| imap_unordered(self, func, iterable, chunksize=1)
| Like `imap()` method but ordering of results is arbitrary.
|
| join(self)
|
| map(self, func, iterable, chunksize=None)
| Apply `func` to each element in `iterable`, collecting the results
| in a list that is returned.
|
| map_async(self, func, iterable, chunksize=None, callback=None, error_callback=None)
| Asynchronous version of `map()` method.
|
| starmap(self, func, iterable, chunksize=None)
| Like `map()` method but the elements of the `iterable` are expected to
| be iterables as well and will be unpacked as arguments. Hence
| `func` and (a, b) becomes func(a, b).
|
| starmap_async(self, func, iterable, chunksize=None, callback=None, error_callback=None)
| Asynchronous version of `starmap()` method.
|
| terminate(self)
|
| ----------------------------------------------------------------------
| Static methods inherited from multiprocessing.pool.Pool:
|
| Process(ctx, *args, **kwds)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from multiprocessing.pool.Pool:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
sleep(...)
sleep(seconds)
Delay execution for a given number of seconds. The argument may be
a floating-point number for subsecond precision.
DATA
HIGHEST_PROTOCOL = 5
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\joblib\pool.py
joblib.register_compressor
Python Library Documentation: function register_compressor in module joblib.compressor
register_compressor(compressor_name, compressor, force=False)
Register a new compressor.
Parameters
----------
compressor_name: str.
The name of the compressor.
compressor: CompressorWrapper
An instance of a 'CompressorWrapper'.
joblib.register_parallel_backend
Python Library Documentation: function register_parallel_backend in module joblib.parallel
register_parallel_backend(name, factory, make_default=False)
Register a new Parallel backend factory.
The new backend can then be selected by passing its name as the backend
argument to the :class:`~Parallel` class. Moreover, the default backend can
be overwritten globally by setting make_default=True.
The factory can be any callable that takes no argument and return an
instance of ``ParallelBackendBase``.
Warning: this function is experimental and subject to change in a future
version of joblib.
.. versionadded:: 0.10
joblib.register_store_backend
Python Library Documentation: function register_store_backend in module joblib.memory
register_store_backend(backend_name, backend)
Extend available store backends.
The Memory, MemorizeResult and MemorizeFunc objects are designed to be
agnostic to the type of store used behind. By default, the local file
system is used but this function gives the possibility to extend joblib's
memory pattern with other types of storage such as cloud storage (S3, GCS,
OpenStack, HadoopFS, etc) or blob DBs.
Parameters
----------
backend_name: str
The name identifying the store backend being registered. For example,
'local' is used with FileSystemStoreBackend.
backend: StoreBackendBase subclass
The name of a class that implements the StoreBackendBase interface.
joblib.wrap_non_picklable_objects
Python Library Documentation: function wrap_non_picklable_objects in module joblib.externals.loky.cloudpickle_wrapper
wrap_non_picklable_objects(obj, keep_wrapper=True)
Wrapper for non-picklable object to use cloudpickle to serialize them.
Note that this wrapper tends to slow down the serialization process as it
is done with cloudpickle which is typically slower compared to pickle. The
proper way to solve serialization issues is to avoid defining functions and
objects in the main scripts and to implement __reduce__ functions for
complex classes.