dask API Reference
Full documentation for dask (2026.7.1)
dask.Expr
Python Library Documentation: class Expr in module dask._expr
class Expr(builtins.object)
| Expr(*args, _determ_token=None, **kwargs)
|
| Methods defined here:
|
| __dask_annotations__(self)
|
| __dask_graph__(self)
| Traverse expression tree, collect layers
|
| Subclasses generally do not want to override this method unless custom
| logic is required to treat (e.g. ignore) specific operands during graph
| generation.
|
| See also
| --------
| Expr._layer
| Expr._task
|
| __dask_keys__(self)
| The keys for this expression
|
| This is used to determine the keys of the output collection
| when this expression is computed.
|
| Returns
| -------
| keys: list
| The keys for this expression
|
| __dask_tokenize__(self)
|
| __getattr__(self, key)
|
| __hash__(self)
| Return hash(self).
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self)
| Return repr(self).
|
| __setattr__(self, name: 'str', value: 'Any') -> 'None'
| Implement setattr(self, name, value).
|
| __str__(self)
| Return str(self).
|
| analyze(self, filename: 'str | None' = None, format: 'str | None' = None) -> 'None'
|
| dependencies(self)
|
| explain(self, stage: 'OptimizerStage' = 'fused', format: 'str | None' = None) -> 'None'
|
| finalize_compute(self)
|
| find_operations(self, operation: 'type | tuple[type]') -> 'Generator[Expr]'
| Search the expression graph for a specific operation type
|
| Parameters
| ----------
| operation
| The operation type to search for.
|
| Returns
| -------
| nodes
| Generator of `operation` instances. Ordering corresponds
| to a depth-first search of the expression graph.
|
| fuse(self) -> 'Expr'
|
| lower_completely(self) -> 'Expr'
| Lower an expression completely
|
| This calls the ``lower_once`` method in a loop
| until nothing changes. This function does not
| apply any other optimizations (like ``simplify``).
|
| Returns
| -------
| expr:
| output expression
|
| See Also
| --------
| Expr.lower_once
| Expr._lower
|
| lower_once(self, lowered: 'dict')
|
| operand(self, key)
|
| optimize(self, fuse: 'bool' = False) -> 'Expr'
|
| pprint(self)
|
| rewrite(self, kind: 'str', rewritten)
| Rewrite an expression
|
| This leverages the ``._{kind}_down`` and ``._{kind}_up``
| methods defined on each class
|
| Returns
| -------
| expr:
| output expression
| changed:
| whether or not any change occurred
|
| simplify(self) -> 'Expr'
|
| simplify_once(self, dependents: 'defaultdict', simplified: 'dict')
| Simplify an expression
|
| This leverages the ``._simplify_down`` and ``._simplify_up``
| methods defined on each class
|
| Parameters
| ----------
|
| dependents: defaultdict[list]
| The dependents for every node.
| simplified: dict
| Cache of simplified expressions for these dependents.
|
| Returns
| -------
| expr:
| output expression
|
| substitute(self, old, new) -> 'Expr'
| Substitute a specific term within the expression
|
| Note that replacing non-`Expr` terms may produce
| unexpected results, and is not recommended.
| Substituting boolean values is not allowed.
|
| Parameters
| ----------
| old:
| Old term to find and replace.
| new:
| New term to replace instances of `old` with.
|
| Examples
| --------
| >>> (df + 10).substitute(10, 20) # doctest: +SKIP
| df + 20
|
| substitute_parameters(self, substitutions: 'dict') -> 'Expr'
| Substitute specific `Expr` parameters
|
| Parameters
| ----------
| substitutions:
| Mapping of parameter keys to new values. Keys that
| are not found in ``self._parameters`` will be ignored.
|
| tree_repr(self)
|
| visualize(self, filename='dask-expr.svg', format=None, **kwargs)
| Visualize the expression graph.
| Requires ``graphviz`` to be installed.
|
| Parameters
| ----------
| filename : str or None, optional
| The name of the file to write to disk. If the provided `filename`
| doesn't include an extension, '.png' will be used by default.
| If `filename` is None, no file will be written, and the graph is
| rendered in the Jupyter notebook only.
| format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
| Format in which to write output file. Default is 'svg'.
| **kwargs
| Additional keyword arguments to forward to ``to_graphviz``.
|
| walk(self) -> 'Generator[Expr]'
| Iterate through all expressions in the tree
|
| Returns
| -------
| nodes
| Generator of Expr instances in the graph.
| Ordering is a depth-first search of the expression tree
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(cls, *args, _determ_token=None, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| dask
|
| deterministic_token
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_defaults': 'dict[str, Any]', '_determ_token': 'st...
dask.HLGExpr
Python Library Documentation: class HLGExpr in module dask._expr
class HLGExpr(Expr)
| HLGExpr(*args, _determ_token=None, **kwargs)
|
| Method resolution order:
| HLGExpr
| Expr
| builtins.object
|
| Methods defined here:
|
| __dask_annotations__(self) -> 'dict[str, dict[Key, object]]'
|
| __dask_keys__(self)
| The keys for this expression
|
| This is used to determine the keys of the output collection
| when this expression is computed.
|
| Returns
| -------
| keys: list
| The keys for this expression
|
| finalize_compute(self)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| from_collection(collection, optimize_graph=True)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| deterministic_token
|
| hlg
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from Expr:
|
| __dask_graph__(self)
| Traverse expression tree, collect layers
|
| Subclasses generally do not want to override this method unless custom
| logic is required to treat (e.g. ignore) specific operands during graph
| generation.
|
| See also
| --------
| Expr._layer
| Expr._task
|
| __dask_tokenize__(self)
|
| __getattr__(self, key)
|
| __hash__(self)
| Return hash(self).
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self)
| Return repr(self).
|
| __setattr__(self, name: 'str', value: 'Any') -> 'None'
| Implement setattr(self, name, value).
|
| __str__(self)
| Return str(self).
|
| analyze(self, filename: 'str | None' = None, format: 'str | None' = None) -> 'None'
|
| dependencies(self)
|
| explain(self, stage: 'OptimizerStage' = 'fused', format: 'str | None' = None) -> 'None'
|
| find_operations(self, operation: 'type | tuple[type]') -> 'Generator[Expr]'
| Search the expression graph for a specific operation type
|
| Parameters
| ----------
| operation
| The operation type to search for.
|
| Returns
| -------
| nodes
| Generator of `operation` instances. Ordering corresponds
| to a depth-first search of the expression graph.
|
| fuse(self) -> 'Expr'
|
| lower_completely(self) -> 'Expr'
| Lower an expression completely
|
| This calls the ``lower_once`` method in a loop
| until nothing changes. This function does not
| apply any other optimizations (like ``simplify``).
|
| Returns
| -------
| expr:
| output expression
|
| See Also
| --------
| Expr.lower_once
| Expr._lower
|
| lower_once(self, lowered: 'dict')
|
| operand(self, key)
|
| optimize(self, fuse: 'bool' = False) -> 'Expr'
|
| pprint(self)
|
| rewrite(self, kind: 'str', rewritten)
| Rewrite an expression
|
| This leverages the ``._{kind}_down`` and ``._{kind}_up``
| methods defined on each class
|
| Returns
| -------
| expr:
| output expression
| changed:
| whether or not any change occurred
|
| simplify(self) -> 'Expr'
|
| simplify_once(self, dependents: 'defaultdict', simplified: 'dict')
| Simplify an expression
|
| This leverages the ``._simplify_down`` and ``._simplify_up``
| methods defined on each class
|
| Parameters
| ----------
|
| dependents: defaultdict[list]
| The dependents for every node.
| simplified: dict
| Cache of simplified expressions for these dependents.
|
| Returns
| -------
| expr:
| output expression
|
| substitute(self, old, new) -> 'Expr'
| Substitute a specific term within the expression
|
| Note that replacing non-`Expr` terms may produce
| unexpected results, and is not recommended.
| Substituting boolean values is not allowed.
|
| Parameters
| ----------
| old:
| Old term to find and replace.
| new:
| New term to replace instances of `old` with.
|
| Examples
| --------
| >>> (df + 10).substitute(10, 20) # doctest: +SKIP
| df + 20
|
| substitute_parameters(self, substitutions: 'dict') -> 'Expr'
| Substitute specific `Expr` parameters
|
| Parameters
| ----------
| substitutions:
| Mapping of parameter keys to new values. Keys that
| are not found in ``self._parameters`` will be ignored.
|
| tree_repr(self)
|
| visualize(self, filename='dask-expr.svg', format=None, **kwargs)
| Visualize the expression graph.
| Requires ``graphviz`` to be installed.
|
| Parameters
| ----------
| filename : str or None, optional
| The name of the file to write to disk. If the provided `filename`
| doesn't include an extension, '.png' will be used by default.
| If `filename` is None, no file will be written, and the graph is
| rendered in the Jupyter notebook only.
| format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
| Format in which to write output file. Default is 'svg'.
| **kwargs
| Additional keyword arguments to forward to ``to_graphviz``.
|
| walk(self) -> 'Generator[Expr]'
| Iterate through all expressions in the tree
|
| Returns
| -------
| nodes
| Generator of Expr instances in the graph.
| Ordering is a depth-first search of the expression tree
|
| ----------------------------------------------------------------------
| Static methods inherited from Expr:
|
| __new__(cls, *args, _determ_token=None, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from Expr:
|
| dask
|
| ----------------------------------------------------------------------
| Data descriptors inherited from Expr:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
dask.LLGExpr
Python Library Documentation: class LLGExpr in module dask._expr
class LLGExpr(Expr)
| LLGExpr(*args, _determ_token=None, **kwargs)
|
| Low Level Graph Expression
|
| Method resolution order:
| LLGExpr
| Expr
| builtins.object
|
| Methods defined here:
|
| __dask_keys__(self)
| The keys for this expression
|
| This is used to determine the keys of the output collection
| when this expression is computed.
|
| Returns
| -------
| keys: list
| The keys for this expression
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from Expr:
|
| __dask_annotations__(self)
|
| __dask_graph__(self)
| Traverse expression tree, collect layers
|
| Subclasses generally do not want to override this method unless custom
| logic is required to treat (e.g. ignore) specific operands during graph
| generation.
|
| See also
| --------
| Expr._layer
| Expr._task
|
| __dask_tokenize__(self)
|
| __getattr__(self, key)
|
| __hash__(self)
| Return hash(self).
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self)
| Return repr(self).
|
| __setattr__(self, name: 'str', value: 'Any') -> 'None'
| Implement setattr(self, name, value).
|
| __str__(self)
| Return str(self).
|
| analyze(self, filename: 'str | None' = None, format: 'str | None' = None) -> 'None'
|
| dependencies(self)
|
| explain(self, stage: 'OptimizerStage' = 'fused', format: 'str | None' = None) -> 'None'
|
| finalize_compute(self)
|
| find_operations(self, operation: 'type | tuple[type]') -> 'Generator[Expr]'
| Search the expression graph for a specific operation type
|
| Parameters
| ----------
| operation
| The operation type to search for.
|
| Returns
| -------
| nodes
| Generator of `operation` instances. Ordering corresponds
| to a depth-first search of the expression graph.
|
| fuse(self) -> 'Expr'
|
| lower_completely(self) -> 'Expr'
| Lower an expression completely
|
| This calls the ``lower_once`` method in a loop
| until nothing changes. This function does not
| apply any other optimizations (like ``simplify``).
|
| Returns
| -------
| expr:
| output expression
|
| See Also
| --------
| Expr.lower_once
| Expr._lower
|
| lower_once(self, lowered: 'dict')
|
| operand(self, key)
|
| optimize(self, fuse: 'bool' = False) -> 'Expr'
|
| pprint(self)
|
| rewrite(self, kind: 'str', rewritten)
| Rewrite an expression
|
| This leverages the ``._{kind}_down`` and ``._{kind}_up``
| methods defined on each class
|
| Returns
| -------
| expr:
| output expression
| changed:
| whether or not any change occurred
|
| simplify(self) -> 'Expr'
|
| simplify_once(self, dependents: 'defaultdict', simplified: 'dict')
| Simplify an expression
|
| This leverages the ``._simplify_down`` and ``._simplify_up``
| methods defined on each class
|
| Parameters
| ----------
|
| dependents: defaultdict[list]
| The dependents for every node.
| simplified: dict
| Cache of simplified expressions for these dependents.
|
| Returns
| -------
| expr:
| output expression
|
| substitute(self, old, new) -> 'Expr'
| Substitute a specific term within the expression
|
| Note that replacing non-`Expr` terms may produce
| unexpected results, and is not recommended.
| Substituting boolean values is not allowed.
|
| Parameters
| ----------
| old:
| Old term to find and replace.
| new:
| New term to replace instances of `old` with.
|
| Examples
| --------
| >>> (df + 10).substitute(10, 20) # doctest: +SKIP
| df + 20
|
| substitute_parameters(self, substitutions: 'dict') -> 'Expr'
| Substitute specific `Expr` parameters
|
| Parameters
| ----------
| substitutions:
| Mapping of parameter keys to new values. Keys that
| are not found in ``self._parameters`` will be ignored.
|
| tree_repr(self)
|
| visualize(self, filename='dask-expr.svg', format=None, **kwargs)
| Visualize the expression graph.
| Requires ``graphviz`` to be installed.
|
| Parameters
| ----------
| filename : str or None, optional
| The name of the file to write to disk. If the provided `filename`
| doesn't include an extension, '.png' will be used by default.
| If `filename` is None, no file will be written, and the graph is
| rendered in the Jupyter notebook only.
| format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
| Format in which to write output file. Default is 'svg'.
| **kwargs
| Additional keyword arguments to forward to ``to_graphviz``.
|
| walk(self) -> 'Generator[Expr]'
| Iterate through all expressions in the tree
|
| Returns
| -------
| nodes
| Generator of Expr instances in the graph.
| Ordering is a depth-first search of the expression tree
|
| ----------------------------------------------------------------------
| Static methods inherited from Expr:
|
| __new__(cls, *args, _determ_token=None, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from Expr:
|
| dask
|
| deterministic_token
|
| ----------------------------------------------------------------------
| Data descriptors inherited from Expr:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
dask.SingletonExpr
Python Library Documentation: class SingletonExpr in module dask._expr
class SingletonExpr(Expr)
| SingletonExpr(*args, _determ_token=None, **kwargs)
|
| A singleton Expr class
|
| This is used to treat the subclassed expression as a singleton. Singletons
| are deduplicated by expr._name which is typically based on the dask.tokenize
| output.
|
| This is a crucial performance optimization for expressions that walk through
| an optimizer and are recreated repeatedly but isn't safe for objects that
| cannot be reliably or quickly tokenized.
|
| Method resolution order:
| SingletonExpr
| Expr
| builtins.object
|
| Static methods defined here:
|
| __new__(cls, *args, _determ_token=None, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_instances': 'weakref.WeakValueDictionary[str, Sin...
|
| ----------------------------------------------------------------------
| Methods inherited from Expr:
|
| __dask_annotations__(self)
|
| __dask_graph__(self)
| Traverse expression tree, collect layers
|
| Subclasses generally do not want to override this method unless custom
| logic is required to treat (e.g. ignore) specific operands during graph
| generation.
|
| See also
| --------
| Expr._layer
| Expr._task
|
| __dask_keys__(self)
| The keys for this expression
|
| This is used to determine the keys of the output collection
| when this expression is computed.
|
| Returns
| -------
| keys: list
| The keys for this expression
|
| __dask_tokenize__(self)
|
| __getattr__(self, key)
|
| __hash__(self)
| Return hash(self).
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self)
| Return repr(self).
|
| __setattr__(self, name: 'str', value: 'Any') -> 'None'
| Implement setattr(self, name, value).
|
| __str__(self)
| Return str(self).
|
| analyze(self, filename: 'str | None' = None, format: 'str | None' = None) -> 'None'
|
| dependencies(self)
|
| explain(self, stage: 'OptimizerStage' = 'fused', format: 'str | None' = None) -> 'None'
|
| finalize_compute(self)
|
| find_operations(self, operation: 'type | tuple[type]') -> 'Generator[Expr]'
| Search the expression graph for a specific operation type
|
| Parameters
| ----------
| operation
| The operation type to search for.
|
| Returns
| -------
| nodes
| Generator of `operation` instances. Ordering corresponds
| to a depth-first search of the expression graph.
|
| fuse(self) -> 'Expr'
|
| lower_completely(self) -> 'Expr'
| Lower an expression completely
|
| This calls the ``lower_once`` method in a loop
| until nothing changes. This function does not
| apply any other optimizations (like ``simplify``).
|
| Returns
| -------
| expr:
| output expression
|
| See Also
| --------
| Expr.lower_once
| Expr._lower
|
| lower_once(self, lowered: 'dict')
|
| operand(self, key)
|
| optimize(self, fuse: 'bool' = False) -> 'Expr'
|
| pprint(self)
|
| rewrite(self, kind: 'str', rewritten)
| Rewrite an expression
|
| This leverages the ``._{kind}_down`` and ``._{kind}_up``
| methods defined on each class
|
| Returns
| -------
| expr:
| output expression
| changed:
| whether or not any change occurred
|
| simplify(self) -> 'Expr'
|
| simplify_once(self, dependents: 'defaultdict', simplified: 'dict')
| Simplify an expression
|
| This leverages the ``._simplify_down`` and ``._simplify_up``
| methods defined on each class
|
| Parameters
| ----------
|
| dependents: defaultdict[list]
| The dependents for every node.
| simplified: dict
| Cache of simplified expressions for these dependents.
|
| Returns
| -------
| expr:
| output expression
|
| substitute(self, old, new) -> 'Expr'
| Substitute a specific term within the expression
|
| Note that replacing non-`Expr` terms may produce
| unexpected results, and is not recommended.
| Substituting boolean values is not allowed.
|
| Parameters
| ----------
| old:
| Old term to find and replace.
| new:
| New term to replace instances of `old` with.
|
| Examples
| --------
| >>> (df + 10).substitute(10, 20) # doctest: +SKIP
| df + 20
|
| substitute_parameters(self, substitutions: 'dict') -> 'Expr'
| Substitute specific `Expr` parameters
|
| Parameters
| ----------
| substitutions:
| Mapping of parameter keys to new values. Keys that
| are not found in ``self._parameters`` will be ignored.
|
| tree_repr(self)
|
| visualize(self, filename='dask-expr.svg', format=None, **kwargs)
| Visualize the expression graph.
| Requires ``graphviz`` to be installed.
|
| Parameters
| ----------
| filename : str or None, optional
| The name of the file to write to disk. If the provided `filename`
| doesn't include an extension, '.png' will be used by default.
| If `filename` is None, no file will be written, and the graph is
| rendered in the Jupyter notebook only.
| format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
| Format in which to write output file. Default is 'svg'.
| **kwargs
| Additional keyword arguments to forward to ``to_graphviz``.
|
| walk(self) -> 'Generator[Expr]'
| Iterate through all expressions in the tree
|
| Returns
| -------
| nodes
| Generator of Expr instances in the graph.
| Ordering is a depth-first search of the expression tree
|
| ----------------------------------------------------------------------
| Readonly properties inherited from Expr:
|
| dask
|
| deterministic_token
|
| ----------------------------------------------------------------------
| Data descriptors inherited from Expr:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
dask.annotate
Python Library Documentation: function annotate in module dask.base
annotate(**annotations: 'Any') -> 'Iterator[None]'
Context Manager for setting HighLevelGraph Layer annotations.
Annotations are metadata or soft constraints associated with
tasks that dask schedulers may choose to respect: They signal intent
without enforcing hard constraints. As such, they are
primarily designed for use with the distributed scheduler.
Almost any object can serve as an annotation, but small Python objects
are preferred, while large objects such as NumPy arrays are discouraged.
Callables supplied as an annotation should take a single *key* argument and
produce the appropriate annotation. Individual task keys in the annotated collection
are supplied to the callable.
Parameters
----------
**annotations : key-value pairs
Examples
--------
All tasks within array A should have priority 100 and be retried 3 times
on failure.
>>> import dask
>>> import dask.array as da
>>> with dask.annotate(priority=100, retries=3):
... A = da.ones((10000, 10000))
Prioritise tasks within Array A on flattened block ID.
>>> nblocks = (10, 10)
>>> with dask.annotate(priority=lambda k: k[1]*nblocks[1] + k[2]):
... A = da.ones((1000, 1000), chunks=(100, 100))
Annotations may be nested.
>>> with dask.annotate(priority=1):
... with dask.annotate(retries=3):
... A = da.ones((1000, 1000))
... B = A + 1
See Also
--------
get_annotations
dask.annotations
Python Library Documentation: _Feature in module __future__ object class _Feature(builtins.object) | _Feature(optionalRelease, mandatoryRelease, compiler_flag) | | Methods defined here: | | __init__(self, optionalRelease, mandatoryRelease, compiler_flag) | Initialize self. See help(type(self)) for accurate signature. | | __repr__(self) | Return repr(self). | | getMandatoryRelease(self) | Return release in which this feature will become mandatory. | | This is a 5-tuple, of the same form as sys.version_info, or, if | the feature was dropped, or the release date is undetermined, is None. | | getOptionalRelease(self) | Return first release in which this feature was recognized. | | This is a 5-tuple, of the same form as sys.version_info. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
dask.base
Python Library Documentation: module dask.base in dask
NAME
dask.base
CLASSES
builtins.object
DaskMethodsMixin
class DaskMethodsMixin(builtins.object)
| A mixin adding standard dask collection methods
|
| Methods defined here:
|
| __await__(self)
|
| compute(self, **kwargs)
| Compute this dask collection
|
| This turns a lazy Dask collection into its in-memory equivalent.
| For example a Dask array turns into a NumPy array and a Dask dataframe
| turns into a Pandas dataframe. The entire dataset must fit into memory
| before calling this operation.
|
| Parameters
| ----------
| scheduler : string, optional
| Which scheduler to use like "threads", "synchronous" or "processes".
| If not provided, the default is to check the global settings first,
| and then fall back to the collection defaults.
| optimize_graph : bool, optional
| If True [default], the graph is optimized before computation.
| Otherwise the graph is run as is. This can be useful for debugging.
| kwargs
| Extra keywords to forward to the scheduler function.
|
| See Also
| --------
| dask.compute
|
| persist(self, **kwargs)
| Persist this dask collection into memory
|
| This turns a lazy Dask collection into a Dask collection with the same
| metadata, but now with the results fully computed or actively computing
| in the background.
|
| The action of function differs significantly depending on the active
| task scheduler. If the task scheduler supports asynchronous computing,
| such as is the case of the dask.distributed scheduler, then persist
| will return *immediately* and the return value's task graph will
| contain Dask Future objects. However if the task scheduler only
| supports blocking computation then the call to persist will *block*
| and the return value's task graph will contain concrete Python results.
|
| This function is particularly useful when using distributed systems,
| because the results will be kept in distributed memory, rather than
| returned to the local process as with compute.
|
| Parameters
| ----------
| scheduler : string, optional
| Which scheduler to use like "threads", "synchronous" or "processes".
| If not provided, the default is to check the global settings first,
| and then fall back to the collection defaults.
| optimize_graph : bool, optional
| If True [default], the graph is optimized before computation.
| Otherwise the graph is run as is. This can be useful for debugging.
| **kwargs
| Extra keywords to forward to the scheduler function.
|
| Returns
| -------
| New dask collections backed by in-memory data
|
| See Also
| --------
| dask.persist
|
| visualize(self, filename='mydask', format=None, optimize_graph=False, **kwargs)
| Render the computation of this object's task graph using graphviz.
|
| Requires ``graphviz`` to be installed.
|
| Parameters
| ----------
| filename : str or None, optional
| The name of the file to write to disk. If the provided `filename`
| doesn't include an extension, '.png' will be used by default.
| If `filename` is None, no file will be written, and we communicate
| with dot using only pipes.
| format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
| Format in which to write output file. Default is 'png'.
| optimize_graph : bool, optional
| If True, the graph is optimized before rendering. Otherwise,
| the graph is displayed as is. Default is False.
| color: {None, 'order'}, optional
| Options to color nodes. Provide ``cmap=`` keyword for additional
| colormap
| **kwargs
| Additional keyword arguments to forward to ``to_graphviz``.
|
| Examples
| --------
| >>> x.visualize(filename='dask.pdf') # doctest: +SKIP
| >>> x.visualize(filename='dask.pdf', color='order') # doctest: +SKIP
|
| Returns
| -------
| result : IPython.display.Image, IPython.display.SVG, or None
| See dask.dot.dot_graph for more information.
|
| See Also
| --------
| dask.visualize
| dask.dot.dot_graph
|
| Notes
| -----
| For more information on optimization see here:
|
| https://docs.dask.org/en/latest/optimize.html
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
FUNCTIONS
annotate(**annotations: 'Any') -> 'Iterator[None]'
Context Manager for setting HighLevelGraph Layer annotations.
Annotations are metadata or soft constraints associated with
tasks that dask schedulers may choose to respect: They signal intent
without enforcing hard constraints. As such, they are
primarily designed for use with the distributed scheduler.
Almost any object can serve as an annotation, but small Python objects
are preferred, while large objects such as NumPy arrays are discouraged.
Callables supplied as an annotation should take a single *key* argument and
produce the appropriate annotation. Individual task keys in the annotated collection
are supplied to the callable.
Parameters
----------
**annotations : key-value pairs
Examples
--------
All tasks within array A should have priority 100 and be retried 3 times
on failure.
>>> import dask
>>> import dask.array as da
>>> with dask.annotate(priority=100, retries=3):
... A = da.ones((10000, 10000))
Prioritise tasks within Array A on flattened block ID.
>>> nblocks = (10, 10)
>>> with dask.annotate(priority=lambda k: k[1]*nblocks[1] + k[2]):
... A = da.ones((1000, 1000), chunks=(100, 100))
Annotations may be nested.
>>> with dask.annotate(priority=1):
... with dask.annotate(retries=3):
... A = da.ones((1000, 1000))
... B = A + 1
See Also
--------
get_annotations
clone_key(key: 'KeyOrStrT', seed: 'Hashable') -> 'KeyOrStrT'
Clone a key from a Dask collection, producing a new key with the same prefix and
indices and a token which is a deterministic function of the previous key and seed.
Examples
--------
>>> clone_key("x", 123) # doctest: +SKIP
'x-c4fb64ccca807af85082413d7ef01721'
>>> clone_key("inc-cbb1eca3bafafbb3e8b2419c4eebb387", 123) # doctest: +SKIP
'inc-bc629c23014a4472e18b575fdaf29ee7'
>>> clone_key(("sum-cbb1eca3bafafbb3e8b2419c4eebb387", 4, 3), 123) # doctest: +SKIP
('sum-c053f3774e09bd0f7de6044dbc40e71d', 4, 3)
compute(*args, traverse=True, optimize_graph=True, scheduler=None, get=None, **kwargs)
Compute several dask collections at once.
Parameters
----------
args : object
Any number of objects. If it is a dask object, it's computed and the
result is returned. By default, python builtin collections are also
traversed to look for dask objects (for more information see the
``traverse`` keyword). Non-dask arguments are passed through unchanged.
traverse : bool, optional
By default dask traverses builtin python collections looking for dask
objects passed to ``compute``. For large collections this can be
expensive. If none of the arguments contain any dask objects, set
``traverse=False`` to avoid doing this traversal.
scheduler : string, optional
Which scheduler to use like "threads", "synchronous" or "processes".
If not provided, the default is to check the global settings first,
and then fall back to the collection defaults.
optimize_graph : bool, optional
If True [default], the optimizations for each collection are applied
before computation. Otherwise the graph is run as is. This can be
useful for debugging.
get : ``None``
Should be left to ``None`` The get= keyword has been removed.
kwargs
Extra keywords to forward to the scheduler function.
Examples
--------
>>> import dask
>>> import dask.array as da
>>> a = da.arange(10, chunks=2).sum()
>>> b = da.arange(10, chunks=2).mean()
>>> dask.compute(a, b)
(np.int64(45), np.float64(4.5))
By default, dask objects inside python collections will also be computed:
>>> dask.compute({'a': a, 'b': b, 'c': 1})
({'a': np.int64(45), 'b': np.float64(4.5), 'c': 1},)
get_annotations() -> 'dict[str, Any]'
Get current annotations.
Returns
-------
Dict of all current annotations
See Also
--------
annotate
get_collection_names(collection) -> 'set[str]'
Infer the collection names from the dask keys, under the assumption that all keys
are either tuples with matching first element, and that element is a string, or
there is exactly one key and it is a string.
Examples
--------
>>> a.__dask_keys__() # doctest: +SKIP
["foo", "bar"]
>>> get_collection_names(a) # doctest: +SKIP
{"foo", "bar"}
>>> b.__dask_keys__() # doctest: +SKIP
[[("foo-123", 0, 0), ("foo-123", 0, 1)], [("foo-123", 1, 0), ("foo-123", 1, 1)]]
>>> get_collection_names(b) # doctest: +SKIP
{"foo-123"}
get_name_from_key(key: 'Key') -> 'str'
Given a dask collection's key, extract the collection name.
Parameters
----------
key: string or tuple
Dask collection's key, which must be either a single string or a tuple whose
first element is a string (commonly referred to as a collection's 'name'),
Examples
--------
>>> get_name_from_key("foo")
'foo'
>>> get_name_from_key(("foo-123", 1, 2))
'foo-123'
is_dask_collection(x) -> 'bool'
Returns ``True`` if ``x`` is a dask collection.
Parameters
----------
x : Any
Object to test.
Returns
-------
result : bool
``True`` if `x` is a Dask collection.
Notes
-----
The DaskCollection typing.Protocol implementation defines a Dask
collection as a class that returns a Mapping from the
``__dask_graph__`` method. This helper function existed before the
implementation of the protocol.
optimize(*args, traverse=True, **kwargs)
Optimize several dask collections at once.
Returns equivalent dask collections that all share the same merged and
optimized underlying graph. This can be useful if converting multiple
collections to delayed objects, or to manually apply the optimizations at
strategic points.
Note that in most cases you shouldn't need to call this function directly.
Warning::
This function triggers a materialization of the collections and looses
any annotations attached to HLG layers.
Parameters
----------
*args : objects
Any number of objects. If a dask object, its graph is optimized and
merged with all those of all other dask objects before returning an
equivalent dask collection. Non-dask arguments are passed through
unchanged.
traverse : bool, optional
By default dask traverses builtin python collections looking for dask
objects passed to ``optimize``. For large collections this can be
expensive. If none of the arguments contain any dask objects, set
``traverse=False`` to avoid doing this traversal.
optimizations : list of callables, optional
Additional optimization passes to perform.
**kwargs
Extra keyword arguments to forward to the optimization passes.
Examples
--------
>>> import dask
>>> import dask.array as da
>>> a = da.arange(10, chunks=2).sum()
>>> b = da.arange(10, chunks=2).mean()
>>> a2, b2 = dask.optimize(a, b)
>>> a2.compute() == a.compute()
np.True_
>>> b2.compute() == b.compute()
np.True_
persist(*args, traverse=True, optimize_graph=True, scheduler=None, **kwargs)
Persist multiple Dask collections into memory
This turns lazy Dask collections into Dask collections with the same
metadata, but now with their results fully computed or actively computing
in the background.
For example a lazy dask.array built up from many lazy calls will now be a
dask.array of the same shape, dtype, chunks, etc., but now with all of
those previously lazy tasks either computed in memory as many small :class:`numpy.array`
(in the single-machine case) or asynchronously running in the
background on a cluster (in the distributed case).
This function operates differently if a ``dask.distributed.Client`` exists
and is connected to a distributed scheduler. In this case this function
will return as soon as the task graph has been submitted to the cluster,
but before the computations have completed. Computations will continue
asynchronously in the background. When using this function with the single
machine scheduler it blocks until the computations have finished.
When using Dask on a single machine you should ensure that the dataset fits
entirely within memory.
Examples
--------
>>> df = dd.read_csv('/path/to/*.csv') # doctest: +SKIP
>>> df = df[df.name == 'Alice'] # doctest: +SKIP
>>> df['in-debt'] = df.balance < 0 # doctest: +SKIP
>>> df = df.persist() # triggers computation # doctest: +SKIP
>>> df.value().min() # future computations are now fast # doctest: +SKIP
-10
>>> df.value().max() # doctest: +SKIP
100
>>> from dask import persist # use persist function on multiple collections
>>> a, b = persist(a, b) # doctest: +SKIP
Parameters
----------
*args: Dask collections
scheduler : string, optional
Which scheduler to use like "threads", "synchronous" or "processes".
If not provided, the default is to check the global settings first,
and then fall back to the collection defaults.
traverse : bool, optional
By default dask traverses builtin python collections looking for dask
objects passed to ``persist``. For large collections this can be
expensive. If none of the arguments contain any dask objects, set
``traverse=False`` to avoid doing this traversal.
optimize_graph : bool, optional
If True [default], the graph is optimized before computation.
Otherwise the graph is run as is. This can be useful for debugging.
**kwargs
Extra keywords to forward to the scheduler function.
Returns
-------
New dask collections backed by in-memory data
replace_name_in_key(key: 'KeyOrStrT', rename: 'Mapping[str, str]') -> 'KeyOrStrT'
Given a dask collection's key, replace the collection name with a new one.
Parameters
----------
key: string or tuple
Dask collection's key, which must be either a single string or a tuple whose
first element is a string (commonly referred to as a collection's 'name'),
rename:
Mapping of zero or more names from : to. Extraneous names will be ignored.
Names not found in this mapping won't be replaced.
Examples
--------
>>> replace_name_in_key("foo", {})
'foo'
>>> replace_name_in_key("foo", {"foo": "bar"})
'bar'
>>> replace_name_in_key(("foo-123", 1, 2), {"foo-123": "bar-456"})
('bar-456', 1, 2)
tokenize(*args: 'object', ensure_deterministic: 'bool | None' = None, **kwargs: 'object') -> 'str'
Deterministic token
>>> tokenize([1, 2, '3']) # doctest: +SKIP
'06961e8de572e73c2e74b51348177918'
>>> tokenize('Hello') == tokenize('Hello')
True
Parameters
----------
args, kwargs:
objects to tokenize
ensure_deterministic: bool, optional
If True, raise TokenizationError if the objects cannot be deterministically
tokenized, e.g. two identical objects will return different tokens.
Defaults to the `tokenize.ensure-deterministic` configuration parameter.
visualize(*args, filename='mydask', traverse=True, optimize_graph=False, maxval=None, engine: "Literal['cytoscape', 'ipycytoscape', 'graphviz'] | None" = None, **kwargs)
Visualize several dask graphs simultaneously.
Requires ``graphviz`` to be installed. All options that are not the dask
graph(s) should be passed as keyword arguments.
Parameters
----------
args : object
Any number of objects. If it is a dask collection (for example, a
dask DataFrame, Array, Bag, or Delayed), its associated graph
will be included in the output of visualize. By default, python builtin
collections are also traversed to look for dask objects (for more
information see the ``traverse`` keyword). Arguments lacking an
associated graph will be ignored.
filename : str or None, optional
The name of the file to write to disk. If the provided `filename`
doesn't include an extension, '.png' will be used by default.
If `filename` is None, no file will be written, and we communicate
with dot using only pipes.
format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
Format in which to write output file. Default is 'png'.
traverse : bool, optional
By default, dask traverses builtin python collections looking for dask
objects passed to ``visualize``. For large collections this can be
expensive. If none of the arguments contain any dask objects, set
``traverse=False`` to avoid doing this traversal.
optimize_graph : bool, optional
If True, the graph is optimized before rendering. Otherwise,
the graph is displayed as is. Default is False.
color : {None, 'order', 'ages', 'freed', 'memoryincreases', 'memorydecreases', 'memorypressure'}, optional
Options to color nodes. colormap:
- None, the default, no colors.
- 'order', colors the nodes' border based on the order they appear in the graph.
- 'ages', how long the data of a node is held.
- 'freed', the number of dependencies released after running a node.
- 'memoryincreases', how many more outputs are held after the lifetime of a node.
Large values may indicate nodes that should have run later.
- 'memorydecreases', how many fewer outputs are held after the lifetime of a node.
Large values may indicate nodes that should have run sooner.
- 'memorypressure', the number of data held when the node is run (circle), or
the data is released (rectangle).
maxval : {int, float}, optional
Maximum value for colormap to normalize form 0 to 1.0. Default is ``None``
will make it the max number of values
collapse_outputs : bool, optional
Whether to collapse output boxes, which often have empty labels.
Default is False.
verbose : bool, optional
Whether to label output and input boxes even if the data aren't chunked.
Beware: these labels can get very long. Default is False.
engine : {"graphviz", "ipycytoscape", "cytoscape"}, optional.
The visualization engine to use. If not provided, this checks the dask config
value "visualization.engine". If that is not set, it tries to import ``graphviz``
and ``ipycytoscape``, using the first one to succeed.
**kwargs
Additional keyword arguments to forward to the visualization engine.
Examples
--------
>>> x.visualize(filename='dask.pdf') # doctest: +SKIP
>>> x.visualize(filename='dask.pdf', color='order') # doctest: +SKIP
Returns
-------
result : IPython.display.Image, IPython.display.SVG, or None
See dask.dot.dot_graph for more information.
See Also
--------
dask.dot.dot_graph
Notes
-----
For more information on optimization see here:
https://docs.dask.org/en/latest/optimize.html
DATA
__all__ = ('DaskMethodsMixin', 'annotate', 'get_annotations', 'is_dask...
__annotations__ = {'_annotations': 'ContextVar[dict[str, Any] | None]'...
normalize_token = <dask.utils.Dispatch object>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\base.py
dask.callbacks
Python Library Documentation: module dask.callbacks in dask
NAME
dask.callbacks
CLASSES
builtins.object
Callback
add_callbacks
class Callback(builtins.object)
| Callback(start=None, start_state=None, pretask=None, posttask=None, finish=None)
|
| Base class for using the callback mechanism
|
| Create a callback with functions of the following signatures:
|
| >>> def start(dsk):
| ... pass
| >>> def start_state(dsk, state):
| ... pass
| >>> def pretask(key, dsk, state):
| ... pass
| >>> def posttask(key, result, dsk, state, worker_id):
| ... pass
| >>> def finish(dsk, state, failed):
| ... pass
|
| You may then construct a callback object with any number of them
|
| >>> cb = Callback(pretask=pretask, finish=finish)
|
| And use it either as a context manager over a compute/get call
|
| >>> with cb: # doctest: +SKIP
| ... x.compute()
|
| Or globally with the ``register`` method
|
| >>> cb.register()
| >>> cb.unregister()
|
| Alternatively subclass the ``Callback`` class with your own methods.
|
| >>> class PrintKeys(Callback):
| ... def _pretask(self, key, dask, state):
| ... print("Computing: {0}!".format(repr(key)))
|
| >>> with PrintKeys(): # doctest: +SKIP
| ... x.compute()
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, *args)
|
| __init__(self, start=None, start_state=None, pretask=None, posttask=None, finish=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| register(self) -> 'None'
|
| unregister(self) -> 'None'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'active': 'ClassVar[set[tuple[Callable | None, ...]...
|
| active = set()
class add_callbacks(builtins.object)
| add_callbacks(*callbacks)
|
| Context manager for callbacks.
|
| Takes several callbacks and applies them only in the enclosed context.
| Callbacks can either be represented as a ``Callback`` object, or as a tuple
| of length 4.
|
| Examples
| --------
| >>> def pretask(key, dsk, state):
| ... print("Now running {0}").format(key)
| >>> callbacks = (None, pretask, None, None)
| >>> with add_callbacks(callbacks): # doctest: +SKIP
| ... res.compute()
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, type, value, traceback)
|
| __init__(self, *callbacks)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
DATA
__all__ = ['Callback', 'add_callbacks']
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\callbacks.py
dask.compute
Python Library Documentation: function compute in module dask.base
compute(*args, traverse=True, optimize_graph=True, scheduler=None, get=None, **kwargs)
Compute several dask collections at once.
Parameters
----------
args : object
Any number of objects. If it is a dask object, it's computed and the
result is returned. By default, python builtin collections are also
traversed to look for dask objects (for more information see the
``traverse`` keyword). Non-dask arguments are passed through unchanged.
traverse : bool, optional
By default dask traverses builtin python collections looking for dask
objects passed to ``compute``. For large collections this can be
expensive. If none of the arguments contain any dask objects, set
``traverse=False`` to avoid doing this traversal.
scheduler : string, optional
Which scheduler to use like "threads", "synchronous" or "processes".
If not provided, the default is to check the global settings first,
and then fall back to the collection defaults.
optimize_graph : bool, optional
If True [default], the optimizations for each collection are applied
before computation. Otherwise the graph is run as is. This can be
useful for debugging.
get : ``None``
Should be left to ``None`` The get= keyword has been removed.
kwargs
Extra keywords to forward to the scheduler function.
Examples
--------
>>> import dask
>>> import dask.array as da
>>> a = da.arange(10, chunks=2).sum()
>>> b = da.arange(10, chunks=2).mean()
>>> dask.compute(a, b)
(np.int64(45), np.float64(4.5))
By default, dask objects inside python collections will also be computed:
>>> dask.compute({'a': a, 'b': b, 'c': 1})
({'a': np.int64(45), 'b': np.float64(4.5), 'c': 1},)
dask.config
Python Library Documentation: module dask.config in dask
NAME
dask.config
CLASSES
builtins.object
set
class set(builtins.object)
| set(arg: 'Mapping | None' = None, config: 'dict | None' = None, lock: 'threading.Lock' = <unlocked _thread.lock object at 0x00000183922F3DC0>, **kwargs)
|
| Temporarily set configuration values within a context manager
|
| Parameters
| ----------
| arg : mapping or None, optional
| A mapping of configuration key-value pairs to set.
| **kwargs :
| Additional key-value pairs to set. If ``arg`` is provided, values set
| in ``arg`` will be applied before those in ``kwargs``.
| Double-underscores (``__``) in keyword arguments will be replaced with
| ``.``, allowing nested values to be easily set.
|
| Examples
| --------
| >>> import dask
|
| Set ``'foo.bar'`` in a context, by providing a mapping.
|
| >>> with dask.config.set({'foo.bar': 123}):
| ... pass
|
| Set ``'foo.bar'`` in a context, by providing a keyword argument.
|
| >>> with dask.config.set(foo__bar=123):
| ... pass
|
| Set ``'foo.bar'`` globally.
|
| >>> dask.config.set(foo__bar=123) # doctest: +SKIP
|
| See Also
| --------
| dask.config.get
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, type, value, traceback)
|
| __init__(self, arg: 'Mapping | None' = None, config: 'dict | None' = None, lock: 'threading.Lock' = <unlocked _thread.lock object at 0x00000183922F3DC0>, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_record': "list[tuple[Literal['insert', 'replace']...
FUNCTIONS
canonical_name(k: 'str', config: 'dict') -> 'str'
Return the canonical name for a key.
Handles user choice of '-' or '_' conventions by standardizing on whichever
version was set first. If a key already exists in either hyphen or
underscore form, the existing version is the canonical name. If neither
version exists the original key is used as is.
check_deprecations(key: 'str', deprecations: 'Mapping[str, str | None]' = {'fuse-ave-width': 'optimization.fuse.ave-width', 'fuse-max-height': 'optimization.fuse.max-height', 'fuse-max-width': 'optimization.fuse.max-width', 'fuse-rename-keys': 'optimization.fuse.rename-keys', 'fuse-max-depth-new-edges': 'optimization.fuse.max-depth-new-edges', 'ucx.cuda-copy': 'distributed.ucx.cuda_copy', 'ucx.tcp': 'distributed.ucx.tcp', 'ucx.nvlink': 'distributed.ucx.nvlink', 'ucx.infiniband': 'distributed.ucx.infiniband', 'ucx.rdmacm': 'distributed.ucx.rdmacm', 'ucx.net-devices': 'distributed.ucx.net-devices', 'ucx.reuse-endpoints': 'distributed.ucx.reuse-endpoints', 'rmm.pool-size': 'distributed.rmm.pool-size', 'shuffle': 'dataframe.shuffle.algorithm', 'array.rechunk-threshold': 'array.rechunk.threshold', 'dataframe.shuffle.algorithm': 'dataframe.shuffle.method', 'dataframe.shuffle-compression': 'dataframe.shuffle.compression', 'admin.traceback.shorten.what': 'admin.traceback.shorten', 'array.shuffle.chunksize-tolerance': 'array.chunk-size-tolerance'}) -> 'str'
Check if the provided value has been renamed or removed
Parameters
----------
key : str
The configuration key to check
deprecations : Dict[str, str]
The mapping of aliases
Examples
--------
>>> deprecations = {"old_key": "new_key", "invalid": None}
>>> check_deprecations("old_key", deprecations=deprecations) # doctest: +SKIP
FutureWarning: Dask configuration key 'old_key' has been deprecated; please use "new_key" instead
>>> check_deprecations("invalid", deprecations=deprecations)
Traceback (most recent call last):
...
ValueError: Dask configuration key 'invalid' has been removed
>>> check_deprecations("another_key", deprecations=deprecations)
'another_key'
Returns
-------
new: str
The proper key, whether the original (if no deprecation) or the aliased
value
See Also
--------
rename
collect(paths: 'list[str]' = ['/etc/dask', 'C:\\Users\\Youse\\AppData\\Local\\Programs\\Python\\Python312\\etc\\dask', 'C:\\Users\\Youse\\.config\\dask'], env: 'Mapping[str, str] | None' = None) -> 'dict'
Collect configuration from paths and environment variables
Parameters
----------
paths : list[str]
A list of paths to search for yaml config files
env : Mapping[str, str]
The system environment variables
Returns
-------
config: dict
See Also
--------
dask.config.refresh: collect configuration and update into primary config
collect_env(env: 'Mapping[str, str] | None' = None) -> 'dict'
Collect config from environment variables
This grabs environment variables of the form "DASK_FOO__BAR_BAZ=123" and
turns these into config variables of the form ``{"foo": {"bar-baz": 123}}``
It transforms the key and value in the following way:
- Lower-cases the key text
- Treats ``__`` (double-underscore) as nested access
- Calls ``ast.literal_eval`` on the value
Any serialized config passed via ``DASK_INTERNAL_INHERIT_CONFIG`` is also set here.
collect_yaml(paths: 'Sequence[str]', *, return_paths: 'bool' = False) -> 'Iterator[dict | tuple[pathlib.Path, dict]]'
Collect configuration from yaml files
This searches through a list of paths, expands to find all yaml or json
files, and then parses each file.
deserialize(data: 'str') -> 'Any'
De-serialize config data into the original object.
Typically when receiving config via the ``DASK_INTERNAL_INHERIT_CONFIG`` environment variable.
Parameters
----------
data: str
String serialized by :func:`dask.config.serialize`
Returns
-------
deserialized_data: obj
The de-serialized data
ensure_file(source: 'str', destination: 'str | None' = None, comment: 'bool' = True) -> 'None'
Copy file to default location if it does not already exist
This tries to move a default configuration file to a default location if
if does not already exist. It also comments out that file by default.
This is to be used by downstream modules (like dask.distributed) that may
have default configuration files that they wish to include in the default
configuration path.
Parameters
----------
source : string, filename
Source configuration file, typically within a source directory.
destination : string, directory
Destination directory. Configurable by ``DASK_CONFIG`` environment
variable, falling back to ~/.config/dask.
comment : bool, True by default
Whether or not to comment out the config file when copying.
expand_environment_variables(config: 'Any') -> 'Any'
Expand environment variables in a nested config dictionary
This function will recursively search through any nested dictionaries
and/or lists.
Parameters
----------
config : dict, iterable, or str
Input object to search for environment variables
Returns
-------
config : same type as input
Examples
--------
>>> expand_environment_variables({'x': [1, 2, '$USER']}) # doctest: +SKIP
{'x': [1, 2, 'my-username']}
get(key: 'str', default: 'Any' = <no_default>, config: 'dict | None' = None, override_with: 'Any' = None) -> 'Any'
Get elements from global config
If ``override_with`` is not None this value will be passed straight back.
Useful for getting kwarg defaults from Dask config.
Use '.' for nested access
Examples
--------
>>> from dask import config
>>> config.get('foo') # doctest: +SKIP
{'x': 1, 'y': 2}
>>> config.get('foo.x') # doctest: +SKIP
1
>>> config.get('foo.x.y', default=123) # doctest: +SKIP
123
>>> config.get('foo.y', override_with=None) # doctest: +SKIP
2
>>> config.get('foo.y', override_with=3) # doctest: +SKIP
3
See Also
--------
dask.config.set
interpret_value(value: 'str') -> 'Any'
merge(*dicts: 'Mapping') -> 'dict'
Update a sequence of nested dictionaries
This prefers the values in the latter dictionaries to those in the former
Examples
--------
>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'y': {'b': 3}}
>>> merge(a, b) # doctest: +SKIP
{'x': 1, 'y': {'a': 2, 'b': 3}}
See Also
--------
dask.config.update
paths_containing_key(key: 'str', paths: 'Sequence[str]' = ['/etc/dask', 'C:\\Users\\Youse\\AppData\\Local\\Programs\\Python\\Python312\\etc\\dask', 'C:\\Users\\Youse\\.config\\dask']) -> 'Iterator[pathlib.Path]'
Generator yielding paths which contain the given key.
pop(key: 'str', default: 'Any' = <no_default>, config: 'dict' = {'temporary-directory': None, 'visualization': {'engine': None}, 'tokenize': {'ensure-deterministic': False}, 'dataframe': {'backend': 'pandas', 'shuffle': {'method': None, 'compression': None}, 'parquet': {'metadata-task-size-local': 512, 'metadata-task-size-remote': 1, 'minimum-partition-size': 75000000}, 'convert-string': None, 'query-planning': None}, 'array': {'backend': 'numpy', 'chunk-size': '128MiB', 'chunk-size-tolerance': 1.25, 'rechunk': {'method': None, 'threshold': 4}, 'svg': {'size': 120}, 'slicing': {'split-large-chunks': None}, 'query-planning': None}, 'optimization': {'annotations': {'fuse': True}, 'tune': {'active': True}, 'fuse': {'active': None, 'ave-width': 1, 'max-width': None, 'max-height': inf, 'max-depth-new-edges': None, 'rename-keys': True, 'delayed': False}}, 'admin': {'async-client-fallback': None, 'traceback': {'shorten': ['concurrent[\\\\\\/]futures[\\\\\\/]', 'dask[\\\\\\/](base|core|local|multiprocessing|optimization|threaded|utils)\\.py', 'dask[\\\\\\/]array[\\\\\\/]core\\.py', 'dask[\\\\\\/]dataframe[\\\\\\/](core|methods)\\.py', 'dask[\\\\\\/]_task_spec\\.py', 'distributed[\\\\\\/](client|scheduler|utils|worker)\\.py', 'tornado[\\\\\\/]gen\\.py', 'pandas[\\\\\\/]core[\\\\\\/]']}}}) -> 'Any'
Like ``get``, but remove the element if found
See Also
--------
dask.config.get
dask.config.set
refresh(config: 'dict | None' = None, defaults: 'list[Mapping]' = [{'temporary-directory': None, 'visualization': {'engine': None}, 'tokenize': {'ensure-deterministic': False}, 'dataframe': {'backend': 'pandas', 'shuffle': {'method': None, 'compression': None}, 'parquet': {'metadata-task-size-local': 512, 'metadata-task-size-remote': 1, 'minimum-partition-size': 75000000}, 'convert-string': None, 'query-planning': None}, 'array': {'backend': 'numpy', 'chunk-size': '128MiB', 'chunk-size-tolerance': 1.25, 'rechunk': {'method': None, 'threshold': 4}, 'svg': {'size': 120}, 'slicing': {'split-large-chunks': None}, 'query-planning': None}, 'optimization': {'annotations': {'fuse': True}, 'tune': {'active': True}, 'fuse': {'active': None, 'ave-width': 1, 'max-width': None, 'max-height': inf, 'max-depth-new-edges': None, 'rename-keys': True, 'delayed': False}}, 'admin': {'async-client-fallback': None, 'traceback': {'shorten': ['concurrent[\\\\\\/]futures[\\\\\\/]', 'dask[\\\\\\/](base|core|local|multiprocessing|optimization|threaded|utils)\\.py', 'dask[\\\\\\/]array[\\\\\\/]core\\.py', 'dask[\\\\\\/]dataframe[\\\\\\/](core|methods)\\.py', 'dask[\\\\\\/]_task_spec\\.py', 'distributed[\\\\\\/](client|scheduler|utils|worker)\\.py', 'tornado[\\\\\\/]gen\\.py', 'pandas[\\\\\\/]core[\\\\\\/]']}}}], paths: 'list[str]' = ['/etc/dask', 'C:\\Users\\Youse\\AppData\\Local\\Programs\\Python\\Python312\\etc\\dask', 'C:\\Users\\Youse\\.config\\dask'], env: 'Mapping[str, str] | None' = None) -> 'None'
Update configuration by re-reading yaml files and env variables
This mutates the global dask.config.config, or the config parameter if
passed in.
This goes through the following stages:
1. Clearing out all old configuration
2. Updating from the stored defaults from downstream libraries
(see update_defaults)
3. Updating from yaml files and environment variables
4. Automatically renaming deprecated keys (with a warning)
Note that some functionality only checks configuration once at startup and
may not change behavior, even if configuration changes. It is recommended
to restart your python process if convenient to ensure that new
configuration changes take place.
See Also
--------
dask.config.collect: for parameters
dask.config.update_defaults
rename(deprecations: 'Mapping[str, str | None]' = {'fuse-ave-width': 'optimization.fuse.ave-width', 'fuse-max-height': 'optimization.fuse.max-height', 'fuse-max-width': 'optimization.fuse.max-width', 'fuse-rename-keys': 'optimization.fuse.rename-keys', 'fuse-max-depth-new-edges': 'optimization.fuse.max-depth-new-edges', 'ucx.cuda-copy': 'distributed.ucx.cuda_copy', 'ucx.tcp': 'distributed.ucx.tcp', 'ucx.nvlink': 'distributed.ucx.nvlink', 'ucx.infiniband': 'distributed.ucx.infiniband', 'ucx.rdmacm': 'distributed.ucx.rdmacm', 'ucx.net-devices': 'distributed.ucx.net-devices', 'ucx.reuse-endpoints': 'distributed.ucx.reuse-endpoints', 'rmm.pool-size': 'distributed.rmm.pool-size', 'shuffle': 'dataframe.shuffle.algorithm', 'array.rechunk-threshold': 'array.rechunk.threshold', 'dataframe.shuffle.algorithm': 'dataframe.shuffle.method', 'dataframe.shuffle-compression': 'dataframe.shuffle.compression', 'admin.traceback.shorten.what': 'admin.traceback.shorten', 'array.shuffle.chunksize-tolerance': 'array.chunk-size-tolerance'}, config: 'dict' = {'temporary-directory': None, 'visualization': {'engine': None}, 'tokenize': {'ensure-deterministic': False}, 'dataframe': {'backend': 'pandas', 'shuffle': {'method': None, 'compression': None}, 'parquet': {'metadata-task-size-local': 512, 'metadata-task-size-remote': 1, 'minimum-partition-size': 75000000}, 'convert-string': None, 'query-planning': None}, 'array': {'backend': 'numpy', 'chunk-size': '128MiB', 'chunk-size-tolerance': 1.25, 'rechunk': {'method': None, 'threshold': 4}, 'svg': {'size': 120}, 'slicing': {'split-large-chunks': None}, 'query-planning': None}, 'optimization': {'annotations': {'fuse': True}, 'tune': {'active': True}, 'fuse': {'active': None, 'ave-width': 1, 'max-width': None, 'max-height': inf, 'max-depth-new-edges': None, 'rename-keys': True, 'delayed': False}}, 'admin': {'async-client-fallback': None, 'traceback': {'shorten': ['concurrent[\\\\\\/]futures[\\\\\\/]', 'dask[\\\\\\/](base|core|local|multiprocessing|optimization|threaded|utils)\\.py', 'dask[\\\\\\/]array[\\\\\\/]core\\.py', 'dask[\\\\\\/]dataframe[\\\\\\/](core|methods)\\.py', 'dask[\\\\\\/]_task_spec\\.py', 'distributed[\\\\\\/](client|scheduler|utils|worker)\\.py', 'tornado[\\\\\\/]gen\\.py', 'pandas[\\\\\\/]core[\\\\\\/]']}}}) -> 'dict[str, str | None]'
Rename deprecated keys to new keys
This helps migrate older configuration versions over time
Returns
-------
Applied deprecations
See Also
--------
check_deprecations
serialize(data: 'Any') -> 'str'
Serialize config data into a string.
Typically used to pass config via the ``DASK_INTERNAL_INHERIT_CONFIG`` environment variable.
Parameters
----------
data: json-serializable object
The data to serialize
Returns
-------
serialized_data: str
The serialized data as a string
update(old: 'dict', new: 'Mapping', priority: "Literal['old', 'new', 'new-defaults']" = 'new', defaults: 'Mapping | None' = None) -> 'dict'
Update a nested dictionary with values from another
This is like dict.update except that it smoothly merges nested values
This operates in-place and modifies old
Parameters
----------
priority: string {'old', 'new', 'new-defaults'}
If new (default) then the new dictionary has preference.
Otherwise the old dictionary does.
If 'new-defaults', a mapping should be given of the current defaults.
Only if a value in ``old`` matches the current default, it will be
updated with ``new``.
Examples
--------
>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'x': 2, 'y': {'b': 3}}
>>> update(a, b) # doctest: +SKIP
{'x': 2, 'y': {'a': 2, 'b': 3}}
>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'x': 2, 'y': {'b': 3}}
>>> update(a, b, priority='old') # doctest: +SKIP
{'x': 1, 'y': {'a': 2, 'b': 3}}
>>> d = {'x': 0, 'y': {'a': 2}}
>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'x': 2, 'y': {'a': 3, 'b': 3}}
>>> update(a, b, priority='new-defaults', defaults=d) # doctest: +SKIP
{'x': 1, 'y': {'a': 3, 'b': 3}}
See Also
--------
dask.config.merge
update_defaults(new: 'Mapping', config: 'dict' = {'temporary-directory': None, 'visualization': {'engine': None}, 'tokenize': {'ensure-deterministic': False}, 'dataframe': {'backend': 'pandas', 'shuffle': {'method': None, 'compression': None}, 'parquet': {'metadata-task-size-local': 512, 'metadata-task-size-remote': 1, 'minimum-partition-size': 75000000}, 'convert-string': None, 'query-planning': None}, 'array': {'backend': 'numpy', 'chunk-size': '128MiB', 'chunk-size-tolerance': 1.25, 'rechunk': {'method': None, 'threshold': 4}, 'svg': {'size': 120}, 'slicing': {'split-large-chunks': None}, 'query-planning': None}, 'optimization': {'annotations': {'fuse': True}, 'tune': {'active': True}, 'fuse': {'active': None, 'ave-width': 1, 'max-width': None, 'max-height': inf, 'max-depth-new-edges': None, 'rename-keys': True, 'delayed': False}}, 'admin': {'async-client-fallback': None, 'traceback': {'shorten': ['concurrent[\\\\\\/]futures[\\\\\\/]', 'dask[\\\\\\/](base|core|local|multiprocessing|optimization|threaded|utils)\\.py', 'dask[\\\\\\/]array[\\\\\\/]core\\.py', 'dask[\\\\\\/]dataframe[\\\\\\/](core|methods)\\.py', 'dask[\\\\\\/]_task_spec\\.py', 'distributed[\\\\\\/](client|scheduler|utils|worker)\\.py', 'tornado[\\\\\\/]gen\\.py', 'pandas[\\\\\\/]core[\\\\\\/]']}}}, defaults: 'list[Mapping]' = [{'temporary-directory': None, 'visualization': {'engine': None}, 'tokenize': {'ensure-deterministic': False}, 'dataframe': {'backend': 'pandas', 'shuffle': {'method': None, 'compression': None}, 'parquet': {'metadata-task-size-local': 512, 'metadata-task-size-remote': 1, 'minimum-partition-size': 75000000}, 'convert-string': None, 'query-planning': None}, 'array': {'backend': 'numpy', 'chunk-size': '128MiB', 'chunk-size-tolerance': 1.25, 'rechunk': {'method': None, 'threshold': 4}, 'svg': {'size': 120}, 'slicing': {'split-large-chunks': None}, 'query-planning': None}, 'optimization': {'annotations': {'fuse': True}, 'tune': {'active': True}, 'fuse': {'active': None, 'ave-width': 1, 'max-width': None, 'max-height': inf, 'max-depth-new-edges': None, 'rename-keys': True, 'delayed': False}}, 'admin': {'async-client-fallback': None, 'traceback': {'shorten': ['concurrent[\\\\\\/]futures[\\\\\\/]', 'dask[\\\\\\/](base|core|local|multiprocessing|optimization|threaded|utils)\\.py', 'dask[\\\\\\/]array[\\\\\\/]core\\.py', 'dask[\\\\\\/]dataframe[\\\\\\/](core|methods)\\.py', 'dask[\\\\\\/]_task_spec\\.py', 'distributed[\\\\\\/](client|scheduler|utils|worker)\\.py', 'tornado[\\\\\\/]gen\\.py', 'pandas[\\\\\\/]core[\\\\\\/]']}}}]) -> 'None'
Add a new set of defaults to the configuration
It does two things:
1. Add the defaults to a global collection to be used by refresh later
2. Updates the global config with the new configuration.
Old values are prioritized over new ones, unless the current value
is the old default, in which case it's updated to the new default.
DATA
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
PATH = r'C:\Users\Youse\.config\dask'
__annotations__ = {'config': 'dict', 'defaults': 'list[Mapping]', 'dep...
config = {'admin': {'async-client-fallback': None, 'traceback': {'shor...
config_lock = <unlocked _thread.lock object>
defaults = [{'admin': {'async-client-fallback': None, 'traceback': {'s...
deprecations = {'admin.traceback.shorten.what': 'admin.traceback.short...
global_config = {'admin': {'async-client-fallback': None, 'traceback':...
no_default = <no_default>
paths = ['/etc/dask', r'C:\Users\Youse\AppData\Local\Programs\Python\P...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\config.py
dask.context
Python Library Documentation: module dask.context in dask
NAME
dask.context - Control global computation context
CLASSES
builtins.object
GlobalMethod
class GlobalMethod(builtins.object)
| GlobalMethod(default, key, falsey=None)
|
| Methods defined here:
|
| __get__(self, instance, owner=None)
|
| __init__(self, default, key, falsey=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
globalmethod(default=None, key=None, falsey=None)
Allow function to be taken over by globals
This modifies a method so that occurrences of it may be taken over by
functions registered in the global options. Can be used as a decorator or a
function.
Parameters
----------
default : callable
The default callable to use.
key : str
Key under which we register this function in the global parameters
falsey : callable, None, optional
A function to use if the option is falsey. If not provided, the default
is used instead.
Examples
--------
>>> import dask
>>> class Foo:
... @globalmethod(key='bar', falsey=lambda: 3)
... def bar():
... return 1
>>> f = Foo()
>>> f.bar()
1
>>> with dask.config.set(bar=lambda: 2):
... print(f.bar())
2
>>> with dask.config.set(bar=False):
... print(f.bar())
3
DATA
thread_state = <_thread._local object>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\context.py
dask.core
Python Library Documentation: module dask.core in dask
NAME
dask.core
CLASSES
builtins.object
literal
class literal(builtins.object)
| literal(data)
|
| A small serializable object to wrap literal values without copying
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __init__(self, data)
| Initialize self. See help(type(self)) for accurate signature.
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| data
FUNCTIONS
flatten(seq, container=<class 'list'>)
>>> list(flatten([1]))
[1]
>>> list(flatten([[1, 2], [1, 2]]))
[1, 2, 1, 2]
>>> list(flatten([[[1], [2]], [[1], [2]]]))
[1, 2, 1, 2]
>>> list(flatten(((1, 2), (1, 2)))) # Don't flatten tuples
[(1, 2), (1, 2)]
>>> list(flatten((1, 2, [3, 4]))) # support heterogeneous
[1, 2, 3, 4]
get(dsk: 'Mapping', out: 'list | Key', cache: 'MutableMapping | None' = None) -> 'Any'
Get value from Dask
Examples
--------
>>> inc = lambda x: x + 1
>>> d = {'x': 1, 'y': (inc, 'x')}
>>> get(d, 'x')
1
>>> get(d, 'y')
2
get_dependencies(dsk: 'Graph', key: 'Key | None' = None, task: 'Key | NoDefault' = <no_default>, as_list: 'bool' = False) -> 'set[Key] | list[Key]'
Get the immediate tasks on which this task depends
Examples
--------
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> dsk = {'x': 1,
... 'y': (inc, 'x'),
... 'z': (add, 'x', 'y'),
... 'w': (inc, 'z'),
... 'a': (add, (inc, 'x'), 1)}
>>> get_dependencies(dsk, 'x')
set()
>>> get_dependencies(dsk, 'y')
{'x'}
>>> get_dependencies(dsk, 'z') # doctest: +SKIP
{'x', 'y'}
>>> get_dependencies(dsk, 'w') # Only direct dependencies
{'z'}
>>> get_dependencies(dsk, 'a') # Ignore non-keys
{'x'}
>>> get_dependencies(dsk, task=(inc, 'x')) # provide tasks directly
{'x'}
get_deps(dsk: 'Graph') -> 'tuple[dict[Key, set[Key]], dict[Key, set[Key]]]'
Get dependencies and dependents from dask dask graph
>>> inc = lambda x: x + 1
>>> dsk = {'a': 1, 'b': (inc, 'a'), 'c': (inc, 'b')}
>>> dependencies, dependents = get_deps(dsk)
>>> dependencies
{'a': set(), 'b': {'a'}, 'c': {'b'}}
>>> dependents # doctest: +SKIP
{'a': {'b'}, 'b': {'c'}, 'c': set()}
getcycle(d, keys)
Return a list of nodes that form a cycle if Dask is not a DAG.
Returns an empty list if no cycle is found.
``keys`` may be a single key or list of keys.
Examples
--------
>>> inc = lambda x: x + 1
>>> d = {'x': (inc, 'z'), 'y': (inc, 'x'), 'z': (inc, 'y')}
>>> getcycle(d, 'x')
['x', 'z', 'y', 'x']
See Also
--------
isdag
isdag(d, keys)
Does Dask form a directed acyclic graph when calculating keys?
``keys`` may be a single key or list of keys.
Examples
--------
>>> inc = lambda x: x + 1
>>> inc = lambda x: x + 1
>>> isdag({'x': 0, 'y': (inc, 'x')}, 'y')
True
>>> isdag({'x': (inc, 'y'), 'y': (inc, 'x')}, 'y')
False
See Also
--------
getcycle
ishashable(x)
Is x hashable?
Examples
--------
>>> ishashable(1)
True
>>> ishashable([1])
False
See Also
--------
iskey
iskey(key: 'object') -> 'bool'
Return True if the given object is a potential dask key; False otherwise.
The definition of a key in a Dask graph is any str, int, float, or tuple
thereof.
See Also
--------
ishashable
validate_key
dask.typing.Key
istask(x)
Is x a runnable task?
A task is a tuple with a callable first argument
Examples
--------
>>> inc = lambda x: x + 1
>>> istask((inc, 1))
True
>>> istask(1)
False
keys_in_tasks(keys: 'Collection[Key]', tasks: 'Iterable[Any]', as_list: 'bool' = False)
Returns the keys in `keys` that are also in `tasks`
Examples
--------
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> dsk = {'x': 1,
... 'y': (inc, 'x'),
... 'z': (add, 'x', 'y'),
... 'w': (inc, 'z'),
... 'a': (add, (inc, 'x'), 1)}
>>> keys_in_tasks(dsk, ['x', 'y', 'j']) # doctest: +SKIP
{'x', 'y'}
lists_to_tuples(res, keys)
preorder_traversal(task)
A generator to preorder-traverse a task.
quote(x)
Ensure that this value remains this value in a dask graph
Some values in dask graph take on special meaning. Sometimes we want to
ensure that our data is not interpreted but remains literal.
>>> add = lambda x, y: x + y
>>> quote((add, 1, 2))
(literal<type=tuple>,)
reshapelist(shape, seq)
Reshape iterator to nested shape
>>> reshapelist((2, 3), range(6))
[[0, 1, 2], [3, 4, 5]]
reverse_dict(d: 'Mapping[T_, Iterable[T_]]') -> 'dict[T_, set[T_]]'
>>> a, b, c = 'abc'
>>> d = {a: [b, c], b: [c]}
>>> reverse_dict(d) # doctest: +SKIP
{'a': set([]), 'b': set(['a']}, 'c': set(['a', 'b'])}
subs(task, key, val)
Perform a substitution on a task
Examples
--------
>>> def inc(x):
... return x + 1
>>> subs((inc, 'x'), 'x', 1) # doctest: +ELLIPSIS
(<function inc at ...>, 1)
toposort(dsk, dependencies=None)
Return a list of keys of dask sorted in topological order.
validate_key(key: 'object') -> 'None'
Validate the format of a dask key.
See Also
--------
iskey
DATA
Graph = collections.abc.Mapping[str | int | float | tuple['Key', ...],...
Key = str | int | float | tuple['Key', ...]
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
NoDefault = typing.Literal[<no_default>]
T_ = ~T_
no_default = <no_default>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\core.py
dask.datasets
Python Library Documentation: module dask.datasets in dask
NAME
dask.datasets
FUNCTIONS
make_people(npartitions=10, records_per_partition=1000, seed=None, locale='en')
Make a dataset of random people
This makes a Dask Bag with dictionary records of randomly generated people.
This requires the optional library ``mimesis`` to generate records.
Parameters
----------
npartitions : int
Number of partitions
records_per_partition : int
Number of records in each partition
seed : int, (optional)
Random seed
locale : str
Language locale, like 'en', 'fr', 'zh', or 'ru'
Returns
-------
b: Dask Bag
timeseries(start='2000-01-01', end='2000-01-31', freq='1s', partition_freq='1D', dtypes=None, seed=None, **kwargs)
Create timeseries dataframe with random data
Parameters
----------
start : datetime (or datetime-like string)
Start of time series
end : datetime (or datetime-like string)
End of time series
dtypes : dict (optional)
Mapping of column names to types.
Valid types include {float, int, str, 'category'}
freq : string
String like '2s' or '1H' or '12W' for the time series frequency
partition_freq : string
String like '1M' or '2Y' to divide the dataframe into partitions
seed : int (optional)
Randomstate seed
kwargs:
Keywords to pass down to individual column creation functions.
Keywords should be prefixed by the column name and then an underscore.
Examples
--------
>>> import dask
>>> df = dask.datasets.timeseries()
>>> df.head() # doctest: +SKIP
timestamp id name x y
2000-01-01 00:00:00 967 Jerry -0.031348 -0.040633
2000-01-01 00:00:01 1066 Michael -0.262136 0.307107
2000-01-01 00:00:02 988 Wendy -0.526331 0.128641
2000-01-01 00:00:03 1016 Yvonne 0.620456 0.767270
2000-01-01 00:00:04 998 Ursula 0.684902 -0.463278
>>> df = dask.datasets.timeseries(
... '2000', '2010',
... freq='2h', partition_freq='1D', seed=1, # data frequency
... dtypes={'value': float, 'name': str, 'id': int}, # data types
... id_lam=1000 # control number of items in id column
... )
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\datasets.py
dask.delayed
Python Library Documentation: curry in module dask.delayed
delayed(obj='__no__default__', name=None, pure=None, nout=None, traverse=True)
Wraps a function or object to produce a ``Delayed``.
``Delayed`` objects act as proxies for the object they wrap, but all
operations on them are done lazily by building up a dask graph internally.
Parameters
----------
obj : object
The function or object to wrap
name : Dask key, optional
The key to use in the underlying graph for the wrapped object. Defaults
to hashing content. Note that this only affects the name of the object
wrapped by this call to delayed, and *not* the output of delayed
function calls - for that use ``dask_key_name=`` as described below.
.. note::
Because this ``name`` is used as the key in task graphs, you should
ensure that it uniquely identifies ``obj``. If you'd like to provide
a descriptive name that is still unique, combine the descriptive name
with :func:`dask.base.tokenize` of the ``array_like``. See
:ref:`graphs` for more.
pure : bool, optional
Indicates whether calling the resulting ``Delayed`` object is a pure
operation. If True, arguments to the call are hashed to produce
deterministic keys. If not provided, the default is to check the global
``delayed_pure`` setting, and fallback to ``False`` if unset.
nout : int, optional
The number of outputs returned from calling the resulting ``Delayed``
object. If provided, the ``Delayed`` output of the call can be iterated
into ``nout`` objects, allowing for unpacking of results. By default
iteration over ``Delayed`` objects will error. Note, that ``nout=1``
expects ``obj`` to return a tuple of length 1, and consequently for
``nout=0``, ``obj`` should return an empty tuple.
traverse : bool, optional
By default dask traverses builtin python collections looking for dask
objects passed to ``delayed``. For large collections this can be
expensive. If ``obj`` doesn't contain any dask objects, set
``traverse=False`` to avoid doing this traversal.
Examples
--------
Apply to functions to delay execution:
>>> from dask import delayed
>>> def inc(x):
... return x + 1
>>> inc(10)
11
>>> x = delayed(inc, pure=True)(10)
>>> type(x) == Delayed
True
>>> x.compute()
11
Can be used as a decorator:
>>> @delayed(pure=True)
... def add(a, b):
... return a + b
>>> add(1, 2).compute()
3
``delayed`` also accepts an optional keyword ``pure``. If False, then
subsequent calls will always produce a different ``Delayed``. This is
useful for non-pure functions (such as ``time`` or ``random``).
>>> from random import random
>>> out1 = delayed(random, pure=False)()
>>> out2 = delayed(random, pure=False)()
>>> out1.key == out2.key
False
If you know a function is pure (output only depends on the input, with no
global state), then you can set ``pure=True``. This will attempt to apply a
consistent name to the output, but will fallback on the same behavior of
``pure=False`` if this fails.
>>> @delayed(pure=True)
... def add(a, b):
... return a + b
>>> out1 = add(1, 2)
>>> out2 = add(1, 2)
>>> out1.key == out2.key
True
Instead of setting ``pure`` as a property of the callable, you can also set
it contextually using the ``delayed_pure`` setting. Note that this
influences the *call* and not the *creation* of the callable:
>>> @delayed
... def mul(a, b):
... return a * b
>>> import dask
>>> with dask.config.set(delayed_pure=True):
... print(mul(1, 2).key == mul(1, 2).key)
True
>>> with dask.config.set(delayed_pure=False):
... print(mul(1, 2).key == mul(1, 2).key)
False
The key name of the result of calling a delayed object is determined by
hashing the arguments by default. To explicitly set the name, you can use
the ``dask_key_name`` keyword when calling the function:
>>> add(1, 2) # doctest: +SKIP
Delayed('add-3dce7c56edd1ac2614add714086e950f')
>>> add(1, 2, dask_key_name='three')
Delayed('three')
Note that objects with the same key name are assumed to have the same
result. If you set the names explicitly you should make sure your key names
are different for different results.
>>> add(1, 2, dask_key_name='three')
Delayed('three')
>>> add(2, 1, dask_key_name='three')
Delayed('three')
>>> add(2, 2, dask_key_name='four')
Delayed('four')
``delayed`` can also be applied to objects to make operations on them lazy:
>>> a = delayed([1, 2, 3])
>>> isinstance(a, Delayed)
True
>>> a.compute()
[1, 2, 3]
The key name of a delayed object is hashed by default if ``pure=True`` or
is generated randomly if ``pure=False`` (default). To explicitly set the
name, you can use the ``name`` keyword. To ensure that the key is unique
you should include the tokenized value as well, or otherwise ensure that
it's unique:
>>> from dask.base import tokenize
>>> data = [1, 2, 3]
>>> a = delayed(data, name='mylist-' + tokenize(data))
>>> a # doctest: +SKIP
Delayed('mylist-55af65871cb378a4fa6de1660c3e8fb7')
Delayed results act as a proxy to the underlying object. Many operators
are supported:
>>> (a + [1, 2]).compute()
[1, 2, 3, 1, 2]
>>> a[1].compute()
2
Method and attribute access also works:
>>> a.count(2).compute()
1
Note that if a method doesn't exist, no error will be thrown until runtime:
>>> res = a.not_a_real_method() # doctest: +SKIP
>>> res.compute() # doctest: +SKIP
AttributeError("'list' object has no attribute 'not_a_real_method'")
"Magic" methods (e.g. operators and attribute access) are assumed to be
pure, meaning that subsequent calls must return the same results. This
behavior is not overridable through the ``delayed`` call, but can be
modified using other ways as described below.
To invoke an impure attribute or operator, you'd need to use it in a
delayed function with ``pure=False``:
>>> class Incrementer:
... def __init__(self):
... self._n = 0
... @property
... def n(self):
... self._n += 1
... return self._n
...
>>> x = delayed(Incrementer())
>>> x.n.key == x.n.key
True
>>> get_n = delayed(lambda x: x.n, pure=False)
>>> get_n(x).key == get_n(x).key
False
In contrast, methods are assumed to be impure by default, meaning that
subsequent calls may return different results. To assume purity, set
``pure=True``. This allows sharing of any intermediate values.
>>> a.count(2, pure=True).key == a.count(2, pure=True).key
True
As with function calls, method calls also respect the global
``delayed_pure`` setting and support the ``dask_key_name`` keyword:
>>> a.count(2, dask_key_name="count_2")
Delayed('count_2')
>>> import dask
>>> with dask.config.set(delayed_pure=True):
... print(a.count(2).key == a.count(2).key)
True
dask.get
Python Library Documentation: function get_sync in module dask.local
get_sync(dsk: 'Mapping', keys: 'Sequence[Key] | Key', **kwargs)
A naive synchronous version of get_async
Can be useful for debugging.
dask.get_annotations
Python Library Documentation: function get_annotations in module dask.base
get_annotations() -> 'dict[str, Any]'
Get current annotations.
Returns
-------
Dict of all current annotations
See Also
--------
annotate
dask.hashing
Python Library Documentation: module dask.hashing in dask
NAME
dask.hashing
FUNCTIONS
hash_buffer(buf, hasher=None)
Hash a bytes-like (buffer-compatible) object. This function returns
a good quality hash but is not cryptographically secure. The fastest
available algorithm is selected. A fixed-length bytes object is returned.
hash_buffer_hex(buf, hasher=None)
Same as hash_buffer, but returns its result in hex-encoded form.
DATA
hashers = [<function _hash_xxhash>, <function _hash_sha1>]
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\hashing.py
dask.highlevelgraph
Python Library Documentation: module dask.highlevelgraph in dask
NAME
dask.highlevelgraph
CLASSES
collections.abc.Mapping(collections.abc.Collection)
HighLevelGraph
Layer
MaterializedLayer
class HighLevelGraph(collections.abc.Mapping)
| HighLevelGraph(layers: 'Mapping[str, Graph]', dependencies: 'Mapping[str, set[str]]', key_dependencies: 'dict[Key, set[Key]] | None' = None)
|
| Task graph composed of layers of dependent subgraphs
|
| This object encodes a Dask task graph that is composed of layers of
| dependent subgraphs, such as commonly occurs when building task graphs
| using high level collections like Dask array, bag, or dataframe.
|
| Typically each high level array, bag, or dataframe operation takes the task
| graphs of the input collections, merges them, and then adds one or more new
| layers of tasks for the new operation. These layers typically have at
| least as many tasks as there are partitions or chunks in the collection.
| The HighLevelGraph object stores the subgraphs for each operation
| separately in sub-graphs, and also stores the dependency structure between
| them.
|
| Parameters
| ----------
| layers : Mapping[str, Mapping]
| The subgraph layers, keyed by a unique name
| dependencies : Mapping[str, set[str]]
| The set of layers on which each layer depends
| key_dependencies : dict[Key, set], optional
| Mapping (some) keys in the high level graph to their dependencies. If
| a key is missing, its dependencies will be calculated on-the-fly.
|
| Examples
| --------
| Here is an idealized example that shows the internal state of a
| HighLevelGraph
|
| >>> import dask.dataframe as dd
|
| >>> df = dd.read_csv('myfile.*.csv') # doctest: +SKIP
| >>> df = df + 100 # doctest: +SKIP
| >>> df = df[df.name == 'Alice'] # doctest: +SKIP
|
| >>> graph = df.__dask_graph__() # doctest: +SKIP
| >>> graph.layers # doctest: +SKIP
| {
| 'read-csv': {('read-csv', 0): (pandas.read_csv, 'myfile.0.csv'),
| ('read-csv', 1): (pandas.read_csv, 'myfile.1.csv'),
| ('read-csv', 2): (pandas.read_csv, 'myfile.2.csv'),
| ('read-csv', 3): (pandas.read_csv, 'myfile.3.csv')},
| 'add': {('add', 0): (operator.add, ('read-csv', 0), 100),
| ('add', 1): (operator.add, ('read-csv', 1), 100),
| ('add', 2): (operator.add, ('read-csv', 2), 100),
| ('add', 3): (operator.add, ('read-csv', 3), 100)}
| 'filter': {('filter', 0): (lambda part: part[part.name == 'Alice'], ('add', 0)),
| ('filter', 1): (lambda part: part[part.name == 'Alice'], ('add', 1)),
| ('filter', 2): (lambda part: part[part.name == 'Alice'], ('add', 2)),
| ('filter', 3): (lambda part: part[part.name == 'Alice'], ('add', 3))}
| }
|
| >>> graph.dependencies # doctest: +SKIP
| {
| 'read-csv': set(),
| 'add': {'read-csv'},
| 'filter': {'add'}
| }
|
| See Also
| --------
| HighLevelGraph.from_collections :
| typically used by developers to make new HighLevelGraphs
|
| Method resolution order:
| HighLevelGraph
| collections.abc.Mapping
| collections.abc.Collection
| collections.abc.Sized
| collections.abc.Iterable
| collections.abc.Container
| builtins.object
|
| Methods defined here:
|
| __getitem__(self, key: 'Key') -> 'Any'
|
| __init__(self, layers: 'Mapping[str, Graph]', dependencies: 'Mapping[str, set[str]]', key_dependencies: 'dict[Key, set[Key]] | None' = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> 'Iterator[Key]'
|
| __len__(self) -> 'int'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'HighLevelGraph'
|
| cull(self, keys: 'Iterable[Key]') -> 'HighLevelGraph'
| Return new HighLevelGraph with only the tasks required to calculate keys.
|
| In other words, remove unnecessary tasks from dask.
|
| Parameters
| ----------
| keys
| iterable of keys or nested list of keys such as the output of
| ``__dask_keys__()``
|
| Returns
| -------
| hlg: HighLevelGraph
| Culled high level graph
|
| cull_layers(self, layers: 'Iterable[str]') -> 'HighLevelGraph'
| Return a new HighLevelGraph with only the given layers and their
| dependencies. Internally, layers are not modified.
|
| This is a variant of :meth:`HighLevelGraph.cull` which is much faster and does
| not risk creating a collision between two layers with the same name and
| different content when two culled graphs are merged later on.
|
| Returns
| -------
| hlg: HighLevelGraph
| Culled high level graph
|
| get_all_dependencies(self) -> 'dict[Key, set[Key]]'
| Get dependencies of all keys
|
| This will in most cases materialize all layers, which makes
| it an expensive operation.
|
| Returns
| -------
| map: Mapping
| A map that maps each key to its dependencies
|
| get_all_external_keys(self) -> 'set[Key]'
| Get all output keys of all layers
|
| This will in most cases _not_ materialize any layers, which makes
| it a relative cheap operation.
|
| Returns
| -------
| keys: set
| A set of all external keys
|
| items(self) -> 'ItemsView[Key, Any]'
| D.items() -> a set-like object providing a view on D's items
|
| keys(self) -> 'KeysView'
| Get all keys of all the layers.
|
| This will in many cases materialize layers, which makes it a relatively
| expensive operation. See :meth:`get_all_external_keys` for a faster alternative.
|
| to_dict(self) -> 'dict[Key, Any]'
| Efficiently convert to plain dict. This method is faster than dict(self).
|
| validate(self) -> 'None'
|
| values(self) -> 'ValuesView[Any]'
| D.values() -> an object providing a view on D's values
|
| visualize(self, filename='dask-hlg.svg', format=None, **kwargs)
| Visualize this dask high level graph.
|
| Requires ``graphviz`` to be installed.
|
| Parameters
| ----------
| filename : str or None, optional
| The name of the file to write to disk. If the provided `filename`
| doesn't include an extension, '.png' will be used by default.
| If `filename` is None, no file will be written, and the graph is
| rendered in the Jupyter notebook only.
| format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
| Format in which to write output file. Default is 'svg'.
| color : {None, 'layer_type'}, optional (default: None)
| Options to color nodes.
| - None, no colors.
| - layer_type, color nodes based on the layer type.
| **kwargs
| Additional keyword arguments to forward to ``to_graphviz``.
|
| Examples
| --------
| >>> x.dask.visualize(filename='dask.svg') # doctest: +SKIP
| >>> x.dask.visualize(filename='dask.svg', color='layer_type') # doctest: +SKIP
|
| Returns
| -------
| result : IPython.display.Image, IPython.display.SVG, or None
| See dask.dot.dot_graph for more information.
|
| See Also
| --------
| dask.dot.dot_graph
| dask.base.visualize # low level variant
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_collections(name: 'str', layer: 'Graph', dependencies: 'Sequence[DaskCollection]' = ()) -> 'HighLevelGraph'
| Construct a HighLevelGraph from a new layer and a set of collections
|
| This constructs a HighLevelGraph in the common case where we have a single
| new layer and a set of old collections on which we want to depend.
|
| This pulls out the ``__dask_layers__()`` method of the collections if
| they exist, and adds them to the dependencies for this new layer. It
| also merges all of the layers from all of the dependent collections
| together into the new layers for this graph.
|
| Parameters
| ----------
| name : str
| The name of the new layer
| layer : Mapping
| The graph layer itself
| dependencies : List of Dask collections
| A list of other dask collections (like arrays or dataframes) that
| have graphs themselves
|
| Examples
| --------
|
| In typical usage we make a new task layer, and then pass that layer
| along with all dependent collections to this method.
|
| >>> def add(self, other):
| ... name = 'add-' + tokenize(self, other)
| ... layer = {(name, i): (add, input_key, other)
| ... for i, input_key in enumerate(self.__dask_keys__())}
| ... graph = HighLevelGraph.from_collections(name, layer, dependencies=[self])
| ... return new_collection(name, graph)
|
| merge(*graphs: 'Graph') -> 'HighLevelGraph'
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| dependents
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {'_all_external_keys': 'set', '_to_dict': 'dict', 'd...
|
| __orig_bases__ = (collections.abc.Mapping[str | int | float | tuple['K...
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.Mapping:
|
| __contains__(self, key)
|
| __eq__(self, other)
| Return self==value.
|
| get(self, key, default=None)
| D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from collections.abc.Mapping:
|
| __hash__ = None
|
| __reversed__ = None
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Collection:
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
class Layer(collections.abc.Mapping)
| Layer(annotations: 'Mapping[str, Any] | None' = None, collection_annotations: 'Mapping[str, Any] | None' = None)
|
| High level graph layer
|
| This abstract class establish a protocol for high level graph layers.
|
| The main motivation of a layer is to represent a collection of tasks
| symbolically in order to speedup a series of operations significantly.
| Ideally, a layer should stay in this symbolic state until execution
| but in practice some operations will force the layer to generate all
| its internal tasks. We say that the layer has been materialized.
|
| Most of the default implementations in this class will materialize the
| layer. It is up to derived classes to implement non-materializing
| implementations.
|
| Method resolution order:
| Layer
| collections.abc.Mapping
| collections.abc.Collection
| collections.abc.Sized
| collections.abc.Iterable
| collections.abc.Container
| builtins.object
|
| Methods defined here:
|
| __copy__(self)
| Default shallow copy implementation
|
| __init__(self, annotations: 'Mapping[str, Any] | None' = None, collection_annotations: 'Mapping[str, Any] | None' = None)
| Initialize Layer object.
|
| Parameters
| ----------
| annotations : Mapping[str, Any], optional
| By default, None.
| Annotations are metadata or soft constraints associated with tasks
| that dask schedulers may choose to respect:
| They signal intent without enforcing hard constraints.
| As such, they are primarily designed for use with the distributed
| scheduler. See the dask.annotate function for more information.
| collection_annotations : Mapping[str, Any], optional. By default, None.
| Experimental, intended to assist with visualizing the performance
| characteristics of Dask computations.
| These annotations are *not* passed to the distributed scheduler.
|
| clone(self, keys: 'set', seed: 'Hashable', bind_to: 'Key | None' = None) -> 'tuple[Layer, bool]'
| Clone selected keys in the layer, as well as references to keys in other
| layers
|
| Parameters
| ----------
| keys
| Keys to be replaced. This never includes keys not listed by
| :meth:`get_output_keys`. It must also include any keys that are outside
| of this layer that may be referenced by it.
| seed
| Common hashable used to alter the keys; see :func:`dask.base.clone_key`
| bind_to
| Optional key to bind the leaf nodes to. A leaf node here is one that does
| not reference any replaced keys; in other words it's a node where the
| replacement graph traversal stops; it may still have dependencies on
| non-replaced nodes.
| A bound node will not be computed until after ``bind_to`` has been computed.
|
| Returns
| -------
| - New layer
| - True if the ``bind_to`` key was injected anywhere; False otherwise
|
| Notes
| -----
| This method should be overridden by subclasses to avoid materializing the layer.
|
| cull(self, keys: 'set[Key]', all_hlg_keys: 'Collection[Key]') -> 'tuple[Layer, Mapping[Key, set[Key]]]'
| Remove unnecessary tasks from the layer
|
| In other words, return a new Layer with only the tasks required to
| calculate `keys` and a map of external key dependencies.
|
| Examples
| --------
| >>> inc = lambda x: x + 1
| >>> add = lambda x, y: x + y
| >>> d = MaterializedLayer({'x': 1, 'y': (inc, 'x'), 'out': (add, 'x', 10)})
| >>> _, deps = d.cull({'out'}, d.keys())
| >>> deps
| {'out': {'x'}, 'x': set()}
|
| Returns
| -------
| layer: Layer
| Culled layer
| deps: Map
| Map of external key dependencies
|
| get_dependencies(self, key: 'Key', all_hlg_keys: 'Collection[Key]') -> 'set'
| Get dependencies of `key` in the layer
|
| Parameters
| ----------
| key:
| The key to find dependencies of
| all_hlg_keys:
| All keys in the high level graph.
|
| Returns
| -------
| deps: set
| A set of dependencies
|
| get_output_keys(self) -> 'Set[Key]'
| Return a set of all output keys
|
| Output keys are all keys in the layer that might be referenced by
| other layers.
|
| Classes overriding this implementation should not cause the layer
| to be materialized.
|
| Returns
| -------
| keys: Set
| All output keys
|
| has_legacy_tasks = <functools.cached_property object>
| Check if the layer has legacy tasks
|
| Legacy tasks are those that are not in the form of a tuple
| (function, *args, **kwargs).
|
| is_materialized(self) -> 'bool'
| Return whether the layer is materialized or not
|
| layer_info_dict(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'__getitem__', '__iter__', '__len__',...
|
| __annotations__ = {'annotations': 'Mapping[str, Any] | None', 'collect...
|
| __orig_bases__ = (collections.abc.Mapping[str | int | float | tuple['K...
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.Mapping:
|
| __contains__(self, key)
|
| __eq__(self, other)
| Return self==value.
|
| __getitem__(self, key)
|
| get(self, key, default=None)
| D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.
|
| items(self)
| D.items() -> a set-like object providing a view on D's items
|
| keys(self)
| D.keys() -> a set-like object providing a view on D's keys
|
| values(self)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from collections.abc.Mapping:
|
| __hash__ = None
|
| __reversed__ = None
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Collection:
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.Sized:
|
| __len__(self)
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.Iterable:
|
| __iter__(self)
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
class MaterializedLayer(Layer)
| MaterializedLayer(mapping: 'Mapping', annotations=None, collection_annotations=None)
|
| Fully materialized layer of `Layer`
|
| Parameters
| ----------
| mapping: Mapping
| The mapping between keys and tasks, typically a dask graph.
|
| Method resolution order:
| MaterializedLayer
| Layer
| collections.abc.Mapping
| collections.abc.Collection
| collections.abc.Sized
| collections.abc.Iterable
| collections.abc.Container
| builtins.object
|
| Methods defined here:
|
| __contains__(self, k)
|
| __getitem__(self, k)
|
| __init__(self, mapping: 'Mapping', annotations=None, collection_annotations=None)
| Initialize Layer object.
|
| Parameters
| ----------
| annotations : Mapping[str, Any], optional
| By default, None.
| Annotations are metadata or soft constraints associated with tasks
| that dask schedulers may choose to respect:
| They signal intent without enforcing hard constraints.
| As such, they are primarily designed for use with the distributed
| scheduler. See the dask.annotate function for more information.
| collection_annotations : Mapping[str, Any], optional. By default, None.
| Experimental, intended to assist with visualizing the performance
| characteristics of Dask computations.
| These annotations are *not* passed to the distributed scheduler.
|
| __iter__(self)
|
| __len__(self)
|
| get_output_keys(self)
| Return a set of all output keys
|
| Output keys are all keys in the layer that might be referenced by
| other layers.
|
| Classes overriding this implementation should not cause the layer
| to be materialized.
|
| Returns
| -------
| keys: Set
| All output keys
|
| is_materialized(self)
| Return whether the layer is materialized or not
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from Layer:
|
| __copy__(self)
| Default shallow copy implementation
|
| clone(self, keys: 'set', seed: 'Hashable', bind_to: 'Key | None' = None) -> 'tuple[Layer, bool]'
| Clone selected keys in the layer, as well as references to keys in other
| layers
|
| Parameters
| ----------
| keys
| Keys to be replaced. This never includes keys not listed by
| :meth:`get_output_keys`. It must also include any keys that are outside
| of this layer that may be referenced by it.
| seed
| Common hashable used to alter the keys; see :func:`dask.base.clone_key`
| bind_to
| Optional key to bind the leaf nodes to. A leaf node here is one that does
| not reference any replaced keys; in other words it's a node where the
| replacement graph traversal stops; it may still have dependencies on
| non-replaced nodes.
| A bound node will not be computed until after ``bind_to`` has been computed.
|
| Returns
| -------
| - New layer
| - True if the ``bind_to`` key was injected anywhere; False otherwise
|
| Notes
| -----
| This method should be overridden by subclasses to avoid materializing the layer.
|
| cull(self, keys: 'set[Key]', all_hlg_keys: 'Collection[Key]') -> 'tuple[Layer, Mapping[Key, set[Key]]]'
| Remove unnecessary tasks from the layer
|
| In other words, return a new Layer with only the tasks required to
| calculate `keys` and a map of external key dependencies.
|
| Examples
| --------
| >>> inc = lambda x: x + 1
| >>> add = lambda x, y: x + y
| >>> d = MaterializedLayer({'x': 1, 'y': (inc, 'x'), 'out': (add, 'x', 10)})
| >>> _, deps = d.cull({'out'}, d.keys())
| >>> deps
| {'out': {'x'}, 'x': set()}
|
| Returns
| -------
| layer: Layer
| Culled layer
| deps: Map
| Map of external key dependencies
|
| get_dependencies(self, key: 'Key', all_hlg_keys: 'Collection[Key]') -> 'set'
| Get dependencies of `key` in the layer
|
| Parameters
| ----------
| key:
| The key to find dependencies of
| all_hlg_keys:
| All keys in the high level graph.
|
| Returns
| -------
| deps: set
| A set of dependencies
|
| has_legacy_tasks = <functools.cached_property object>
| Check if the layer has legacy tasks
|
| Legacy tasks are those that are not in the form of a tuple
| (function, *args, **kwargs).
|
| layer_info_dict(self)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from Layer:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from Layer:
|
| __orig_bases__ = (collections.abc.Mapping[str | int | float | tuple['K...
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.Mapping:
|
| __eq__(self, other)
| Return self==value.
|
| get(self, key, default=None)
| D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.
|
| items(self)
| D.items() -> a set-like object providing a view on D's items
|
| keys(self)
| D.keys() -> a set-like object providing a view on D's keys
|
| values(self)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from collections.abc.Mapping:
|
| __hash__ = None
|
| __reversed__ = None
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Collection:
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
FUNCTIONS
compute_layer_dependencies(layers)
Returns the dependencies between layers
register_highlevelgraph(hlg)
to_graphviz(hg, data_attributes=None, function_attributes=None, rankdir='BT', graph_attr=None, node_attr=None, edge_attr=None, **kwargs)
DATA
Graph = collections.abc.Mapping[str | int | float | tuple['Key', ...],...
Key = str | int | float | tuple['Key', ...]
normalize_token = <dask.utils.Dispatch object>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\highlevelgraph.py
dask.is_dask_collection
Python Library Documentation: function is_dask_collection in module dask.base
is_dask_collection(x) -> 'bool'
Returns ``True`` if ``x`` is a dask collection.
Parameters
----------
x : Any
Object to test.
Returns
-------
result : bool
``True`` if `x` is a Dask collection.
Notes
-----
The DaskCollection typing.Protocol implementation defines a Dask
collection as a class that returns a Mapping from the
``__dask_graph__`` method. This helper function existed before the
implementation of the protocol.
dask.istask
Python Library Documentation: function istask in module dask.core
istask(x)
Is x a runnable task?
A task is a tuple with a callable first argument
Examples
--------
>>> inc = lambda x: x + 1
>>> istask((inc, 1))
True
>>> istask(1)
False
dask.local
Python Library Documentation: module dask.local in dask
NAME
dask.local - Asynchronous Shared-Memory Scheduler for Dask Graphs.
DESCRIPTION
This scheduler coordinates several workers to execute tasks in a dask graph in
parallel. It depends on a ``concurrent.futures.Executor``
and a corresponding Queue for worker-to-scheduler communication.
It tries to execute tasks in an order which maintains a small memory footprint
throughout execution. It does this by running tasks that allow us to release
data resources.
Task Selection Policy
=====================
When we complete a task we add more data in to our set of available data; this
new data makes new tasks available. We preferentially choose tasks that were
just made available in a last-in-first-out fashion. We implement this as a
simple stack. This results in more depth-first rather than breadth first
behavior which encourages us to process batches of data to completion before
starting in on new data when possible.
When the addition of new data readies multiple tasks simultaneously we add
tasks to the stack in sorted order so that tasks with greater keynames are run
first. This can be handy to break ties in a predictable fashion.
State
=====
Many functions pass around a ``state`` variable that holds the current state of
the computation. This variable consists of several other dictionaries and
sets, explained below.
Constant state
--------------
1. dependencies: {x: [a, b ,c]} a,b,c, must be run before x
2. dependents: {a: [x, y]} a must run before x or y
Changing state
--------------
### Data
1. cache: available concrete data. {key: actual-data}
2. released: data that we've seen, used, and released because it is no longer
needed
### Jobs
1. ready: A fifo stack of ready-to-run tasks
2. running: A set of tasks currently in execution
3. finished: A set of finished tasks
4. waiting: which tasks are still waiting on others :: {key: {keys}}
Real-time equivalent of dependencies
5. waiting_data: available data to yet-to-be-run-tasks :: {key: {keys}}
Real-time equivalent of dependents
Examples
--------
>>> import pprint # doctest: +SKIP
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> dsk = {'x': 1, 'y': 2, 'z': (inc, 'x'), 'w': (add, 'z', 'y')} # doctest: +SKIP
>>> pprint.pprint(start_state_from_dask(dsk)) # doctest: +SKIP
{'cache': {'x': 1, 'y': 2},
'dependencies': {'w': {'z', 'y'}, 'x': set(), 'y': set(), 'z': {'x'}},
'dependents': defaultdict(None, {'w': set(), 'x': {'z'}, 'y': {'w'}, 'z': {'w'}}),
'finished': set(),
'ready': ['z'],
'released': set(),
'running': set(),
'waiting': {'w': {'z'}},
'waiting_data': {'x': {'z'}, 'y': {'w'}, 'z': {'w'}}}
Optimizations
=============
We build this scheduler with out-of-core array operations in mind. To this end
we have encoded some particular optimizations.
Compute to release data
-----------------------
When we choose a new task to execute we often have many options. Policies at
this stage are cheap and can significantly impact performance. One could
imagine policies that expose parallelism, drive towards a particular output,
etc..
Our current policy is to run tasks that were most recently made available.
Inlining computations
---------------------
We hold on to intermediate computations either in memory or on disk.
For very cheap computations that may emit new copies of the data, like
``np.transpose`` or possibly even ``x + 1`` we choose not to store these as
separate pieces of data / tasks. Instead we combine them with the computations
that require them. This may result in repeated computation but saves
significantly on space and computation complexity.
See the function ``inline_functions`` for more information.
CLASSES
concurrent.futures._base.Executor(builtins.object)
MultiprocessingPoolExecutor
SynchronousExecutor
class MultiprocessingPoolExecutor(concurrent.futures._base.Executor)
| MultiprocessingPoolExecutor(pool)
|
| Method resolution order:
| MultiprocessingPoolExecutor
| concurrent.futures._base.Executor
| builtins.object
|
| Methods defined here:
|
| __init__(self, pool)
| Initialize self. See help(type(self)) for accurate signature.
|
| submit(self, fn, *args, **kwargs)
| Submits a callable to be executed with the given arguments.
|
| Schedules the callable to be executed as fn(*args, **kwargs) and returns
| a Future instance representing the execution of the callable.
|
| Returns:
| A Future representing the given call.
|
| ----------------------------------------------------------------------
| Methods inherited from concurrent.futures._base.Executor:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| map(self, fn, *iterables, timeout=None, chunksize=1)
| Returns an iterator equivalent to map(fn, iter).
|
| Args:
| fn: A callable that will take as many arguments as there are
| passed iterables.
| timeout: The maximum number of seconds to wait. If None, then there
| is no limit on the wait time.
| chunksize: The size of the chunks the iterable will be broken into
| before being passed to a child process. This argument is only
| used by ProcessPoolExecutor; it is ignored by
| ThreadPoolExecutor.
|
| Returns:
| An iterator equivalent to: map(func, *iterables) but the calls may
| be evaluated out-of-order.
|
| Raises:
| TimeoutError: If the entire result iterator could not be generated
| before the given timeout.
| Exception: If fn(*args) raises for any values.
|
| shutdown(self, wait=True, *, cancel_futures=False)
| Clean-up the resources associated with the Executor.
|
| It is safe to call this method several times. Otherwise, no other
| methods can be called after this one.
|
| Args:
| wait: If True then shutdown will not return until all running
| futures have finished executing and the resources used by the
| executor have been reclaimed.
| cancel_futures: If True then shutdown will cancel all pending
| futures. Futures that are completed or running will not be
| cancelled.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from concurrent.futures._base.Executor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class SynchronousExecutor(concurrent.futures._base.Executor)
| Method resolution order:
| SynchronousExecutor
| concurrent.futures._base.Executor
| builtins.object
|
| Methods defined here:
|
| submit(self, fn, *args, **kwargs)
| Submits a callable to be executed with the given arguments.
|
| Schedules the callable to be executed as fn(*args, **kwargs) and returns
| a Future instance representing the execution of the callable.
|
| Returns:
| A Future representing the given call.
|
| ----------------------------------------------------------------------
| Methods inherited from concurrent.futures._base.Executor:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| map(self, fn, *iterables, timeout=None, chunksize=1)
| Returns an iterator equivalent to map(fn, iter).
|
| Args:
| fn: A callable that will take as many arguments as there are
| passed iterables.
| timeout: The maximum number of seconds to wait. If None, then there
| is no limit on the wait time.
| chunksize: The size of the chunks the iterable will be broken into
| before being passed to a child process. This argument is only
| used by ProcessPoolExecutor; it is ignored by
| ThreadPoolExecutor.
|
| Returns:
| An iterator equivalent to: map(func, *iterables) but the calls may
| be evaluated out-of-order.
|
| Raises:
| TimeoutError: If the entire result iterator could not be generated
| before the given timeout.
| Exception: If fn(*args) raises for any values.
|
| shutdown(self, wait=True, *, cancel_futures=False)
| Clean-up the resources associated with the Executor.
|
| It is safe to call this method several times. Otherwise, no other
| methods can be called after this one.
|
| Args:
| wait: If True then shutdown will not return until all running
| futures have finished executing and the resources used by the
| executor have been reclaimed.
| cancel_futures: If True then shutdown will cancel all pending
| futures. Futures that are completed or running will not be
| cancelled.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from concurrent.futures._base.Executor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
batch_execute_tasks(it)
Batch computing of multiple tasks with `execute_task`
default_get_id()
Default get_id
default_pack_exception(e, dumps)
execute_task(key, task_info, dumps, loads, get_id, pack_exception)
Compute task and handle all administration
See Also
--------
_execute_task : actually execute task
finish_task(dsk, key, state, results, sortkey, delete=True, release_data=<function release_data at 0x00000183927D77E0>)
Update execution state after a task finishes
Mutates. This should run atomically (with a lock).
get_apply_async(apply_async, num_workers, *args, **kwargs)
get_async(submit, num_workers, dsk, result, cache=None, get_id=<function default_get_id at 0x00000183927D79C0>, rerun_exceptions_locally=None, pack_exception=<function default_pack_exception at 0x00000183927D7A60>, raise_exception=<function reraise at 0x00000183927D7B00>, callbacks=None, dumps=<function identity at 0x00000183927D7BA0>, loads=<function identity at 0x00000183927D7BA0>, chunksize=None, **kwargs)
Asynchronous get function
This is a general version of various asynchronous schedulers for dask. It
takes a ``concurrent.futures.Executor.submit`` function to form a more
specific ``get`` method that walks through the dask array with parallel
workers, avoiding repeat computation and minimizing memory use.
Parameters
----------
submit : function
A ``concurrent.futures.Executor.submit`` function
num_workers : int
The number of workers that task submissions can be spread over
dsk : dict
A dask dictionary specifying a workflow
result : key or list of keys
Keys corresponding to desired data
cache : dict-like, optional
Temporary storage of results
get_id : callable, optional
Function to return the worker id, takes no arguments. Examples are
`threading.current_thread` and `multiprocessing.current_process`.
rerun_exceptions_locally : bool, optional
Whether to rerun failing tasks in local process to enable debugging
(False by default)
pack_exception : callable, optional
Function to take an exception and ``dumps`` method, and return a
serialized tuple of ``(exception, traceback)`` to send back to the
scheduler. Default is to just raise the exception.
raise_exception : callable, optional
Function that takes an exception and a traceback, and raises an error.
callbacks : tuple or list of tuples, optional
Callbacks are passed in as tuples of length 5. Multiple sets of
callbacks may be passed in as a list of tuples. For more information,
see the dask.diagnostics documentation.
dumps: callable, optional
Function to serialize task data and results to communicate between
worker and parent. Defaults to identity.
loads: callable, optional
Inverse function of `dumps`. Defaults to identity.
chunksize: int, optional
Size of chunks to use when dispatching work. Defaults to 1.
If -1, will be computed to evenly divide ready work across workers.
See Also
--------
threaded.get
get_sync(dsk: 'Mapping', keys: 'Sequence[Key] | Key', **kwargs)
A naive synchronous version of get_async
Can be useful for debugging.
identity(x)
Identity function. Returns x.
>>> identity(3)
3
nested_get(ind, coll)
Get nested index from collection
Examples
--------
>>> nested_get(1, 'abc')
'b'
>>> nested_get([1, 0], 'abc')
('b', 'a')
>>> nested_get([[1, 0], [0, 1]], 'abc')
(('b', 'a'), ('a', 'b'))
queue_get(q)
# Python 3 windows Queue.get doesn't handle interrupts properly. To
# workaround this we poll at a sufficiently large interval that it
# shouldn't affect performance, but small enough that users trying to kill
# an application shouldn't care.
release_data(key, state, delete=True)
Remove data from temporary storage
See Also
--------
finish_task
reraise(exc, tb=None)
sortkey(item)
Sorting key function that is robust to different types
Both strings and tuples are common key types in dask graphs.
However In Python 3 one can not compare strings with tuples directly.
This function maps many types to a form where they can be compared
Examples
--------
>>> sortkey('Hello')
('str', 'Hello')
>>> sortkey(('x', 1))
('tuple', ('x', 1))
start_state_from_dask(dsk, cache=None, sortkey=None, keys=None)
Start state from a dask
Examples
--------
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> dsk = {'x': 1, 'y': 2, 'z': (inc, 'x'), 'w': (add, 'z', 'y')} # doctest: +SKIP
>>> from pprint import pprint # doctest: +SKIP
>>> pprint(start_state_from_dask(dsk)) # doctest: +SKIP
{'cache': {'x': 1, 'y': 2},
'dependencies': {'w': {'z', 'y'}, 'x': set(), 'y': set(), 'z': {'x'}},
'dependents': defaultdict(None, {'w': set(), 'x': {'z'}, 'y': {'w'}, 'z': {'w'}}),
'finished': set(),
'ready': ['z'],
'released': set(),
'running': set(),
'waiting': {'w': {'z'}},
'waiting_data': {'x': {'z'}, 'y': {'w'}, 'z': {'w'}}}
submit_apply_async(apply_async, fn, *args, **kwargs)
DATA
Key = str | int | float | tuple['Key', ...]
synchronous_executor = <dask.local.SynchronousExecutor object>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\local.py
dask.multiprocessing
Python Library Documentation: module dask.multiprocessing in dask
NAME
dask.multiprocessing
CLASSES
builtins.Exception(builtins.BaseException)
RemoteException
class RemoteException(builtins.Exception)
| RemoteException(exception, traceback)
|
| Remote Exception
|
| Contains the exception and traceback from a remotely run task
|
| Method resolution order:
| RemoteException
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Methods defined here:
|
| __dir__(self)
| Default dir() implementation.
|
| __getattr__(self, key)
|
| __init__(self, exception, traceback)
| Initialize self. See help(type(self)) for accurate signature.
|
| __str__(self)
| Return str(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
FUNCTIONS
default_initializer()
get(dsk: 'Mapping', keys: 'Sequence[Key] | Key', num_workers=None, func_loads=None, func_dumps=None, optimize_graph=True, pool=None, initializer=None, chunksize=None, **kwargs)
Multiprocessed get function appropriate for Bags
Parameters
----------
dsk : dict
dask graph
keys : object or list
Desired results from graph
num_workers : int
Number of worker processes (defaults to number of cores)
func_dumps : function
Function to use for function serialization (defaults to cloudpickle.dumps)
func_loads : function
Function to use for function deserialization (defaults to cloudpickle.loads)
optimize_graph : bool
If True [default], `fuse` is applied to the graph before computation.
pool : Executor or Pool
Some sort of `Executor` or `Pool` to use
initializer: function
Ignored if ``pool`` has been set.
Function to initialize a worker process before running any tasks in it.
chunksize: int, optional
Size of chunks to use when dispatching work.
Defaults to 6 as some batching is helpful.
If -1, will be computed to evenly divide ready work across workers.
get_context()
Return the current multiprocessing context.
initialize_worker_process(user_initializer=None)
Initialize a worker process before running any tasks in it.
pack_exception(e, dumps)
remote_exception(exc: 'Exception', tb) -> 'Exception'
Metaclass that wraps exception type in RemoteException
reraise(exc, tb=None)
warn(...)
Issue a warning, or maybe ignore it or raise an exception.
message
Text of the warning message.
category
The Warning category subclass. Defaults to UserWarning.
stacklevel
How far up the call stack to make this warning appear. A value of 2 for
example attributes the warning to the caller of the code calling warn().
source
If supplied, the destroyed object which emitted a ResourceWarning
skip_file_prefixes
An optional tuple of module filename prefixes indicating frames to skip
during stacklevel computations for stack frame attribution.
DATA
CPU_COUNT = 4
Key = str | int | float | tuple['Key', ...]
__annotations__ = {'exceptions': 'dict[type[Exception], type[Exception...
exceptions = {}
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\multiprocessing.py
dask.optimization
Python Library Documentation: module dask.optimization in dask
NAME
dask.optimization
CLASSES
enum.Enum(builtins.object)
Default
class Default(enum.Enum)
| Default(*values)
|
| # PEP-484 compliant singleton constant
| # https://www.python.org/dev/peps/pep-0484/#support-for-singleton-types-in-unions
|
| Method resolution order:
| Default
| enum.Enum
| builtins.object
|
| Data and other attributes defined here:
|
| token = <default>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from enum.Enum:
|
| name
| The name of the Enum member.
|
| value
| The value of the Enum member.
|
| ----------------------------------------------------------------------
| Static methods inherited from enum.EnumType:
|
| __contains__(value)
| Return True if `value` is in `cls`.
|
| `value` is in `cls` if:
| 1) `value` is a member of `cls`, or
| 2) `value` is the value of one of the `cls`'s members.
|
| __getitem__(name)
| Return the member matching `name`.
|
| __iter__()
| Return members in definition order.
|
| __len__()
| Return the number of members (no aliases)
|
| ----------------------------------------------------------------------
| Readonly properties inherited from enum.EnumType:
|
| __members__
| Returns a mapping of member name->value.
|
| This mapping lists all enum members, including aliases. Note that this
| is a read-only view of the internal mapping.
FUNCTIONS
cull(dsk, keys)
Return new dask with only the tasks required to calculate keys.
In other words, remove unnecessary tasks from dask.
``keys`` may be a single key or list of keys.
Examples
--------
>>> def inc(x):
... return x + 1
>>> def add(x, y):
... return x + y
>>> d = {'x': 1, 'y': (inc, 'x'), 'out': (add, 'x', 10)}
>>> dsk, dependencies = cull(d, 'out')
>>> dsk # doctest: +ELLIPSIS
{'out': (<function add at ...>, 'x', 10), 'x': 1}
>>> dependencies # doctest: +ELLIPSIS
{'out': ['x'], 'x': []}
Returns
-------
dsk: culled dask graph
dependencies: Dict mapping {key: [deps]}. Useful side effect to accelerate
other optimizations, notably fuse.
default_fused_keys_renamer(keys, max_fused_key_length=120)
Create new keys for ``fuse`` tasks.
The optional parameter `max_fused_key_length` is used to limit the maximum string length for each renamed key.
If this parameter is set to `None`, there is no limit.
default_fused_linear_keys_renamer(keys)
Create new keys for fused tasks
functions_of(task)
Set of functions contained within nested task
Examples
--------
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> mul = lambda x, y: x * y
>>> task = (add, (mul, 1, 2), (inc, 3)) # doctest: +SKIP
>>> functions_of(task) # doctest: +SKIP
set([add, mul, inc])
fuse(dsk, keys=None, dependencies=None, ave_width=<default>, max_width=<default>, max_height=<default>, max_depth_new_edges=<default>, rename_keys=<default>)
Fuse tasks that form reductions; more advanced than ``fuse_linear``
This trades parallelism opportunities for faster scheduling by making tasks
less granular. It can replace ``fuse_linear`` in optimization passes.
This optimization applies to all reductions--tasks that have at most one
dependent--so it may be viewed as fusing "multiple input, single output"
groups of tasks into a single task. There are many parameters to fine
tune the behavior, which are described below. ``ave_width`` is the
natural parameter with which to compare parallelism to granularity, so
it should always be specified. Reasonable values for other parameters
will be determined using ``ave_width`` if necessary.
Parameters
----------
dsk: dict
dask graph
keys: list or set, optional
Keys that must remain in the returned dask graph
dependencies: dict, optional
{key: [list-of-keys]}. Must be a list to provide count of each key
This optional input often comes from ``cull``
ave_width: float (default 1)
Upper limit for ``width = num_nodes / height``, a good measure of
parallelizability.
dask.config key: ``optimization.fuse.ave-width``
max_width: int (default infinite)
Don't fuse if total width is greater than this.
dask.config key: ``optimization.fuse.max-width``
max_height: int or None (default None)
Don't fuse more than this many levels. Set to None to dynamically
adjust to ``1.5 + ave_width * log(ave_width + 1)``.
dask.config key: ``optimization.fuse.max-height``
max_depth_new_edges: int or None (default None)
Don't fuse if new dependencies are added after this many levels.
Set to None to dynamically adjust to ave_width * 1.5.
dask.config key: ``optimization.fuse.max-depth-new-edges``
rename_keys: bool or func, optional (default True)
Whether to rename the fused keys with ``default_fused_keys_renamer``
or not. Renaming fused keys can keep the graph more understandable
and comprehensive, but it comes at the cost of additional processing.
If False, then the top-most key will be used. For advanced usage, a
function to create the new name is also accepted.
dask.config key: ``optimization.fuse.rename-keys``
Returns
-------
dsk
output graph with keys fused
dependencies
dict mapping dependencies after fusion. Useful side effect to accelerate other
downstream optimizations.
fuse_linear(dsk, keys=None, dependencies=None, rename_keys=True)
Return new dask graph with linear sequence of tasks fused together.
If specified, the keys in ``keys`` keyword argument are *not* fused.
Supply ``dependencies`` from output of ``cull`` if available to avoid
recomputing dependencies.
**This function is mostly superseded by ``fuse``**
Parameters
----------
dsk: dict
keys: list
dependencies: dict, optional
{key: [list-of-keys]}. Must be a list to provide count of each key
This optional input often comes from ``cull``
rename_keys: bool or func, optional
Whether to rename fused keys with ``default_fused_linear_keys_renamer``
or not. Renaming fused keys can keep the graph more understandable
and comprehensive, but it comes at the cost of additional processing.
If False, then the top-most key will be used. For advanced usage, a
func is also accepted, ``new_key = rename_keys(fused_key_list)``.
Examples
--------
>>> def inc(x):
... return x + 1
>>> def add(x, y):
... return x + y
>>> d = {'a': 1, 'b': (inc, 'a'), 'c': (inc, 'b')}
>>> dsk, dependencies = fuse(d)
>>> dsk # doctest: +SKIP
{'a-b-c': (inc, (inc, 1)), 'c': 'a-b-c'}
>>> dsk, dependencies = fuse(d, rename_keys=False)
>>> dsk # doctest: +ELLIPSIS
{'c': (<function inc at ...>, (<function inc at ...>, 1))}
>>> dsk, dependencies = fuse(d, keys=['b'], rename_keys=False)
>>> dsk # doctest: +ELLIPSIS
{'b': (<function inc at ...>, 1), 'c': (<function inc at ...>, 'b')}
Returns
-------
dsk: output graph with keys fused
dependencies: dict mapping dependencies after fusion. Useful side effect
to accelerate other downstream optimizations.
inline(dsk, keys=None, inline_constants=True, dependencies=None)
Return new dask with the given keys inlined with their values.
Inlines all constants if ``inline_constants`` keyword is True. Note that
the constant keys will remain in the graph, to remove them follow
``inline`` with ``cull``.
Examples
--------
>>> def inc(x):
... return x + 1
>>> def add(x, y):
... return x + y
>>> d = {'x': 1, 'y': (inc, 'x'), 'z': (add, 'x', 'y')}
>>> inline(d) # doctest: +ELLIPSIS
{'x': 1, 'y': (<function inc at ...>, 1), 'z': (<function add at ...>, 1, 'y')}
>>> inline(d, keys='y') # doctest: +ELLIPSIS
{'x': 1, 'y': (<function inc at ...>, 1), 'z': (<function add at ...>, 1, (<function inc at ...>, 1))}
>>> inline(d, keys='y', inline_constants=False) # doctest: +ELLIPSIS
{'x': 1, 'y': (<function inc at ...>, 'x'), 'z': (<function add at ...>, 'x', (<function inc at ...>, 'x'))}
inline_functions(dsk, output, fast_functions=None, inline_constants=False, dependencies=None)
Inline cheap functions into larger operations
Examples
--------
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> double = lambda x: x * 2
>>> dsk = {'out': (add, 'i', 'd'), # doctest: +SKIP
... 'i': (inc, 'x'),
... 'd': (double, 'y'),
... 'x': 1, 'y': 1}
>>> inline_functions(dsk, [], [inc]) # doctest: +SKIP
{'out': (add, (inc, 'x'), 'd'),
'd': (double, 'y'),
'x': 1, 'y': 1}
Protect output keys. In the example below ``i`` is not inlined because it
is marked as an output key.
>>> inline_functions(dsk, ['i', 'out'], [inc, double]) # doctest: +SKIP
{'out': (add, 'i', (double, 'y')),
'i': (inc, 'x'),
'x': 1, 'y': 1}
unwrap_partial(func)
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\optimization.py
dask.optimize
Python Library Documentation: function optimize in module dask.base
optimize(*args, traverse=True, **kwargs)
Optimize several dask collections at once.
Returns equivalent dask collections that all share the same merged and
optimized underlying graph. This can be useful if converting multiple
collections to delayed objects, or to manually apply the optimizations at
strategic points.
Note that in most cases you shouldn't need to call this function directly.
Warning::
This function triggers a materialization of the collections and looses
any annotations attached to HLG layers.
Parameters
----------
*args : objects
Any number of objects. If a dask object, its graph is optimized and
merged with all those of all other dask objects before returning an
equivalent dask collection. Non-dask arguments are passed through
unchanged.
traverse : bool, optional
By default dask traverses builtin python collections looking for dask
objects passed to ``optimize``. For large collections this can be
expensive. If none of the arguments contain any dask objects, set
``traverse=False`` to avoid doing this traversal.
optimizations : list of callables, optional
Additional optimization passes to perform.
**kwargs
Extra keyword arguments to forward to the optimization passes.
Examples
--------
>>> import dask
>>> import dask.array as da
>>> a = da.arange(10, chunks=2).sum()
>>> b = da.arange(10, chunks=2).mean()
>>> a2, b2 = dask.optimize(a, b)
>>> a2.compute() == a.compute()
np.True_
>>> b2.compute() == b.compute()
np.True_
dask.order
Python Library Documentation: module dask.order in dask
NAME
dask.order
CLASSES
builtins.tuple(builtins.object)
Order
OrderInfo
class Order(builtins.tuple)
| Order(priority: ForwardRef('int'), critical_path: ForwardRef('float | int'))
|
| Order(priority, critical_path)
|
| Method resolution order:
| Order
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __getnewargs__(self) from collections.Order
| Return self as a plain tuple. Used by copy and pickle.
|
| __repr__(self) from collections.Order
| Return a nicely formatted representation string
|
| _asdict(self) from collections.Order
| Return a new dict which maps field names to their values.
|
| _replace(self, /, **kwds) from collections.Order
| Return a new Order object replacing specified fields with new values
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| _make(iterable) from collections.Order
| Make a new Order object from a sequence or iterable
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(_cls, priority: ForwardRef('int'), critical_path: ForwardRef('float | int')) from namedtuple_Order.Order
| Create new instance of Order(priority, critical_path)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| priority
| Alias for field number 0
|
| critical_path
| Alias for field number 1
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'critical_path': ForwardRef('float | int'), 'priori...
|
| __match_args__ = ('priority', 'critical_path')
|
| __orig_bases__ = (<function NamedTuple>,)
|
| _field_defaults = {}
|
| _fields = ('priority', 'critical_path')
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class OrderInfo(builtins.tuple)
| OrderInfo(order, age, num_data_when_run, num_data_when_released, num_dependencies_freed)
|
| OrderInfo(order, age, num_data_when_run, num_data_when_released, num_dependencies_freed)
|
| Method resolution order:
| OrderInfo
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __getnewargs__(self) from collections.OrderInfo
| Return self as a plain tuple. Used by copy and pickle.
|
| __repr__(self) from collections.OrderInfo
| Return a nicely formatted representation string
|
| _asdict(self) from collections.OrderInfo
| Return a new dict which maps field names to their values.
|
| _replace(self, /, **kwds) from collections.OrderInfo
| Return a new OrderInfo object replacing specified fields with new values
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| _make(iterable) from collections.OrderInfo
| Make a new OrderInfo object from a sequence or iterable
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(_cls, order, age, num_data_when_run, num_data_when_released, num_dependencies_freed) from namedtuple_OrderInfo.OrderInfo
| Create new instance of OrderInfo(order, age, num_data_when_run, num_data_when_released, num_dependencies_freed)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| order
| Alias for field number 0
|
| age
| Alias for field number 1
|
| num_data_when_run
| Alias for field number 2
|
| num_data_when_released
| Alias for field number 3
|
| num_dependencies_freed
| Alias for field number 4
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('order', 'age', 'num_data_when_run', 'num_data_when_...
|
| _field_defaults = {}
|
| _fields = ('order', 'age', 'num_data_when_run', 'num_data_when_release...
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
FUNCTIONS
diagnostics(dsk: 'MutableMapping[Key, Any]', o: 'Mapping[Key, int] | None' = None, dependencies: 'MutableMapping[Key, set[Key]] | None' = None) -> 'tuple[dict[Key, OrderInfo], list[int]]'
Simulate runtime metrics as though running tasks one at a time in order.
These diagnostics can help reveal behaviors of and issues with ``order``.
Returns a dict of `namedtuple("OrderInfo")` and a list of the number of outputs held over time.
OrderInfo fields:
- order : the order in which the node is run.
- age : how long the output of a node is held.
- num_data_when_run : the number of outputs held in memory when a node is run.
- num_data_when_released : the number of outputs held in memory when the output is released.
- num_dependencies_freed : the number of dependencies freed by running the node.
ndependencies(dependencies: 'Mapping[Key, set[Key]]', dependents: 'Mapping[Key, set[Key]]') -> 'tuple[dict[Key, int], dict[Key, int]]'
Number of total data elements on which this key depends
For each key we return the number of tasks that must be run for us to run
this task.
Examples
--------
>>> inc = lambda x: x + 1
>>> dsk = {'a': 1, 'b': (inc, 'a'), 'c': (inc, 'b')}
>>> dependencies, dependents = get_deps(dsk)
>>> num_dependencies, total_dependencies = ndependencies(dependencies, dependents)
>>> sorted(total_dependencies.items())
[('a', 1), ('b', 2), ('c', 3)]
Returns
-------
num_dependencies: Dict[key, int]
total_dependencies: Dict[key, int]
order(dsk: 'Mapping[Key, Any]', dependencies: 'Mapping[Key, set[Key]] | None' = None, *, return_stats: 'bool' = False) -> 'dict[Key, Order] | dict[Key, int]'
Order nodes in dask graph
This produces an ordering over our tasks that we use to break ties when
executing. We do this ahead of time to reduce a bit of stress on the
scheduler and also to assist in static analysis.
This currently traverses the graph as a single-threaded scheduler would
traverse it.
Examples
--------
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> dsk = {'a': 1, 'b': 2, 'c': (inc, 'a'), 'd': (add, 'b', 'c')}
>>> order(dsk)
{'a': 0, 'c': 1, 'b': 2, 'd': 3}
sanitize_dsk(dsk: 'MutableMapping[Key, Any]') -> 'dict'
Take a dask graph and replace callables with a dummy function and remove
payload data like numpy arrays, dataframes, etc.
DATA
Key = str | int | float | tuple['Key', ...]
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\order.py
dask.persist
Python Library Documentation: function persist in module dask.base
persist(*args, traverse=True, optimize_graph=True, scheduler=None, **kwargs)
Persist multiple Dask collections into memory
This turns lazy Dask collections into Dask collections with the same
metadata, but now with their results fully computed or actively computing
in the background.
For example a lazy dask.array built up from many lazy calls will now be a
dask.array of the same shape, dtype, chunks, etc., but now with all of
those previously lazy tasks either computed in memory as many small :class:`numpy.array`
(in the single-machine case) or asynchronously running in the
background on a cluster (in the distributed case).
This function operates differently if a ``dask.distributed.Client`` exists
and is connected to a distributed scheduler. In this case this function
will return as soon as the task graph has been submitted to the cluster,
but before the computations have completed. Computations will continue
asynchronously in the background. When using this function with the single
machine scheduler it blocks until the computations have finished.
When using Dask on a single machine you should ensure that the dataset fits
entirely within memory.
Examples
--------
>>> df = dd.read_csv('/path/to/*.csv') # doctest: +SKIP
>>> df = df[df.name == 'Alice'] # doctest: +SKIP
>>> df['in-debt'] = df.balance < 0 # doctest: +SKIP
>>> df = df.persist() # triggers computation # doctest: +SKIP
>>> df.value().min() # future computations are now fast # doctest: +SKIP
-10
>>> df.value().max() # doctest: +SKIP
100
>>> from dask import persist # use persist function on multiple collections
>>> a, b = persist(a, b) # doctest: +SKIP
Parameters
----------
*args: Dask collections
scheduler : string, optional
Which scheduler to use like "threads", "synchronous" or "processes".
If not provided, the default is to check the global settings first,
and then fall back to the collection defaults.
traverse : bool, optional
By default dask traverses builtin python collections looking for dask
objects passed to ``persist``. For large collections this can be
expensive. If none of the arguments contain any dask objects, set
``traverse=False`` to avoid doing this traversal.
optimize_graph : bool, optional
If True [default], the graph is optimized before computation.
Otherwise the graph is run as is. This can be useful for debugging.
**kwargs
Extra keywords to forward to the scheduler function.
Returns
-------
New dask collections backed by in-memory data
dask.sizeof
Python Library Documentation: module dask.sizeof in dask
NAME
dask.sizeof
CLASSES
builtins.object
SimpleSizeof
class SimpleSizeof(builtins.object)
| Sentinel class to mark a class to be skipped by the dispatcher. This only
| works if this sentinel mixin is first in the mro.
|
| Examples
| --------
| >>> def _get_gc_overhead():
| ... class _CustomObject:
| ... def __sizeof__(self):
| ... return 0
| ...
| ... return sys.getsizeof(_CustomObject())
|
| >>> class TheAnswer(SimpleSizeof):
| ... def __sizeof__(self):
| ... # Sizeof always add overhead of an object for GC
| ... return 42 - _get_gc_overhead()
|
| >>> sizeof(TheAnswer())
| 42
|
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
register_cupy()
register_numba()
register_numpy()
register_pandas()
register_pyarrow()
register_rmm()
register_spmatrix()
register_xarray()
sizeof_array(o)
sizeof_blocked(d)
sizeof_bytes(o)
sizeof_default(o)
sizeof_memoryview(o)
sizeof_python_collection(seq)
sizeof_python_dict(d)
DATA
logger = <Logger dask.sizeof (WARNING)>
sizeof = <dask.utils.Dispatch object>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\sizeof.py
dask.system
Python Library Documentation: module dask.system in dask
NAME
dask.system
FUNCTIONS
cpu_count() -> 'int'
Get the available CPU count for this system.
Takes the minimum value from the following locations:
- Total system cpus available on the host.
- CPU Affinity (if set)
- Cgroups limit (if set)
DATA
CPU_COUNT = 4
__all__ = ('cpu_count', 'CPU_COUNT')
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\system.py
dask.threaded
Python Library Documentation: module dask.threaded in dask
NAME
dask.threaded - A threaded shared-memory scheduler
DESCRIPTION
See local.py
CLASSES
concurrent.futures.thread.ThreadPoolExecutor(concurrent.futures._base.Executor)
ContextAwareThreadPoolExecutor
class ContextAwareThreadPoolExecutor(concurrent.futures.thread.ThreadPoolExecutor)
| ContextAwareThreadPoolExecutor(max_workers=None, thread_name_prefix='', initializer=None, initargs=())
|
| Variant ThreadPoolExecutor that propagates contextvars
| from the submitting thread to the worker threads.
|
| With a vanilla ThreadPoolExecutor, contextvars are automatically propagated on
| CPython 3.14t (noGIL) if and only if they are set before the worker threads are
| started.
| This variant propagates contextvars on all Python interpreters and also when the
| worker threads are already warm when the variables are set in the submitting thread.
|
| This also affects catching warnings, which on 3.14t use contextvars.
|
| See Also
| --------
| https://docs.python.org/3/using/cmdline.html#envvar-PYTHON_THREAD_INHERIT_CONTEXT
| https://docs.python.org/3/using/cmdline.html#envvar-PYTHON_CONTEXT_AWARE_WARNINGS
|
| Method resolution order:
| ContextAwareThreadPoolExecutor
| concurrent.futures.thread.ThreadPoolExecutor
| concurrent.futures._base.Executor
| builtins.object
|
| Methods defined here:
|
| submit(self, fn, /, *args, **kwargs)
| Submits a callable to be executed with the given arguments.
|
| Schedules the callable to be executed as fn(*args, **kwargs) and returns
| a Future instance representing the execution of the callable.
|
| Returns:
| A Future representing the given call.
|
| ----------------------------------------------------------------------
| Methods inherited from concurrent.futures.thread.ThreadPoolExecutor:
|
| __init__(self, max_workers=None, thread_name_prefix='', initializer=None, initargs=())
| Initializes a new ThreadPoolExecutor instance.
|
| Args:
| max_workers: The maximum number of threads that can be used to
| execute the given calls.
| thread_name_prefix: An optional name prefix to give our threads.
| initializer: A callable used to initialize worker threads.
| initargs: A tuple of arguments to pass to the initializer.
|
| shutdown(self, wait=True, *, cancel_futures=False)
| Clean-up the resources associated with the Executor.
|
| It is safe to call this method several times. Otherwise, no other
| methods can be called after this one.
|
| Args:
| wait: If True then shutdown will not return until all running
| futures have finished executing and the resources used by the
| executor have been reclaimed.
| cancel_futures: If True then shutdown will cancel all pending
| futures. Futures that are completed or running will not be
| cancelled.
|
| ----------------------------------------------------------------------
| Methods inherited from concurrent.futures._base.Executor:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| map(self, fn, *iterables, timeout=None, chunksize=1)
| Returns an iterator equivalent to map(fn, iter).
|
| Args:
| fn: A callable that will take as many arguments as there are
| passed iterables.
| timeout: The maximum number of seconds to wait. If None, then there
| is no limit on the wait time.
| chunksize: The size of the chunks the iterable will be broken into
| before being passed to a child process. This argument is only
| used by ProcessPoolExecutor; it is ignored by
| ThreadPoolExecutor.
|
| Returns:
| An iterator equivalent to: map(func, *iterables) but the calls may
| be evaluated out-of-order.
|
| Raises:
| TimeoutError: If the entire result iterator could not be generated
| before the given timeout.
| Exception: If fn(*args) raises for any values.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from concurrent.futures._base.Executor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
Lock = allocate_lock(...)
allocate_lock() -> lock object
(allocate() is an obsolete synonym)
Create a new lock object. See help(type(threading.Lock())) for
information about locks.
get(dsk: 'Mapping', keys: 'Sequence[Key] | Key', cache=None, num_workers=None, pool=None, **kwargs)
Threaded cached implementation of dask.get
Parameters
----------
dsk: dict
A dask dictionary specifying a workflow
keys: key or list of keys
Keys corresponding to desired data
num_workers: integer of thread count
The number of threads to use in the ThreadPool that will actually execute tasks
cache: dict-like (optional)
Temporary storage of results
Examples
--------
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> dsk = {'x': 1, 'y': 2, 'z': (inc, 'x'), 'w': (add, 'z', 'y')}
>>> get(dsk, 'w')
4
>>> get(dsk, ['w', 'y'])
(4, 2)
pack_exception(e, dumps)
DATA
CPU_COUNT = 4
Key = str | int | float | tuple['Key', ...]
__annotations__ = {'default_pool': 'Executor | None', 'pools': 'defaul...
default_pool = None
main_thread = <_MainThread(MainThread, started 46512)>
pools = defaultdict(<class 'dict'>, {})
pools_lock = <unlocked _thread.lock object>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\threaded.py
dask.tokenize
Python Library Documentation: module dask.tokenize in dask
NAME
dask.tokenize
CLASSES
builtins.RuntimeError(builtins.Exception)
TokenizationError
class TokenizationError(builtins.RuntimeError)
| Method resolution order:
| TokenizationError
| builtins.RuntimeError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.RuntimeError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.RuntimeError:
|
| __new__(*args, **kwargs) class method of builtins.RuntimeError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
FUNCTIONS
normalize_bound_method(meth)
normalize_builtin_function_or_method(func)
normalize_compose(func)
normalize_dict(d)
normalize_literal(lit)
normalize_object(o)
normalize_ordered_dict(d)
normalize_partial(func)
normalize_seq(seq)
normalize_set(s)
register_numba()
register_numpy()
register_pandas()
register_pyarrow()
tokenize(*args: 'object', ensure_deterministic: 'bool | None' = None, **kwargs: 'object') -> 'str'
Deterministic token
>>> tokenize([1, 2, '3']) # doctest: +SKIP
'06961e8de572e73c2e74b51348177918'
>>> tokenize('Hello') == tokenize('Hello')
True
Parameters
----------
args, kwargs:
objects to tokenize
ensure_deterministic: bool, optional
If True, raise TokenizationError if the objects cannot be deterministically
tokenized, e.g. two identical objects will return different tokens.
Defaults to the `tokenize.ensure-deterministic` configuration parameter.
DATA
__annotations__ = {'_ENSURE_DETERMINISTIC': 'ContextVar[bool | None]',...
normalize_token = <dask.utils.Dispatch object>
tokenize_lock = <unlocked _thread.RLock object owner=0 count=0>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\tokenize.py
dask.typing
Python Library Documentation: module dask.typing in dask
NAME
dask.typing
CLASSES
typing.Protocol(typing.Generic)
DaskCollection
HLGDaskCollection(DaskCollection, typing.Protocol)
PostPersistCallable
SchedulerGetCallable
class DaskCollection(typing.Protocol)
| DaskCollection(*args, **kwargs)
|
| Protocol defining the interface of a Dask collection.
|
| Method resolution order:
| DaskCollection
| typing.Protocol
| typing.Generic
| builtins.object
|
| Methods defined here:
|
| __dask_graph__(self) -> 'Graph'
| The Dask task graph.
|
| The core Dask collections (Array, DataFrame, Bag, and Delayed)
| use a :py:class:`~dask.highlevelgraph.HighLevelGraph` to
| represent the collection task graph. It is also possible to
| represent the task graph as a low level graph using a Python
| dictionary.
|
| Returns
| -------
| Mapping
| The Dask task graph. If the instance returns a
| :py:class:`dask.highlevelgraph.HighLevelGraph` then the
| :py:func:`__dask_layers__` method must be implemented, as
| defined by the :py:class:`~dask.typing.HLGDaskCollection`
| protocol.
|
| __dask_keys__(self) -> 'NestedKeys'
| The output keys of the task graph.
|
| Note that there are additional constraints on keys for a Dask
| collection than those described in the :doc:`task graph
| specification documentation <spec>`. These additional
| constraints are described below.
|
| All keys must either be non-empty strings or tuples where the first element is a
| non-empty string, followed by zero or more arbitrary str, bytes, int, float, or
| tuples thereof. The non-empty string is commonly known as the *collection name*.
| All collections embedded in the dask package have exactly one name, but this is
| not a requirement.
|
| These are all valid outputs:
|
| - ``[]``
| - ``["x", "y"]``
| - ``[[("y", "a", 0), ("y", "a", 1)], [("y", "b", 0), ("y", "b", 1)]``
|
| Returns
| -------
| list
| A possibly nested list of keys that represent the outputs
| of the graph. After computation, the results will be
| returned in the same layout, with the keys replaced with
| their corresponding outputs.
|
| __dask_postcompute__(self) -> 'tuple[PostComputeCallable, tuple]'
| Finalizer function and optional arguments to construct final result.
|
| Upon computation each key in the collection will have an in
| memory result, the postcompute function combines each key's
| result into a final in memory representation. For example,
| dask.array.Array concatenates the arrays at each chunk into a
| final in-memory array.
|
| Returns
| -------
| PostComputeCallable
| Callable that receives the sequence of the results of each
| final key along with optional arguments. An example signature
| would be ``finalize(results: Sequence[Any], *args)``.
| tuple[Any, ...]
| Optional arguments passed to the function following the
| key results (the `*args` part of the
| ``PostComputeCallable``. If no additional arguments are to
| be passed then this must be an empty tuple.
|
| __dask_postpersist__(self) -> 'tuple[PostPersistCallable, tuple]'
| Rebuilder function and optional arguments to construct a persisted collection.
|
| See also the documentation for :py:class:`dask.typing.PostPersistCallable`.
|
| Returns
| -------
| PostPersistCallable
| Callable that rebuilds the collection. The signature
| should be
| ``rebuild(dsk: Mapping, *args: Any, rename: Mapping[str, str] | None)``
| (as defined by the
| :py:class:`~dask.typing.PostPersistCallable` protocol).
| The callable should return an equivalent Dask collection
| with the same keys as `self`, but with results that are
| computed through a different graph. In the case of
| :py:func:`dask.persist`, the new graph will have just the
| output keys and the values already computed.
| tuple[Any, ...]
| Optional arguments passed to the rebuild callable. If no
| additional arguments are to be passed then this must be an
| empty tuple.
|
| __dask_tokenize__(self) -> 'Hashable'
| Value that must fully represent the object.
|
| __init__ = _no_init_or_replace_init(self, *args, **kwargs) from typing
|
| compute(self, **kwargs: 'Any') -> 'Any'
| Compute this dask collection.
|
| This turns a lazy Dask collection into its in-memory
| equivalent. For example a Dask array turns into a NumPy array
| and a Dask dataframe turns into a Pandas dataframe. The entire
| dataset must fit into memory before calling this operation.
|
| Parameters
| ----------
| scheduler : string, optional
| Which scheduler to use like "threads", "synchronous" or
| "processes". If not provided, the default is to check the
| global settings first, and then fall back to the
| collection defaults.
| optimize_graph : bool, optional
| If True [default], the graph is optimized before
| computation. Otherwise the graph is run as is. This can be
| useful for debugging.
| kwargs :
| Extra keywords to forward to the scheduler function.
|
| Returns
| -------
| The collection's computed result.
|
| See Also
| --------
| dask.compute
|
| persist(self: 'CollType', **kwargs: 'Any') -> 'CollType'
| Persist this dask collection into memory
|
| This turns a lazy Dask collection into a Dask collection with
| the same metadata, but now with the results fully computed or
| actively computing in the background.
|
| The action of function differs significantly depending on the
| active task scheduler. If the task scheduler supports
| asynchronous computing, such as is the case of the
| dask.distributed scheduler, then persist will return
| *immediately* and the return value's task graph will contain
| Dask Future objects. However if the task scheduler only
| supports blocking computation then the call to persist will
| *block* and the return value's task graph will contain
| concrete Python results.
|
| This function is particularly useful when using distributed
| systems, because the results will be kept in distributed
| memory, rather than returned to the local process as with
| compute.
|
| Parameters
| ----------
| scheduler : string, optional
| Which scheduler to use like "threads", "synchronous" or
| "processes". If not provided, the default is to check the
| global settings first, and then fall back to the
| collection defaults.
| optimize_graph : bool, optional
| If True [default], the graph is optimized before
| computation. Otherwise the graph is run as is. This can be
| useful for debugging.
| **kwargs
| Extra keywords to forward to the scheduler function.
|
| Returns
| -------
| New dask collections backed by in-memory data
|
| See Also
| --------
| dask.persist
|
| visualize(self, filename: 'str' = 'mydask', format: 'str | None' = None, optimize_graph: 'bool' = False, **kwargs: 'Any') -> 'DisplayObject | None'
| Render the computation of this object's task graph using graphviz.
|
| Requires ``graphviz`` to be installed.
|
| Parameters
| ----------
| filename : str or None, optional
| The name of the file to write to disk. If the provided
| `filename` doesn't include an extension, '.png' will be
| used by default. If `filename` is None, no file will be
| written, and we communicate with dot using only pipes.
| format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
| Format in which to write output file. Default is 'png'.
| optimize_graph : bool, optional
| If True, the graph is optimized before rendering.
| Otherwise, the graph is displayed as is. Default is False.
| color: {None, 'order'}, optional
| Options to color nodes. Provide ``cmap=`` keyword for
| additional colormap
| **kwargs
| Additional keyword arguments to forward to ``to_graphviz``.
|
| Examples
| --------
| >>> x.visualize(filename='dask.pdf') # doctest: +SKIP
| >>> x.visualize(filename='dask.pdf', color='order') # doctest: +SKIP
|
| Returns
| -------
| result : IPython.display.Image, IPython.display.SVG, or None
| See dask.dot.dot_graph for more information.
|
| See Also
| --------
| dask.visualize
| dask.dot.dot_graph
|
| Notes
| -----
| For more information on optimization see here:
|
| https://docs.dask.org/en/latest/optimize.html
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __subclasshook__ = _proto_hook(other) from typing
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'__dask_graph__', '__dask_keys__', '_...
|
| __annotations__ = {'__dask_optimize__': 'Any', '__dask_scheduler__': '...
|
| __non_callable_proto_members__ = {'__dask_optimize__', '__dask_schedul...
|
| __parameters__ = ()
|
| __protocol_attrs__ = {'__dask_graph__', '__dask_keys__', '__dask_optim...
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Protocol:
|
| __init_subclass__(*args, **kwargs)
| Function to initialize subclasses.
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Generic:
|
| __class_getitem__(...)
| Parameterizes a generic class.
|
| At least, parameterizing a generic class is the *main* thing this
| method does. For example, for some generic class `Foo`, this is called
| when we do `Foo[int]` - there, with `cls=Foo` and `params=int`.
|
| However, note that this method is also called when defining generic
| classes in the first place with `class Foo[T]: ...`.
class HLGDaskCollection(DaskCollection, typing.Protocol)
| HLGDaskCollection(*args, **kwargs)
|
| Protocol defining a Dask collection that uses HighLevelGraphs.
|
| This protocol is nearly identical to
| :py:class:`~dask.typing.DaskCollection`, with the addition of the
| ``__dask_layers__`` method (required for collections backed by
| high level graphs).
|
| Method resolution order:
| HLGDaskCollection
| DaskCollection
| typing.Protocol
| typing.Generic
| builtins.object
|
| Methods defined here:
|
| __dask_layers__(self) -> 'Sequence[str]'
| Names of the HighLevelGraph layers.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __subclasshook__ = _proto_hook(other) from typing
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'__dask_graph__', '__dask_keys__', '_...
|
| __annotations__ = {}
|
| __non_callable_proto_members__ = {'__dask_optimize__', '__dask_schedul...
|
| __parameters__ = ()
|
| __protocol_attrs__ = {'__dask_graph__', '__dask_keys__', '__dask_layer...
|
| ----------------------------------------------------------------------
| Methods inherited from DaskCollection:
|
| __dask_graph__(self) -> 'Graph'
| The Dask task graph.
|
| The core Dask collections (Array, DataFrame, Bag, and Delayed)
| use a :py:class:`~dask.highlevelgraph.HighLevelGraph` to
| represent the collection task graph. It is also possible to
| represent the task graph as a low level graph using a Python
| dictionary.
|
| Returns
| -------
| Mapping
| The Dask task graph. If the instance returns a
| :py:class:`dask.highlevelgraph.HighLevelGraph` then the
| :py:func:`__dask_layers__` method must be implemented, as
| defined by the :py:class:`~dask.typing.HLGDaskCollection`
| protocol.
|
| __dask_keys__(self) -> 'NestedKeys'
| The output keys of the task graph.
|
| Note that there are additional constraints on keys for a Dask
| collection than those described in the :doc:`task graph
| specification documentation <spec>`. These additional
| constraints are described below.
|
| All keys must either be non-empty strings or tuples where the first element is a
| non-empty string, followed by zero or more arbitrary str, bytes, int, float, or
| tuples thereof. The non-empty string is commonly known as the *collection name*.
| All collections embedded in the dask package have exactly one name, but this is
| not a requirement.
|
| These are all valid outputs:
|
| - ``[]``
| - ``["x", "y"]``
| - ``[[("y", "a", 0), ("y", "a", 1)], [("y", "b", 0), ("y", "b", 1)]``
|
| Returns
| -------
| list
| A possibly nested list of keys that represent the outputs
| of the graph. After computation, the results will be
| returned in the same layout, with the keys replaced with
| their corresponding outputs.
|
| __dask_postcompute__(self) -> 'tuple[PostComputeCallable, tuple]'
| Finalizer function and optional arguments to construct final result.
|
| Upon computation each key in the collection will have an in
| memory result, the postcompute function combines each key's
| result into a final in memory representation. For example,
| dask.array.Array concatenates the arrays at each chunk into a
| final in-memory array.
|
| Returns
| -------
| PostComputeCallable
| Callable that receives the sequence of the results of each
| final key along with optional arguments. An example signature
| would be ``finalize(results: Sequence[Any], *args)``.
| tuple[Any, ...]
| Optional arguments passed to the function following the
| key results (the `*args` part of the
| ``PostComputeCallable``. If no additional arguments are to
| be passed then this must be an empty tuple.
|
| __dask_postpersist__(self) -> 'tuple[PostPersistCallable, tuple]'
| Rebuilder function and optional arguments to construct a persisted collection.
|
| See also the documentation for :py:class:`dask.typing.PostPersistCallable`.
|
| Returns
| -------
| PostPersistCallable
| Callable that rebuilds the collection. The signature
| should be
| ``rebuild(dsk: Mapping, *args: Any, rename: Mapping[str, str] | None)``
| (as defined by the
| :py:class:`~dask.typing.PostPersistCallable` protocol).
| The callable should return an equivalent Dask collection
| with the same keys as `self`, but with results that are
| computed through a different graph. In the case of
| :py:func:`dask.persist`, the new graph will have just the
| output keys and the values already computed.
| tuple[Any, ...]
| Optional arguments passed to the rebuild callable. If no
| additional arguments are to be passed then this must be an
| empty tuple.
|
| __dask_tokenize__(self) -> 'Hashable'
| Value that must fully represent the object.
|
| __init__ = _no_init_or_replace_init(self, *args, **kwargs) from typing
|
| compute(self, **kwargs: 'Any') -> 'Any'
| Compute this dask collection.
|
| This turns a lazy Dask collection into its in-memory
| equivalent. For example a Dask array turns into a NumPy array
| and a Dask dataframe turns into a Pandas dataframe. The entire
| dataset must fit into memory before calling this operation.
|
| Parameters
| ----------
| scheduler : string, optional
| Which scheduler to use like "threads", "synchronous" or
| "processes". If not provided, the default is to check the
| global settings first, and then fall back to the
| collection defaults.
| optimize_graph : bool, optional
| If True [default], the graph is optimized before
| computation. Otherwise the graph is run as is. This can be
| useful for debugging.
| kwargs :
| Extra keywords to forward to the scheduler function.
|
| Returns
| -------
| The collection's computed result.
|
| See Also
| --------
| dask.compute
|
| persist(self: 'CollType', **kwargs: 'Any') -> 'CollType'
| Persist this dask collection into memory
|
| This turns a lazy Dask collection into a Dask collection with
| the same metadata, but now with the results fully computed or
| actively computing in the background.
|
| The action of function differs significantly depending on the
| active task scheduler. If the task scheduler supports
| asynchronous computing, such as is the case of the
| dask.distributed scheduler, then persist will return
| *immediately* and the return value's task graph will contain
| Dask Future objects. However if the task scheduler only
| supports blocking computation then the call to persist will
| *block* and the return value's task graph will contain
| concrete Python results.
|
| This function is particularly useful when using distributed
| systems, because the results will be kept in distributed
| memory, rather than returned to the local process as with
| compute.
|
| Parameters
| ----------
| scheduler : string, optional
| Which scheduler to use like "threads", "synchronous" or
| "processes". If not provided, the default is to check the
| global settings first, and then fall back to the
| collection defaults.
| optimize_graph : bool, optional
| If True [default], the graph is optimized before
| computation. Otherwise the graph is run as is. This can be
| useful for debugging.
| **kwargs
| Extra keywords to forward to the scheduler function.
|
| Returns
| -------
| New dask collections backed by in-memory data
|
| See Also
| --------
| dask.persist
|
| visualize(self, filename: 'str' = 'mydask', format: 'str | None' = None, optimize_graph: 'bool' = False, **kwargs: 'Any') -> 'DisplayObject | None'
| Render the computation of this object's task graph using graphviz.
|
| Requires ``graphviz`` to be installed.
|
| Parameters
| ----------
| filename : str or None, optional
| The name of the file to write to disk. If the provided
| `filename` doesn't include an extension, '.png' will be
| used by default. If `filename` is None, no file will be
| written, and we communicate with dot using only pipes.
| format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
| Format in which to write output file. Default is 'png'.
| optimize_graph : bool, optional
| If True, the graph is optimized before rendering.
| Otherwise, the graph is displayed as is. Default is False.
| color: {None, 'order'}, optional
| Options to color nodes. Provide ``cmap=`` keyword for
| additional colormap
| **kwargs
| Additional keyword arguments to forward to ``to_graphviz``.
|
| Examples
| --------
| >>> x.visualize(filename='dask.pdf') # doctest: +SKIP
| >>> x.visualize(filename='dask.pdf', color='order') # doctest: +SKIP
|
| Returns
| -------
| result : IPython.display.Image, IPython.display.SVG, or None
| See dask.dot.dot_graph for more information.
|
| See Also
| --------
| dask.visualize
| dask.dot.dot_graph
|
| Notes
| -----
| For more information on optimization see here:
|
| https://docs.dask.org/en/latest/optimize.html
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DaskCollection:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Protocol:
|
| __init_subclass__(*args, **kwargs)
| Function to initialize subclasses.
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Generic:
|
| __class_getitem__(...)
| Parameterizes a generic class.
|
| At least, parameterizing a generic class is the *main* thing this
| method does. For example, for some generic class `Foo`, this is called
| when we do `Foo[int]` - there, with `cls=Foo` and `params=int`.
|
| However, note that this method is also called when defining generic
| classes in the first place with `class Foo[T]: ...`.
class PostPersistCallable(typing.Protocol)
| PostPersistCallable(*args, **kwargs)
|
| Protocol defining the signature of a ``__dask_postpersist__`` callable.
|
| Method resolution order:
| PostPersistCallable
| typing.Protocol
| typing.Generic
| builtins.object
|
| Methods defined here:
|
| __call__(self, dsk: 'Graph', *args: 'Any', rename: 'Mapping[str, str] | None' = None) -> 'CollType_co'
| Method called to rebuild a persisted collection.
|
| Parameters
| ----------
| dsk: Mapping
| A mapping which contains at least the output keys returned
| by __dask_keys__().
| *args : Any
| Additional optional arguments If no extra arguments are
| necessary, it must be an empty tuple.
| rename : Mapping[str, str], optional
| If defined, it indicates that output keys may be changing
| too; e.g. if the previous output of :meth:`__dask_keys__`
| was ``[("a", 0), ("a", 1)]``, after calling
| ``rebuild(dsk, *extra_args, rename={"a": "b"})``
| it must become ``[("b", 0), ("b", 1)]``.
| The ``rename`` mapping may not contain the collection
| name(s); in such case the associated keys do not change.
| It may contain replacements for unexpected names, which
| must be ignored.
|
| Returns
| -------
| Collection
| An equivalent Dask collection with the same keys as
| computed through a different graph.
|
| __init__ = _no_init_or_replace_init(self, *args, **kwargs) from typing
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __subclasshook__ = _proto_hook(other) from typing
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| __orig_bases__ = (typing.Protocol[+CollType_co],)
|
| __parameters__ = (+CollType_co,)
|
| __protocol_attrs__ = {'__call__'}
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Protocol:
|
| __init_subclass__(*args, **kwargs)
| Function to initialize subclasses.
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Generic:
|
| __class_getitem__(...)
| Parameterizes a generic class.
|
| At least, parameterizing a generic class is the *main* thing this
| method does. For example, for some generic class `Foo`, this is called
| when we do `Foo[int]` - there, with `cls=Foo` and `params=int`.
|
| However, note that this method is also called when defining generic
| classes in the first place with `class Foo[T]: ...`.
class SchedulerGetCallable(typing.Protocol)
| SchedulerGetCallable(*args, **kwargs)
|
| Protocol defining the signature of a ``__dask_scheduler__`` callable.
|
| Method resolution order:
| SchedulerGetCallable
| typing.Protocol
| typing.Generic
| builtins.object
|
| Methods defined here:
|
| __call__(self, dsk: 'Graph', keys: 'Sequence[Key] | Key', **kwargs: 'Any') -> 'Any'
| Method called as the default scheduler for a collection.
|
| Parameters
| ----------
| dsk :
| The task graph.
| keys :
| Key(s) corresponding to the desired data.
| **kwargs :
| Additional arguments.
|
| Returns
| -------
| Any
| Result(s) associated with `keys`
|
| __init__ = _no_init_or_replace_init(self, *args, **kwargs) from typing
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __subclasshook__ = _proto_hook(other) from typing
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| __parameters__ = ()
|
| __protocol_attrs__ = {'__call__'}
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Protocol:
|
| __init_subclass__(*args, **kwargs)
| Function to initialize subclasses.
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Generic:
|
| __class_getitem__(...)
| Parameterizes a generic class.
|
| At least, parameterizing a generic class is the *main* thing this
| method does. For example, for some generic class `Foo`, this is called
| when we do `Foo[int]` - there, with `cls=Foo` and `params=int`.
|
| However, note that this method is also called when defining generic
| classes in the first place with `class Foo[T]: ...`.
DATA
CollType = ~CollType
CollType_co = +CollType_co
Graph = collections.abc.Mapping[str | int | float | tuple['Key', ...],...
Key = str | int | float | tuple['Key', ...]
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
NestedKeys = list[typing.Union[str, int, float, tuple['Key', ...], For...
NoDefault = typing.Literal[<no_default>]
TYPE_CHECKING = False
TypeAlias = typing.TypeAlias
Special form for marking type aliases.
Use TypeAlias to indicate that an assignment should
be recognized as a proper type alias definition by type
checkers.
For example::
Predicate: TypeAlias = Callable[..., bool]
It's invalid when used anywhere except as in the example above.
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
__annotations__ = {'Graph': 'TypeAlias', 'Key': 'TypeAlias', 'NestedKe...
no_default = <no_default>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\typing.py
dask.utils
Python Library Documentation: module dask.utils in dask
NAME
dask.utils
CLASSES
builtins.object
Dispatch
IndexCallable
MethodCache
OperatorMethodMixin
SerializableLock
itemgetter
methodcaller
shorten_traceback
contextlib.ContextDecorator(builtins.object)
disable_gc
functools.cached_property(builtins.object)
cached_property
class Dispatch(builtins.object)
| Dispatch(name=None)
|
| Methods defined here:
|
| __call__(self, arg, *args, **kwargs)
| Call the corresponding method based on type of argument.
|
| __init__(self, name=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| dispatch(self, cls)
| Return the function implementation for the given ``cls``
|
| register(self, type, func=None)
| Register dispatch of `func` on arguments of type `type`
|
| register_lazy(self, toplevel, func=None)
| Register a registration function which will be called if the
| *toplevel* module (e.g. 'pandas') is ever loaded.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class IndexCallable(builtins.object)
| IndexCallable(fn)
|
| Provide getitem syntax for functions
|
| >>> def inc(x):
| ... return x + 1
|
| >>> I = IndexCallable(inc)
| >>> I[3]
| 4
|
| Methods defined here:
|
| __getitem__(self, key)
|
| __init__(self, fn)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| fn
class MethodCache(builtins.object)
| Attribute access on this object returns a methodcaller for that
| attribute.
|
| Examples
| --------
| >>> a = [1, 3, 3]
| >>> M.count(a, 3) == a.count(3)
| True
|
| Methods defined here:
|
| __dir__(self)
| Default dir() implementation.
|
| __getattr__(self, item)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class OperatorMethodMixin(builtins.object)
| A mixin for dynamically implementing operators
class SerializableLock(builtins.object)
| SerializableLock(token: 'Hashable | None' = None)
|
| A Serializable per-process Lock
|
| This wraps a normal ``threading.Lock`` object and satisfies the same
| interface. However, this lock can also be serialized and sent to different
| processes. It will not block concurrent operations between processes (for
| this you should look at ``multiprocessing.Lock`` or ``locket.lock_file``
| but will consistently deserialize into the same lock.
|
| So if we make a lock in one process::
|
| lock = SerializableLock()
|
| And then send it over to another process multiple times::
|
| bytes = pickle.dumps(lock)
| a = pickle.loads(bytes)
| b = pickle.loads(bytes)
|
| Then the deserialized objects will operate as though they were the same
| lock, and collide as appropriate.
|
| This is useful for consistently protecting resources on a per-process
| level.
|
| The creation of locks is itself not threadsafe.
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, *args)
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, token: 'Hashable | None' = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__ = __str__(self)
|
| __setstate__(self, token)
|
| __str__(self)
| Return str(self).
|
| acquire(self, *args, **kwargs)
|
| locked(self)
|
| release(self, *args, **kwargs)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_locks': 'ClassVar[WeakValueDictionary[Hashable, L...
class cached_property(functools.cached_property)
| cached_property(func)
|
| Read only version of functools.cached_property.
|
| Method resolution order:
| cached_property
| functools.cached_property
| builtins.object
|
| Methods defined here:
|
| __set__(self, instance, val)
| Raise an error when attempting to set a cached property.
|
| ----------------------------------------------------------------------
| Methods inherited from functools.cached_property:
|
| __get__(self, instance, owner=None)
|
| __init__(self, func)
| Initialize self. See help(type(self)) for accurate signature.
|
| __set_name__(self, owner, name)
|
| ----------------------------------------------------------------------
| Class methods inherited from functools.cached_property:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from functools.cached_property:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class disable_gc(contextlib.ContextDecorator)
| disable_gc(collect=False)
|
| Context manager to disable garbage collection.
|
| Method resolution order:
| disable_gc
| contextlib.ContextDecorator
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_value, traceback)
|
| __init__(self, collect=False)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from contextlib.ContextDecorator:
|
| __call__(self, func)
| Call self as a function.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from contextlib.ContextDecorator:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class itemgetter(builtins.object)
| itemgetter(index)
|
| Variant of operator.itemgetter that supports equality tests
|
| Methods defined here:
|
| __call__(self, x)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, index)
| Initialize self. See help(type(self)) for accurate signature.
|
| __reduce__(self)
| Helper for pickle.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| index
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __hash__ = None
class methodcaller(builtins.object)
| methodcaller(method: 'str')
|
| Return a callable object that calls the given method on its operand.
|
| Unlike the builtin `operator.methodcaller`, instances of this class are
| cached and arguments are passed at call time instead of build time.
|
| Methods defined here:
|
| __call__(self, _methodcaller__obj, *args, **kwargs)
| Call self as a function.
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__ = __str__(self)
|
| __str__(self)
| Return str(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(cls, method: 'str')
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| func
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| method
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'method': 'str'}
class shorten_traceback(builtins.object)
| Context manager that removes irrelevant stack elements from traceback.
|
| * omits frames from modules that match `admin.traceback.shorten`
| * always keeps the first and last frame.
|
| Methods defined here:
|
| __enter__(self) -> 'None'
|
| __exit__(self, exc_type: 'type[BaseException] | None', exc_val: 'BaseException | None', exc_tb: 'types.TracebackType | None') -> 'None'
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| shorten(exc_tb: 'types.TracebackType') -> 'types.TracebackType'
FUNCTIONS
Lock = allocate_lock(...)
allocate_lock() -> lock object
(allocate() is an obsolete synonym)
Create a new lock object. See help(type(threading.Lock())) for
information about locks.
add(a, b, /)
Same as a + b.
apply(func, args, kwargs=None)
Apply a function given its positional and keyword arguments.
Equivalent to ``func(*args, **kwargs)``
Most Dask users will never need to use the ``apply`` function.
It is typically only used by people who need to inject
keyword argument values into a low level Dask task graph.
Parameters
----------
func : callable
The function you want to apply.
args : tuple
A tuple containing all the positional arguments needed for ``func``
(eg: ``(arg_1, arg_2, arg_3)``)
kwargs : dict, optional
A dictionary mapping the keyword arguments
(eg: ``{"kwarg_1": value, "kwarg_2": value}``
Examples
--------
>>> from dask.utils import apply
>>> def add(number, second_number=5):
... return number + second_number
...
>>> apply(add, (10,), {"second_number": 2}) # equivalent to add(*args, **kwargs)
12
>>> task = apply(add, (10,), {"second_number": 2})
>>> dsk = {'task-name': task} # adds the task to a low level Dask task graph
asciitable(columns, rows)
Formats an ascii table for given columns and rows.
Parameters
----------
columns : list
The column names
rows : list of tuples
The rows in the table. Each tuple must be the same length as
``columns``.
cached_cumsum(seq, initial_zero=False)
Compute :meth:`toolz.accumulate` with caching.
Caching is by the identify of `seq` rather than the value. It is thus
important that `seq` is a tuple of immutable objects, and this function
is intended for use where `seq` is a value that will persist (generally
block sizes).
Parameters
----------
seq : tuple
Values to cumulatively sum.
initial_zero : bool, optional
If true, the return value is prefixed with a zero.
Returns
-------
tuple
cached_max(seq)
Compute max with caching.
Caching is by the identity of `seq` rather than the value. It is thus
important that `seq` is a tuple of immutable objects, and this function
is intended for use where `seq` is a value that will persist (generally
block sizes).
Parameters
----------
seq : tuple
Values to reduce
Returns
-------
tuple
changed_cwd(new_cwd)
concrete(seq)
Make nested iterators concrete lists
>>> data = [[1, 2], [3, 4]]
>>> seq = iter(map(iter, data))
>>> concrete(seq)
[[1, 2], [3, 4]]
deepmap(func, *seqs)
Apply function inside nested lists
>>> inc = lambda x: x + 1
>>> deepmap(inc, [[1, 2], [3, 4]])
[[2, 3], [4, 5]]
>>> add = lambda x, y: x + y
>>> deepmap(add, [[1, 2], [3, 4]], [[10, 20], [30, 40]])
[[11, 22], [33, 44]]
derived_from(original_klass, version=None, ua_args=None, skipblocks=0, inconsistencies=None)
Decorator to attach original class's docstring to the wrapped method.
The output structure will be: top line of docstring, disclaimer about this
being auto-derived, any extra text associated with the method being patched,
the body of the docstring and finally, the list of keywords that exist in
the original method but not in the dask version.
Parameters
----------
original_klass: type
Original class which the method is derived from
version : str
Original package version which supports the wrapped method
ua_args : list
List of keywords which Dask doesn't support. Keywords existing in
original but not in Dask will automatically be added.
skipblocks : int
How many text blocks (paragraphs) to skip from the start of the
docstring. Useful for cases where the target has extra front-matter.
inconsistencies: list
List of known inconsistencies with method whose docstrings are being
copied.
digit(n, k, base)
>>> digit(1234, 0, 10)
4
>>> digit(1234, 1, 10)
3
>>> digit(1234, 2, 10)
2
>>> digit(1234, 3, 10)
1
ensure_bytes(s) -> 'bytes'
Attempt to turn `s` into bytes.
Parameters
----------
s : Any
The object to be converted. Will correctly handled
* str
* bytes
* objects implementing the buffer protocol (memoryview, ndarray, etc.)
Returns
-------
b : bytes
Raises
------
TypeError
When `s` cannot be converted
Examples
--------
>>> ensure_bytes('123')
b'123'
>>> ensure_bytes(b'123')
b'123'
>>> ensure_bytes(bytearray(b'123'))
b'123'
ensure_dict(d: 'Mapping[K, V]', *, copy: 'bool' = False) -> 'dict[K, V]'
Convert a generic Mapping into a dict.
Optimize use case of :class:`~dask.highlevelgraph.HighLevelGraph`.
Parameters
----------
d : Mapping
copy : bool
If True, guarantee that the return value is always a shallow copy of d;
otherwise it may be the input itself.
ensure_not_exists(filename) -> 'None'
Ensure that a file does not exist.
ensure_set(s: 'Set[T]', *, copy: 'bool' = False) -> 'set[T]'
Convert a generic Set into a set.
Parameters
----------
s : Set
copy : bool
If True, guarantee that the return value is always a shallow copy of s;
otherwise it may be the input itself.
ensure_unicode(s) -> 'str'
Turn string or bytes to string
>>> ensure_unicode('123')
'123'
>>> ensure_unicode(b'123')
'123'
extra_titles(doc)
filetext(text, extension='', open=<built-in function open>, mode='w')
filetexts(d, open=<built-in function open>, mode='t', use_tmpdir=True)
Dumps a number of textfiles to disk
Parameters
----------
d : dict
a mapping from filename to text like {'a.csv': '1,1
2,2'}
Since this is meant for use in tests, this context manager will
automatically switch to a temporary current directory, to avoid
race conditions when running tests in parallel.
format_bytes(n: 'int') -> 'str'
Format bytes as text
>>> from dask.utils import format_bytes
>>> format_bytes(1)
'1 B'
>>> format_bytes(1234)
'1.21 kiB'
>>> format_bytes(12345678)
'11.77 MiB'
>>> format_bytes(1234567890)
'1.15 GiB'
>>> format_bytes(1234567890000)
'1.12 TiB'
>>> format_bytes(1234567890000000)
'1.10 PiB'
For all values < 2**60, the output is always <= 10 characters.
format_time(n: 'float') -> 'str'
format integers as time
>>> from dask.utils import format_time
>>> format_time(1)
'1.00 s'
>>> format_time(0.001234)
'1.23 ms'
>>> format_time(0.00012345)
'123.45 us'
>>> format_time(123.456)
'123.46 s'
>>> format_time(1234.567)
'20m 34s'
>>> format_time(12345.67)
'3hr 25m'
>>> format_time(123456.78)
'34hr 17m'
>>> format_time(1234567.89)
'14d 6hr'
format_time_ago(n: 'datetime') -> 'str'
Calculate a '3 hours ago' type string from a Python datetime.
Examples
--------
>>> from datetime import datetime, timedelta
>>> now = datetime.now()
>>> format_time_ago(now)
'Just now'
>>> past = datetime.now() - timedelta(minutes=1)
>>> format_time_ago(past)
'1 minute ago'
>>> past = datetime.now() - timedelta(minutes=2)
>>> format_time_ago(past)
'2 minutes ago'
>>> past = datetime.now() - timedelta(hours=1)
>>> format_time_ago(past)
'1 hour ago'
>>> past = datetime.now() - timedelta(hours=6)
>>> format_time_ago(past)
'6 hours ago'
>>> past = datetime.now() - timedelta(days=1)
>>> format_time_ago(past)
'1 day ago'
>>> past = datetime.now() - timedelta(days=5)
>>> format_time_ago(past)
'5 days ago'
>>> past = datetime.now() - timedelta(days=8)
>>> format_time_ago(past)
'1 week ago'
>>> past = datetime.now() - timedelta(days=16)
>>> format_time_ago(past)
'2 weeks ago'
>>> past = datetime.now() - timedelta(days=190)
>>> format_time_ago(past)
'6 months ago'
>>> past = datetime.now() - timedelta(days=800)
>>> format_time_ago(past)
'2 years ago'
funcname(func) -> 'str'
Get the name of a function.
get_default_shuffle_method() -> 'str'
get_meta_library(like)
get_named_args(func) -> 'list[str]'
Get all non ``*args/**kwargs`` arguments for a function
get_scheduler_lock(collection=None, scheduler=None)
Get an instance of the appropriate lock for a certain situation based on
scheduler used.
getargspec(func)
Version of inspect.getargspec that works with partial and warps.
has_keyword(func, keyword)
homogeneous_deepmap(func, seq)
ignore_warning(doc, cls, name, extra='', skipblocks=0, inconsistencies=None)
Expand docstring by adding disclaimer and extra text
import_required(mod_name, error_msg)
Attempt to import a required dependency.
Raises a RuntimeError if the requested module is not available.
insert(tup, loc, val)
>>> insert(('a', 'b', 'c'), 0, 'x')
('x', 'b', 'c')
is_arraylike(x) -> 'bool'
Is this object a numpy array or something similar?
This function tests specifically for an object that already has
array attributes (e.g. np.ndarray, dask.array.Array, cupy.ndarray,
sparse.COO), **NOT** for something that can be coerced into an
array object (e.g. Python lists and tuples). It is meant for dask
developers and developers of downstream libraries.
Note that this function does not correspond with NumPy's
definition of array_like, which includes any object that can be
coerced into an array (see definition in the NumPy glossary):
https://numpy.org/doc/stable/glossary.html
Examples
--------
>>> import numpy as np
>>> is_arraylike(np.ones(5))
True
>>> is_arraylike(np.ones(()))
True
>>> is_arraylike(5)
False
>>> is_arraylike('cat')
False
is_cupy_type(x) -> 'bool'
is_dataframe_like(df) -> 'bool'
Looks like a Pandas DataFrame
is_empty(obj)
Duck-typed check for “emptiness” of an object.
Works for standard sequences (lists, tuples, etc.), NumPy arrays,
and sparse-like objects (e.g., SciPy sparse arrays).
The function checks:
1. If the object supports len(), returns True if len(obj) == 0.
2. If the object has a `.nnz` attribute (number of non-zero elements),
returns True if `.nnz == 0`.
3. If the object has a `.shape` attribute, returns True if any
dimension is zero.
4. Otherwise, returns False (assumes non-empty).
Parameters
----------
obj : any
The object to check for emptiness.
Returns
-------
bool
True if the object is considered empty, False otherwise.
is_index_like(s) -> 'bool'
Looks like a Pandas Index
is_integer(i) -> 'bool'
>>> is_integer(6)
True
>>> is_integer(42.0)
True
>>> is_integer('abc')
False
is_namedtuple_instance(obj: 'Any') -> 'bool'
Returns True if obj is an instance of a namedtuple.
Note: This function checks for the existence of the methods and
attributes that make up the namedtuple API, so it will return True
IFF obj's type implements that API.
is_series_like(s) -> 'bool'
Looks like a Pandas Series
iter_chunks(sizes, max_size)
Split sizes into chunks of total max_size each
Parameters
----------
sizes : iterable of numbers
The sizes to be chunked
max_size : number
Maximum total size per chunk.
It must be greater or equal than each size in sizes
key_split(s)
>>> key_split('x')
'x'
>>> key_split('x-1')
'x'
>>> key_split('x-1-2-3')
'x'
>>> key_split(('x-2', 1))
'x'
>>> key_split("('x-2', 1)")
'x'
>>> key_split("('x', 1)")
'x'
>>> key_split('hello-world-1')
'hello-world'
>>> key_split(b'hello-world-1')
'hello-world'
>>> key_split('ae05086432ca935f6eba409a8ecd4896')
'data'
>>> key_split('<module.submodule.myclass object at 0xdaf372')
'myclass'
>>> key_split(None)
'Other'
>>> key_split('x-abcdefab') # ignores hex
'x'
>>> key_split('_(x)') # strips unpleasant characters
'x'
maybe_pluralize(count, noun, plural_form=None)
Pluralize a count-noun string pattern when necessary
memory_repr(num)
natural_sort_key(s: 'str') -> 'list[str | int]'
Sorting `key` function for performing a natural sort on a collection of
strings
See https://en.wikipedia.org/wiki/Natural_sort_order
Parameters
----------
s : str
A string that is an element of the collection being sorted
Returns
-------
tuple[str or int]
Tuple of the parts of the input string where each part is either a
string or an integer
Examples
--------
>>> a = ['f0', 'f1', 'f2', 'f8', 'f9', 'f10', 'f11', 'f19', 'f20', 'f21']
>>> sorted(a)
['f0', 'f1', 'f10', 'f11', 'f19', 'f2', 'f20', 'f21', 'f8', 'f9']
>>> sorted(a, key=natural_sort_key)
['f0', 'f1', 'f2', 'f8', 'f9', 'f10', 'f11', 'f19', 'f20', 'f21']
ndeepmap(n, func, seq)
Call a function on every element within a nested container
>>> def inc(x):
... return x + 1
>>> L = [[1, 2], [3, 4, 5]]
>>> ndeepmap(2, inc, L)
[[2, 3], [4, 5, 6]]
ndimlist(seq)
parse_bytes(s: 'float | str') -> 'int'
Parse byte string to numbers
>>> from dask.utils import parse_bytes
>>> parse_bytes('100')
100
>>> parse_bytes('100 MB')
100000000
>>> parse_bytes('100M')
100000000
>>> parse_bytes('5kB')
5000
>>> parse_bytes('5.4 kB')
5400
>>> parse_bytes('1kiB')
1024
>>> parse_bytes('1e6')
1000000
>>> parse_bytes('1e6 kB')
1000000000
>>> parse_bytes('MB')
1000000
>>> parse_bytes(123)
123
>>> parse_bytes('5 foos')
Traceback (most recent call last):
...
ValueError: Could not interpret 'foos' as a byte unit
parse_timedelta(s, default='seconds')
Parse timedelta string to number of seconds
Parameters
----------
s : str, float, timedelta, or None
default: str or False, optional
Unit of measure if s does not specify one. Defaults to seconds.
Set to False to require s to explicitly specify its own unit.
Examples
--------
>>> from datetime import timedelta
>>> from dask.utils import parse_timedelta
>>> parse_timedelta('3s')
3
>>> parse_timedelta('3.5 seconds')
3.5
>>> parse_timedelta('300ms')
0.3
>>> parse_timedelta(timedelta(seconds=3)) # also supports timedeltas
3
partial_by_order(*args, **kwargs)
>>> from operator import add
>>> partial_by_order(5, function=add, other=[(1, 10)])
15
pseudorandom(n: 'int', p, random_state=None)
Pseudorandom array of integer indexes
>>> pseudorandom(5, [0.5, 0.5], random_state=123)
array([1, 0, 0, 1, 1], dtype=int8)
>>> pseudorandom(10, [0.5, 0.2, 0.2, 0.1], random_state=5)
array([0, 2, 0, 3, 0, 1, 2, 1, 0, 0], dtype=int8)
put_lines(buf, lines)
random_state_data(n: 'int', random_state=None) -> 'list'
Return a list of arrays that can initialize
``np.random.RandomState``.
Parameters
----------
n : int
Number of arrays to return.
random_state : int or np.random.RandomState, optional
If an int, is used to seed a new ``RandomState``.
show_versions() -> 'None'
Provide version information for bug reports.
skip_doctest(doc)
stringify(obj, exclusive: 'Iterable | None' = None)
Convert an object to a string
If ``exclusive`` is specified, search through `obj` and convert
values that are in ``exclusive``.
Note that when searching through dictionaries, only values are
converted, not the keys.
Parameters
----------
obj : Any
Object (or values within) to convert to string
exclusive: Iterable, optional
Set of values to search for when converting values to strings
Returns
-------
result : type(obj)
Stringified copy of ``obj`` or ``obj`` itself if it is already a
string or bytes.
Examples
--------
>>> stringify(b'x')
b'x'
>>> stringify('x')
'x'
>>> stringify({('a',0):('a',0), ('a',1): ('a',1)})
"{('a', 0): ('a', 0), ('a', 1): ('a', 1)}"
>>> stringify({('a',0):('a',0), ('a',1): ('a',1)}, exclusive={('a',0)})
{('a', 0): "('a', 0)", ('a', 1): ('a', 1)}
takes_multiple_arguments(func, varargs=True)
Does this function take multiple arguments?
>>> def f(x, y): pass
>>> takes_multiple_arguments(f)
True
>>> def f(x): pass
>>> takes_multiple_arguments(f)
False
>>> def f(x, y=None): pass
>>> takes_multiple_arguments(f)
False
>>> def f(*args): pass
>>> takes_multiple_arguments(f)
True
>>> class Thing:
... def __init__(self, a): pass
>>> takes_multiple_arguments(Thing)
False
tmp_cwd(dir=None)
tmpdir(dir=None)
Function to create and return a unique temporary directory.
Parameters
----------
dir : str
If ``dir`` is not None, the directory will be created in that directory; otherwise,
Python's default temporary directory is used.
Returns
-------
out : str
Path to the temporary directory
Notes
-----
This context manager is particularly useful on Windows for opening temporary directories multiple times.
tmpfile(extension='', dir=None)
Function to create and return a unique temporary file with the given extension, if provided.
Parameters
----------
extension : str
The extension of the temporary file to be created
dir : str
If ``dir`` is not None, the file will be created in that directory; otherwise,
Python's default temporary directory is used.
Returns
-------
out : str
Path to the temporary file
See Also
--------
NamedTemporaryFile : Built-in alternative for creating temporary files
tmp_path : pytest fixture for creating a temporary directory unique to the test invocation
Notes
-----
This context manager is particularly useful on Windows for opening temporary files multiple times.
typename(typ: 'Any', short: 'bool' = False) -> 'str'
Return the name of a type
Examples
--------
>>> typename(int)
'int'
>>> from dask.core import literal
>>> typename(literal)
'dask.core.literal'
>>> typename(literal, short=True)
'dask.literal'
unsupported_arguments(doc, args)
Mark unsupported arguments with a disclaimer
unzip(ls, nout)
Unzip a list of lists into ``nout`` outputs.
DATA
ClassVar = typing.ClassVar
Special type construct to mark class variables.
An annotation wrapped in ClassVar indicates that a given
attribute is intended to be used as a class variable and
should not be set on instances of that class.
Usage::
class Starship:
stats: ClassVar[dict[str, int]] = {} # class variable
damage: int = 10 # instance variable
ClassVar accepts only types and cannot be further subscribed.
Note that ClassVar is not a class itself, and should not
be used with isinstance() or issubclass().
ENOENT = 2
F = ~F
FuncType = collections.abc.Callable[..., typing.Any]
K = ~K
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
M = <dask.utils.MethodCache object>
MULTI_ARITY_BUILTINS = {<built-in function delattr>, <built-in functio...
ONE_ARITY_BUILTINS = {<class 'int'>, <built-in function id>, <class 's...
T = ~T
V = ~V
__annotations__ = {'_method_cache': 'dict[str, methodcaller]'}
byte_sizes = {'': 1, 'b': 1, 'g': 1000000000, 'gb': 1000000000, 'gi': ...
hex_pattern = re.compile('[a-f]+')
no_default = <no_default>
system_encoding = 'utf-8'
tds2 = {'day': 86400, 'days': 86400, 'hour': 3600, 'hours': 3600, 'mic...
timedelta_sizes = {'D': 86400, 'DAY': 86400, 'DAYS': 86400, 'H': 3600,...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\utils.py
dask.visualize
Python Library Documentation: function visualize in module dask.base
visualize(*args, filename='mydask', traverse=True, optimize_graph=False, maxval=None, engine: "Literal['cytoscape', 'ipycytoscape', 'graphviz'] | None" = None, **kwargs)
Visualize several dask graphs simultaneously.
Requires ``graphviz`` to be installed. All options that are not the dask
graph(s) should be passed as keyword arguments.
Parameters
----------
args : object
Any number of objects. If it is a dask collection (for example, a
dask DataFrame, Array, Bag, or Delayed), its associated graph
will be included in the output of visualize. By default, python builtin
collections are also traversed to look for dask objects (for more
information see the ``traverse`` keyword). Arguments lacking an
associated graph will be ignored.
filename : str or None, optional
The name of the file to write to disk. If the provided `filename`
doesn't include an extension, '.png' will be used by default.
If `filename` is None, no file will be written, and we communicate
with dot using only pipes.
format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional
Format in which to write output file. Default is 'png'.
traverse : bool, optional
By default, dask traverses builtin python collections looking for dask
objects passed to ``visualize``. For large collections this can be
expensive. If none of the arguments contain any dask objects, set
``traverse=False`` to avoid doing this traversal.
optimize_graph : bool, optional
If True, the graph is optimized before rendering. Otherwise,
the graph is displayed as is. Default is False.
color : {None, 'order', 'ages', 'freed', 'memoryincreases', 'memorydecreases', 'memorypressure'}, optional
Options to color nodes. colormap:
- None, the default, no colors.
- 'order', colors the nodes' border based on the order they appear in the graph.
- 'ages', how long the data of a node is held.
- 'freed', the number of dependencies released after running a node.
- 'memoryincreases', how many more outputs are held after the lifetime of a node.
Large values may indicate nodes that should have run later.
- 'memorydecreases', how many fewer outputs are held after the lifetime of a node.
Large values may indicate nodes that should have run sooner.
- 'memorypressure', the number of data held when the node is run (circle), or
the data is released (rectangle).
maxval : {int, float}, optional
Maximum value for colormap to normalize form 0 to 1.0. Default is ``None``
will make it the max number of values
collapse_outputs : bool, optional
Whether to collapse output boxes, which often have empty labels.
Default is False.
verbose : bool, optional
Whether to label output and input boxes even if the data aren't chunked.
Beware: these labels can get very long. Default is False.
engine : {"graphviz", "ipycytoscape", "cytoscape"}, optional.
The visualization engine to use. If not provided, this checks the dask config
value "visualization.engine". If that is not set, it tries to import ``graphviz``
and ``ipycytoscape``, using the first one to succeed.
**kwargs
Additional keyword arguments to forward to the visualization engine.
Examples
--------
>>> x.visualize(filename='dask.pdf') # doctest: +SKIP
>>> x.visualize(filename='dask.pdf', color='order') # doctest: +SKIP
Returns
-------
result : IPython.display.Image, IPython.display.SVG, or None
See dask.dot.dot_graph for more information.
See Also
--------
dask.dot.dot_graph
Notes
-----
For more information on optimization see here:
https://docs.dask.org/en/latest/optimize.html
dask.widgets
Python Library Documentation: package dask.widgets in dask
NAME
dask.widgets
PACKAGE CONTENTS
widgets
DATA
FILTERS = {'datetime_from_timestamp': <built-in method fromtimestamp o...
TEMPLATE_PATHS = [r'C:\Users\Youse\AppData\Local\Programs\Python\Pytho...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\dask\widgets\__init__.py