polars API Reference
Full documentation for polars (1.43.1)
polars.Any
Python Library Documentation: class Any in module typing class Any(builtins.object) | Any(*args, **kwargs) | | Special type indicating an unconstrained type. | | - Any is compatible with every type. | - Any assumed to have all methods. | - All values assumed to be instances of Any. | | Note that all the above statements are true from the point of view of | static type checkers. At runtime, Any should not be used with instance | checks. | | Static methods defined here: | | __new__(cls, *args, **kwargs) | Create and return a new object. See help(type) for accurate signature. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
polars.Array
Python Library Documentation: class Array in module polars.datatypes.classes
class Array(NestedType)
| Array(inner: 'PolarsDataType | PythonDataType', shape: 'int | tuple[int, ...] | None' = None, *, width: 'int | None' = None) -> 'None'
|
| Fixed length list type.
|
| Parameters
| ----------
| inner
| The `DataType` of the values within each array.
| shape
| The shape of the arrays.
| width
| The length of the arrays.
|
| .. deprecated:: 0.20.31
| The `width` parameter for `Array` is deprecated. Use `shape` instead.
|
| Examples
| --------
| >>> s = pl.Series("a", [[1, 2], [4, 3]], dtype=pl.Array(pl.Int64, 2))
| >>> s
| shape: (2,)
| Series: 'a' [array[i64, 2]]
| [
| [1, 2]
| [4, 3]
| ]
|
| Method resolution order:
| Array
| NestedType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, inner: 'PolarsDataType | PythonDataType', shape: 'int | tuple[int, ...] | None' = None, *, width: 'int | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| width
| The size of the Array.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'inner': 'PolarsDataType', 'shape': 'tuple[int, ......
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.BaseExtension
Python Library Documentation: class BaseExtension in module polars.datatypes.classes
class BaseExtension(DataType)
| BaseExtension(name: 'str', storage: 'PolarsDataType', metadata: 'str | None' = None) -> 'None'
|
| Base class for extension data types.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| See Also
| --------
| Extension
| polars.register_extension_type
|
| Method resolution order:
| BaseExtension
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __getstate__(self) -> 'tuple[str, PolarsDataType, str | None]'
| Helper for pickle.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, name: 'str', storage: 'PolarsDataType', metadata: 'str | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, state: 'tuple[str, PolarsDataType, str | None]') -> 'None'
|
| ext_metadata(self) -> 'str | None'
| Returns the metadata for this extension type.
|
| ext_name(self) -> 'str'
| Returns the name of this extension type.
|
| ext_storage(self) -> 'PolarsDataType'
| Returns the storage type for this extension type.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| ext_from_params(name: 'str', storage: 'PolarsDataType', metadata: 'str | None') -> 'Any'
| Creates an Extension type instance from its parameters.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Binary
Python Library Documentation: class Binary in module polars.datatypes.classes
class Binary(DataType)
| Binary type.
|
| Method resolution order:
| Binary
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Boolean
Python Library Documentation: class Boolean in module polars.datatypes.classes
class Boolean(DataType)
| Boolean type.
|
| Method resolution order:
| Boolean
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Catalog
Python Library Documentation: class Catalog in module polars.catalog.unity.client
class Catalog(builtins.object)
| Catalog(workspace_url: 'str', *, bearer_token: 'str | None' = 'auto', require_https: 'bool' = True) -> 'None'
|
| Unity catalog client.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Methods defined here:
|
| __init__(self, workspace_url: 'str', *, bearer_token: 'str | None' = 'auto', require_https: 'bool' = True) -> 'None'
| Initialize a catalog client.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| workspace_url
| URL of the workspace, or alternatively the URL of the Unity catalog
| API endpoint.
| bearer_token
| Bearer token to authenticate with. This can also be set to:
|
| * "auto": Automatically retrieve bearer tokens from the environment.
| * "databricks-sdk": Use the Databricks SDK to retrieve and use the
| bearer token from the environment.
| require_https
| Require the `workspace_url` to use HTTPS.
|
| create_catalog(self, catalog_name: 'str', *, comment: 'str | None' = None, storage_root: 'str | None' = None) -> 'CatalogInfo'
| Create a catalog.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
| comment
| Leaves a comment about the catalog.
| storage_root
| Base location at which to store the catalog.
|
| create_namespace(self, catalog_name: 'str', namespace: 'str', *, comment: 'str | None' = None, storage_root: 'str | None' = None) -> 'NamespaceInfo'
| Create a namespace (unity schema) in the catalog.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
| namespace
| Name of the namespace (unity schema).
| comment
| Leaves a comment about the table.
| storage_root
| Base location at which to store the namespace.
|
| create_table(self, catalog_name: 'str', namespace: 'str', table_name: 'str', *, schema: 'SchemaDict | None', table_type: 'TableType', data_source_format: 'DataSourceFormat | None' = None, comment: 'str | None' = None, storage_root: 'str | None' = None, properties: 'dict[str, str] | None' = None) -> 'TableInfo'
| Create a table in the catalog.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
| namespace
| Name of the namespace (unity schema).
| table_name
| Name of the table.
| schema
| Schema of the table.
| table_type
| Type of the table
| data_source_format
| Storage format of the table.
| comment
| Leaves a comment about the table.
| storage_root
| Base location at which to store the table.
| properties
| Extra key-value metadata to store.
|
| delete_catalog(self, catalog_name: 'str', *, force: 'bool' = False) -> 'None'
| Delete a catalog.
|
| Note that depending on the table type and catalog server, this may not
| delete the actual data files from storage. For more details, please
| consult the documentation of the catalog provider you are using.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
| force
| Forcibly delete the catalog even if it is not empty.
|
| delete_namespace(self, catalog_name: 'str', namespace: 'str', *, force: 'bool' = False) -> 'None'
| Delete a namespace (unity schema) in the catalog.
|
| Note that depending on the table type and catalog server, this may not
| delete the actual data files from storage. For more details, please
| consult the documentation of the catalog provider you are using.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
| namespace
| Name of the namespace (unity schema).
| force
| Forcibly delete the namespace even if it is not empty.
|
| delete_table(self, catalog_name: 'str', namespace: 'str', table_name: 'str') -> 'None'
| Delete the table stored at this location.
|
| Note that depending on the table type and catalog server, this may not
| delete the actual data files from storage. For more details, please
| consult the documentation of the catalog provider you are using.
|
| If you would like to perform manual deletions, the storage location of
| the files can be found using `get_table_info`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
| namespace
| Name of the namespace (unity schema).
| table_name
| Name of the table.
|
| get_table_info(self, catalog_name: 'str', namespace: 'str', table_name: 'str') -> 'TableInfo'
| Retrieve the metadata of the specified table.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
| namespace
| Name of the namespace (unity schema).
| table_name
| Name of the table.
|
| list_catalogs(self) -> 'list[CatalogInfo]'
| List the available catalogs.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| list_namespaces(self, catalog_name: 'str') -> 'list[NamespaceInfo]'
| List the available namespaces (unity schema) under the specified catalog.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
|
| list_tables(self, catalog_name: 'str', namespace: 'str') -> 'list[TableInfo]'
| List the available tables under the specified schema.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
| namespace
| Name of the namespace (unity schema).
|
| scan_table(self, catalog_name: 'str', namespace: 'str', table_name: 'str', *, delta_table_version: 'int | str | datetime | None' = None, delta_table_options: 'dict[str, Any] | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None) -> 'LazyFrame'
| Retrieve the metadata of the specified table.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| catalog_name
| Name of the catalog.
| namespace
| Name of the namespace (unity schema).
| table_name
| Name of the table.
| delta_table_version
| Version of the table to scan (Deltalake only).
| delta_table_options
| Additional keyword arguments while reading a Deltalake table.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
|
| write_table(self, df: 'DataFrame', catalog_name: 'str', namespace: 'str', table_name: 'str', *, delta_mode: "Literal['error', 'append', 'overwrite', 'ignore', 'merge']" = 'error', delta_write_options: 'dict[str, Any] | None' = None, delta_merge_options: 'dict[str, Any] | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto') -> 'None | deltalake.table.TableMerger'
| Write a DataFrame to a catalog table.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| df
| DataFrame to write.
| catalog_name
| Name of the catalog.
| namespace
| Name of the namespace (unity schema).
| table_name
| Name of the table.
| delta_mode : {'error', 'append', 'overwrite', 'ignore', 'merge'}
| (For delta tables) How to handle existing data.
|
| - If 'error', throw an error if the table already exists (default).
| - If 'append', will add new data.
| - If 'overwrite', will replace table with new data.
| - If 'ignore', will not write anything if table already exists.
| - If 'merge', return a `TableMerger` object to merge data from the DataFrame
| with the existing data.
| delta_write_options
| (For delta tables) Additional keyword arguments while writing a
| Delta lake Table.
| See a list of supported write options `here <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake>`__.
| delta_merge_options
| (For delta tables) Keyword arguments which are required to `MERGE` a
| Delta lake Table.
| See a list of supported merge options `here <https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.merge>`__.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Categorical
Python Library Documentation: class Categorical in module polars.datatypes.classes
class Categorical(DataType)
| Categorical(categories: 'Categories | str | None' = None, *, ordering: 'CategoricalOrdering | None' = None) -> 'None'
|
| A categorical encoding of a set of strings.
|
| Parameters
| ----------
| categories
| The categories used for this type; must be a :py:class:`Categories`
| instance, or a string which is interpreted as the name of a
| :py:class:`Categories`. If not provided, the global categories
| (`pl.Categories()`) are used.
|
| For legacy reasons if the string is either `"physical"` or `"lexical"`,
| it is ignored and a warning is issued. If you wish to use a `Categories`
| named `"physical"` or `"lexical"`, please pass it using
| :py:class:`Categories` explicitly.
|
| ordering : {'lexical', 'physical'}
| This used to specify how this type was ordered, but now does nothing.
|
| .. deprecated:: 1.32.0
| Parameter is now ignored. Always behaves as if `'lexical'` was passed.
|
| See Also
| --------
| Categories
|
| Method resolution order:
| Categorical
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, categories: 'Categories | str | None' = None, *, ordering: 'CategoricalOrdering | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'categories': 'Categories', 'ordering': 'Categorica...
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Categories
Python Library Documentation: class Categories in module polars.datatypes.classes
class Categories(builtins.object)
| Categories(name: 'str | None' = None, namespace: 'str' = '', physical: 'PolarsDataType' = UInt32) -> 'None'
|
| A named collection of categories for :py:class:`Categorical`.
|
| Two categories are considered equal (and will use the same physical mapping of
| categories to strings) if they have the same name, namespace and physical backing
| type, even if they are created in separate calls to `Categories`.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| name
| The name of this `Categories`. If set to `None` or an empty string, this
| refers to the global categories.
|
| namespace
| An optional namespace for this `Categories`. Defaults to the empty string.
| If the name is empty or `None` indicating the global categories, the
| namespace must also be empty.
|
| physical : {UInt8, UInt16, UInt32}
| The physical type used to represent the categories. Defaults to
| :py:class:`UInt32`.
|
| See Also
| --------
| Categorical
|
| Examples
| --------
| A `Categories` instance can be indexed using either string or integer keys:
|
| >>> fruit = pl.Categories("fruit")
| >>> s = pl.Series(["apple", "banana", "orange"], dtype=pl.Categorical(fruit))
| >>> fruit[0]
| 'apple'
| >>> fruit["apple"]
| 0
|
| All `Categories` objects with the same name, namespace and physical type
| share the same mapping, even if they're created separately:
|
| >>> fruit2 = pl.Categories("fruit")
| >>> fruit2["banana"]
| 1
|
| To get a list of all categories, you can iterate over the `Categories` instance:
|
| >>> list(fruit)
| ['apple', 'banana', 'orange']
|
| .. note::
| Because the categories are backed by a concurrent data structure, physical
| category values may be reserved before they are assigned a string lexical
| value if concurrent queries are running. As a result, the resulting `Series`
| may contain `None` values.
|
| The `Categories` instance is only a weak reference to the actual
| mapping stored in Polars. If no actual data exists using this mapping (like
| a `Series` or `DataFrame`), the mapping is cleaned up by Polars:
|
| >>> del s
| >>> print(fruit["apple"])
| None
|
| If you wish to keep a persistent mapping, simply keep alive some object which
| uses the mapping, e.g. `keepalive = pl.Series([], dtype=pl.Categorical(fruit))`.
|
| Methods defined here:
|
| __contains__(self, item: 'str | int') -> 'bool'
|
| __eq__(self, other: 'object') -> 'bool'
| Return self==value.
|
| __getitem__(self, key: 'str | int | None') -> 'str | int | None'
|
| __getstate__(self) -> 'tuple[str, str, PolarsDataType]'
| Helper for pickle.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, name: 'str | None' = None, namespace: 'str' = '', physical: 'PolarsDataType' = UInt32) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> 'Iterator[str | None]'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, state: 'tuple[str, str, PolarsDataType]') -> 'None'
|
| is_global(self) -> 'bool'
| Returns whether this refers to the global categories.
|
| name(self) -> 'str'
| The name of this `Categories`.
|
| namespace(self) -> 'str'
| The namespace of this `Categories`.
|
| physical(self) -> 'PolarsDataType'
| The physical type used to represent the categories.
|
| to_dict(self) -> 'dict[str, int]'
| Return a dictionary mapping category strings to their physical category values.
|
| Examples
| --------
| >>> fruit = pl.Categories("fruit")
| >>> s = pl.Series(["apple", "banana", "orange"], dtype=pl.Categorical(fruit))
| >>> fruit.to_dict()
| {'apple': 0, 'banana': 1, 'orange': 2}
|
| to_series(self) -> 'Series'
| Return a :class:`Series` containing all categories in this `Categories`.
|
| The categories are ordered by their physical category value.
|
| .. note::
| Because the categories are backed by a concurrent data structure, physical
| category values may be reserved before they are assigned a string lexical
| value if concurrent queries are running. As a result, the resulting `Series`
| may contain `None` values.
|
| Examples
| --------
| >>> fruit = pl.Categories("fruit")
| >>> s = pl.Series(["apple", "banana", "orange"], dtype=pl.Categorical(fruit))
| >>> fruit.to_series()
| shape: (3,)
| Series: 'fruit' [str]
| [
| "apple"
| "banana"
| "orange"
| ]
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| random(namespace: 'str' = '', physical: 'PolarsDataType' = UInt32) -> 'Categories'
| Creates a new `Categories` with a random name.
|
| Parameters
| ----------
| namespace
| An optional namespace for this `Categories`. Defaults to the empty string.
|
| physical : {UInt8, UInt16, UInt32}
| The physical type used to represent the categories. Defaults
| to :py:class:`UInt32`.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_categories': 'PyCategories'}
polars.CompatLevel
Python Library Documentation: class CompatLevel in module polars.interchange.protocol
class CompatLevel(builtins.object)
| CompatLevel() -> 'None'
|
| Data structure compatibility level.
|
| Methods defined here:
|
| __init__(self) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| newest() -> 'CompatLevel'
| Get the highest supported compatibility level.
|
| .. warning::
| Highest compatibility level is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| oldest() -> 'CompatLevel'
| Get the most compatible level.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_version': 'int'}
polars.Config
Python Library Documentation: class Config in module polars.config
class Config(contextlib.ContextDecorator)
| Config(*, restore_defaults: 'bool' = False, apply_on_context_enter: 'bool' = False, **options: 'Unpack[ConfigParameters]') -> 'None'
|
| Configure polars; offers options for table formatting and more.
|
| Notes
| -----
| Can also be used as a context manager OR a function decorator in order to
| temporarily scope the lifetime of specific options. For example:
|
| >>> with pl.Config() as cfg:
| ... # set verbose for more detailed output within the scope
| ... cfg.set_verbose(True) # doctest: +IGNORE_RESULT
| >>> # scope exit - no longer in verbose mode
|
| This can also be written more compactly as:
|
| >>> with pl.Config(verbose=True):
| ... pass
|
| (The compact format is available for all `Config` methods that take a single value).
|
| Alternatively, you can use as a decorator in order to scope the duration of the
| selected options to a specific function:
|
| >>> @pl.Config(verbose=True, apply_on_context_enter=True)
| ... def test():
| ... pass
|
| Method resolution order:
| Config
| contextlib.ContextDecorator
| builtins.object
|
| Methods defined here:
|
| __enter__(self) -> 'Self'
| Support setting Config options that are reset on scope exit.
|
| __eq__(self, other: 'object') -> 'bool'
| Return self==value.
|
| __exit__(self, exc_type: 'type[BaseException] | None', exc_val: 'BaseException | None', exc_tb: 'TracebackType | None') -> 'None'
| Reset any Config options that were set within the scope.
|
| __init__(self, *, restore_defaults: 'bool' = False, apply_on_context_enter: 'bool' = False, **options: 'Unpack[ConfigParameters]') -> 'None'
| Initialise a Config object instance for context manager usage.
|
| Any `options` kwargs should correspond to the available named "set_*"
| methods, but are allowed to omit the "set_" prefix for brevity.
|
| Parameters
| ----------
| restore_defaults
| set all options to their default values (this is applied before
| setting any other options).
| apply_on_context_enter
| defer applying the options until a context is entered. This allows you
| to create multiple `Config` instances with different options, and then
| reuse them independently as context managers or function decorators
| with specific bundles of parameters.
| **options
| keyword args that will set the option; equivalent to calling the
| named "set_<option>" method with the given value.
|
| Notes
| -----
| The `apply_on_context_enter` parameter should almost *always* be set True
| when using `Config` as a decorator to ensure that the options are applied
| only within the scope of the decorated function (and not globally at
| module import time).
|
| Examples
| --------
| Customise Polars table formatting while in context scope:
|
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> with pl.Config(
| ... # these options will be set for scope duration
| ... tbl_formatting="MARKDOWN",
| ... tbl_hide_dataframe_shape=True,
| ... tbl_rows=10,
| ... ):
| ... print(df)
| | abc | xyz |
| | --- | --- |
| | f64 | bool |
| |-----|-------|
| | 1.0 | true |
| | 2.5 | false |
| | 5.0 | true |
|
| Establish several independent Config instances for use in different contexts;
| setting `apply_on_context_enter=True` defers setting the parameters until a
| context (or function, when used as a decorator) is actually entered:
|
| >>> cfg_polars_verbose = pl.Config(
| ... verbose=True,
| ... apply_on_context_enter=True,
| ... )
| >>> cfg_polars_detailed_tables = pl.Config(
| ... tbl_rows=25,
| ... tbl_cols=25,
| ... tbl_width_chars=200,
| ... apply_on_context_enter=True,
| ... )
|
| These Config instances can now be applied independently and re-used:
|
| >>> @cfg_polars_verbose
| ... def traced_function(df: pl.DataFrame) -> pl.DataFrame:
| ... return polars_operations(df)
|
| >>> @cfg_polars_detailed_tables
| ... def print_detailed_frames(*frames: pl.DataFrame) -> None:
| ... for df in frames:
| ... print(df)
|
| __ne__(self, other: 'object') -> 'bool'
| Return self!=value.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| load(cfg: 'str') -> 'Config'
| Load (and set) previously saved Config options from a JSON string.
|
| Parameters
| ----------
| cfg : str
| JSON string produced by `Config.save()`.
|
| See Also
| --------
| load_from_file : Load (and set) Config options from a JSON file.
| save : Save the current set of Config options as a JSON string or file.
|
| load_from_file(file: 'Path | str') -> 'Config'
| Load (and set) previously saved Config options from file.
|
| Parameters
| ----------
| file : Path | str
| File path to a JSON string produced by `Config.save()`.
|
| See Also
| --------
| load : Load (and set) Config options from a JSON string.
| save : Save the current set of Config options as a JSON string or file.
|
| reload_env_vars() -> 'None'
| Update the Polars config from the set environment variables.
|
| Normally you need not call this, it is only necessary when updating
| undocumented (environment-variable-only) config flags from Python after
| importing Polars.
|
| restore_defaults() -> 'type[Config]'
| Reset all polars Config settings to their default state.
|
| Notes
| -----
| This method operates by removing all Config options from the environment,
| and then setting any local (non-env) options back to their default value.
|
| Examples
| --------
| >>> cfg = pl.Config.restore_defaults() # doctest: +SKIP
|
| save(*, if_set: 'bool' = False) -> 'str'
| Save the current set of Config options as a JSON string.
|
| Parameters
| ----------
| if_set
| By default this will save the state of all configuration options; set
| to `False` to save only those that have been set to a non-default value.
|
| See Also
| --------
| load : Load (and set) Config options from a JSON string.
| load_from_file : Load (and set) Config options from a JSON file.
| save_to_file : Save the current set of Config options as a JSON file.
|
| Examples
| --------
| >>> json_state = pl.Config.save()
|
| Returns
| -------
| str
| JSON string containing current Config options.
|
| save_to_file(file: 'Path | str') -> 'None'
| Save the current set of Config options as a JSON file.
|
| Parameters
| ----------
| file
| Optional path to a file into which the JSON string will be written.
| Leave as `None` to return the JSON string directly.
|
| See Also
| --------
| load : Load (and set) Config options from a JSON string.
| load_from_file : Load (and set) Config options from a JSON file.
| save : Save the current set of Config options as a JSON string.
|
| Examples
| --------
| >>> pl.Config().save_to_file("~/polars/config.json") # doctest: +SKIP
|
| set_ascii_tables(active: 'bool | None' = True) -> 'type[Config]'
| Use ASCII characters to display table outlines.
|
| Set False to revert to the standard UTF8_FULL_CONDENSED formatting style.
|
| See Also
| --------
| set_tbl_formatting : Set the table formatting style (includes Markdown option).
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_ascii_tables(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ +-----+-------+
| # │ abc ┆ xyz │ | abc | xyz |
| # │ --- ┆ --- │ | --- | --- |
| # │ f64 ┆ bool │ | f64 | bool |
| # ╞═════╪═══════╡ +=============+
| # │ 1.0 ┆ true │ >> | 1.0 | true |
| # │ 2.5 ┆ false │ | 2.5 | false |
| # │ 5.0 ┆ true │ | 5.0 | true |
| # └─────┴───────┘ +-----+-------+
|
| set_auto_structify(active: 'bool | None' = False) -> 'type[Config]'
| Allow multi-output expressions to be automatically turned into Structs.
|
| .. note::
| Deprecated since 1.32.0.
|
| Examples
| --------
| >>> df = pl.DataFrame({"v": [1, 2, 3], "v2": [4, 5, 6]})
| >>> with pl.Config(set_auto_structify=True): # doctest: +SKIP
| ... out = df.select(pl.all())
| >>> out # doctest: +SKIP
| shape: (3, 1)
| ┌───────────┐
| │ v │
| │ --- │
| │ struct[2] │
| ╞═══════════╡
| │ {1,4} │
| │ {2,5} │
| │ {3,6} │
| └───────────┘
|
| set_decimal_separator(separator: 'str | None' = None) -> 'type[Config]'
| Set the decimal separator character.
|
| Parameters
| ----------
| separator : str, bool
| Character to use as the decimal separator.
| Set to ``None`` to revert to the default (".").
|
| See Also
| --------
| set_thousands_separator : Set the thousands grouping separator character.
|
| Examples
| --------
| >>> df = pl.DataFrame({"v": [9876.54321, 1010101.0, -123456.78]})
| >>> with pl.Config(
| ... tbl_cell_numeric_alignment="RIGHT",
| ... thousands_separator=".",
| ... decimal_separator=",",
| ... float_precision=3,
| ... ):
| ... print(df)
| shape: (3, 1)
| ┌───────────────┐
| │ v │
| │ --- │
| │ f64 │
| ╞═══════════════╡
| │ 9.876,543 │
| │ 1.010.101,000 │
| │ -123.456,780 │
| └───────────────┘
|
| set_default_credential_provider(credential_provider: "CredentialProviderFunction | Literal['auto'] | None") -> 'type[Config]'
| Set a default credential provider.
|
| Sets the default credential provider to be used for functions that
| read / write to cloud storage.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| Can also be set to None, which globally disables auto-initialization
| of credential providers, or "auto" (the default behavior).
|
| Examples
| --------
| >>> pl.Config.set_default_credential_provider(
| ... pl.CredentialProviderAWS(
| ... assume_role={"RoleArn": "...", "RoleSessionName": "..."}
| ... )
| ... )
| <class 'polars.config.Config'>
|
| set_engine_affinity(engine: 'EngineType | None' = None) -> 'type[Config]'
| Set which engine to use by default.
|
| Parameters
| ----------
| engine : {None, 'auto', 'in-memory', 'streaming', 'gpu'}
| The default execution engine Polars will attempt to use
| when calling `.collect()`. However, the query is not
| guaranteed to execute with the specified engine.
|
| Examples
| --------
| >>> pl.Config.set_engine_affinity("streaming") # doctest: +SKIP
| >>> lf = pl.LazyFrame({"v": [1, 2, 3], "v2": [4, 5, 6]}) # doctest: +SKIP
| >>> lf.max().collect() # doctest: +SKIP
| shape: (3, 2)
| ┌─────┬─────┐
| │ v ┆ v2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| │ 3 ┆ 6 │
| └─────┴─────┘
| >>> pl.Config.set_engine_affinity("gpu") # doctest: +SKIP
| >>> lf.max().collect() # doctest: +SKIP
| shape: (3, 2)
| ┌─────┬─────┐
| │ v ┆ v2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| │ 3 ┆ 6 │
| └─────┴─────┘
|
| Raises
| ------
| ValueError: if engine is not recognised.
| NotImplementedError: if engine is a GPUEngine object
|
| set_expr_depth_warning(limit: 'int') -> 'type[Config]'
| Set the expression depth that Polars will accept without triggering a warning.
|
| Having too deep expressions (several 1000s) can lead to overflowing the stack and might be worth a refactor.
|
| set_float_precision(precision: 'int | None' = None) -> 'type[Config]'
| Control the number of decimal places displayed for floating point values.
|
| Parameters
| ----------
| precision : int
| Number of decimal places to display; set to `None` to revert to the
| default/standard behaviour.
|
| Notes
| -----
| When setting this to a larger value you should ensure that you are aware of both
| the limitations of floating point representations, and of the precision of the
| data that you are looking at.
|
| This setting only applies to :class:`.Float16`, :class:`.Float32`, and
| :class:`.Float64` dtypes; it does not cover :class:`.Decimal` dtype values
| (which are displayed at their native level of precision).
|
| Examples
| --------
| Set a large maximum float precision:
|
| >>> from math import pi, e
| >>> df = pl.DataFrame({"const": ["pi", "e"], "value": [pi, e]})
| >>> with pl.Config(float_precision=15):
| ... print(df)
| shape: (2, 2)
| ┌───────┬───────────────────┐
| │ const ┆ value │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪═══════════════════╡
| │ pi ┆ 3.141592653589793 │
| │ e ┆ 2.718281828459045 │
| └───────┴───────────────────┘
|
| Set a fixed float precision and align numeric columns to the
| right in order to cleanly line-up the decimal separator:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["xx", "yy"],
| ... "b": [-11111111, 44444444444],
| ... "c": [100000.987654321, -23456789],
| ... }
| ... )
| >>> with pl.Config(
| ... tbl_cell_numeric_alignment="RIGHT",
| ... thousands_separator=",",
| ... float_precision=3,
| ... ):
| ... print(df)
| shape: (2, 3)
| ┌─────┬────────────────┬─────────────────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ f64 │
| ╞═════╪════════════════╪═════════════════╡
| │ xx ┆ -11,111,111 ┆ 100,000.988 │
| │ yy ┆ 44,444,444,444 ┆ -23,456,789.000 │
| └─────┴────────────────┴─────────────────┘
|
| set_fmt_float(fmt: 'FloatFmt | None' = 'mixed') -> 'type[Config]'
| Control how floating point values are displayed.
|
| Parameters
| ----------
| fmt : {"mixed", "full"}
| How to format floating point numbers:
|
| - "mixed": Limit the number of decimal places and use scientific
| notation for large/small values.
| - "full": Print the full precision of the floating point number.
|
| Examples
| --------
| "mixed" float formatting:
|
| >>> s = pl.Series([1.2304980958725870923, 1e6, 1e-8])
| >>> with pl.Config(set_fmt_float="mixed"):
| ... print(s)
| shape: (3,)
| Series: '' [f64]
| [
| 1.230498
| 1e6
| 1.0000e-8
| ]
|
| "full" float formatting:
|
| >>> with pl.Config(set_fmt_float="full"):
| ... print(s)
| shape: (3,)
| Series: '' [f64]
| [
| 1.230498095872587
| 1000000
| 0.00000001
| ]
|
| set_fmt_str_lengths(n: 'int | None') -> 'type[Config]'
| Set the number of characters used to display string values.
|
| Parameters
| ----------
| n : int
| Number of characters to display.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "txt": [
| ... "Play it, Sam. Play 'As Time Goes By'.",
| ... "This is the beginning of a beautiful friendship.",
| ... ]
| ... }
| ... )
| >>> df.with_columns(pl.col("txt").str.len_bytes().alias("len"))
| shape: (2, 2)
| ┌─────────────────────────────────┬─────┐
| │ txt ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═════════════════════════════════╪═════╡
| │ Play it, Sam. Play 'As Time Go… ┆ 37 │
| │ This is the beginning of a bea… ┆ 48 │
| └─────────────────────────────────┴─────┘
| >>> with pl.Config(fmt_str_lengths=50):
| ... print(df)
| shape: (2, 1)
| ┌──────────────────────────────────────────────────┐
| │ txt │
| │ --- │
| │ str │
| ╞══════════════════════════════════════════════════╡
| │ Play it, Sam. Play 'As Time Goes By'. │
| │ This is the beginning of a beautiful friendship. │
| └──────────────────────────────────────────────────┘
|
| set_fmt_table_cell_list_len(n: 'int | None') -> 'type[Config]'
| Set the number of elements to display for List values.
|
| Empty lists will always print "[]". Negative values will result in all values
| being printed. A value of 0 will always "[…]" for lists with contents. A value
| of 1 will print only the final item in the list.
|
| Parameters
| ----------
| n : int
| Number of values to display.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "nums": [
| ... [1, 2, 3, 4, 5, 6],
| ... ]
| ... }
| ... )
| >>> df
| shape: (1, 1)
| ┌─────────────┐
| │ nums │
| │ --- │
| │ list[i64] │
| ╞═════════════╡
| │ [1, 2, … 6] │
| └─────────────┘
| >>> with pl.Config(fmt_table_cell_list_len=10):
| ... print(df)
| shape: (1, 1)
| ┌────────────────────┐
| │ nums │
| │ --- │
| │ list[i64] │
| ╞════════════════════╡
| │ [1, 2, 3, 4, 5, 6] │
| └────────────────────┘
|
| set_streaming_chunk_size(size: 'int | None') -> 'type[Config]'
| Overwrite chunk size used in `streaming` engine.
|
| By default, the chunk size is determined by the schema
| and size of the thread pool. For some datasets (esp.
| when you have large string elements) this can be too
| optimistic and lead to Out of Memory errors.
|
| Parameters
| ----------
| size
| Number of rows per chunk. Every thread will process chunks
| of this size.
|
| set_tbl_cell_alignment(format: 'Alignment | None') -> 'type[Config]'
| Set table cell alignment.
|
| Parameters
| ----------
| format : str
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"column_abc": [1.0, 2.5, 5.0], "column_xyz": [True, False, True]}
| ... )
| >>> pl.Config.set_tbl_cell_alignment("RIGHT") # doctest: +IGNORE_RESULT
| >>> print(df)
| shape: (3, 2)
| ┌────────────┬────────────┐
| │ column_abc ┆ column_xyz │
| │ --- ┆ --- │
| │ f64 ┆ bool │
| ╞════════════╪════════════╡
| │ 1.0 ┆ true │
| │ 2.5 ┆ false │
| │ 5.0 ┆ true │
| └────────────┴────────────┘
|
| Raises
| ------
| ValueError: if alignment string not recognised.
|
| set_tbl_cell_numeric_alignment(format: 'Alignment | None') -> 'type[Config]'
| Set table cell alignment for numeric columns.
|
| Parameters
| ----------
| format : str
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| Examples
| --------
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "abc": [11, 2, 333],
| ... "mno": [date(2023, 10, 29), None, date(2001, 7, 5)],
| ... "xyz": [True, False, None],
| ... }
| ... )
| >>> pl.Config.set_tbl_cell_numeric_alignment("RIGHT") # doctest: +IGNORE_RESULT
| >>> print(df)
| shape: (3, 3)
| ┌─────┬────────────┬───────┐
| │ abc ┆ mno ┆ xyz │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ date ┆ bool │
| ╞═════╪════════════╪═══════╡
| │ 11 ┆ 2023-10-29 ┆ true │
| │ 2 ┆ null ┆ false │
| │ 333 ┆ 2001-07-05 ┆ null │
| └─────┴────────────┴───────┘
|
| Raises
| ------
| KeyError: if alignment string not recognised.
|
| set_tbl_cols(n: 'int | None') -> 'type[Config]'
| Set the number of columns that are visible when displaying tables.
|
| Parameters
| ----------
| n : int
| Number of columns to display; if `n < 0` (eg: -1), display all columns.
|
| Examples
| --------
| Set number of displayed columns to a low value:
|
| >>> with pl.Config() as cfg:
| ... cfg.set_tbl_cols(5)
| ... df = pl.DataFrame({str(i): [i] for i in range(100)})
| ... print(df)
| <class 'polars.config.Config'>
| shape: (1, 100)
| ┌─────┬─────┬─────┬───┬─────┬─────┐
| │ 0 ┆ 1 ┆ 2 ┆ … ┆ 98 ┆ 99 │
| │ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 ┆ … ┆ 98 ┆ 99 │
| └─────┴─────┴─────┴───┴─────┴─────┘
|
| >>> with pl.Config(tbl_cols=10):
| ... print(df)
| shape: (1, 100)
| ┌─────┬─────┬─────┬─────┬─────┬───┬─────┬─────┬─────┬─────┬─────┐
| │ 0 ┆ 1 ┆ 2 ┆ 3 ┆ 4 ┆ … ┆ 95 ┆ 96 ┆ 97 ┆ 98 ┆ 99 │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╪═════╪═══╪═════╪═════╪═════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 ┆ 3 ┆ 4 ┆ … ┆ 95 ┆ 96 ┆ 97 ┆ 98 ┆ 99 │
| └─────┴─────┴─────┴─────┴─────┴───┴─────┴─────┴─────┴─────┴─────┘
|
| set_tbl_column_data_type_inline(active: 'bool | None' = True) -> 'type[Config]'
| Display the data type next to the column name (to the right, in parentheses).
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_column_data_type_inline(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ ┌───────────┬────────────┐
| # │ abc ┆ xyz │ │ abc (f64) ┆ xyz (bool) │
| # │ --- ┆ --- │ ╞═══════════╪════════════╡
| # │ f64 ┆ bool │ │ 1.0 ┆ true │
| # ╞═════╪═══════╡ >> │ 2.5 ┆ false │
| # │ 1.0 ┆ true │ │ 5.0 ┆ true │
| # │ 2.5 ┆ false │ └───────────┴────────────┘
| # │ 5.0 ┆ true │
| # └─────┴───────┘
|
| set_tbl_dataframe_shape_below(active: 'bool | None' = True) -> 'type[Config]'
| Print the DataFrame shape information below the data when displaying tables.
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_dataframe_shape_below(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) ┌─────┬───────┐
| # ┌─────┬───────┐ │ abc ┆ xyz │
| # │ abc ┆ xyz │ │ --- ┆ --- │
| # │ --- ┆ --- │ │ f64 ┆ bool │
| # │ f64 ┆ bool │ ╞═════╪═══════╡
| # ╞═════╪═══════╡ >> │ 1.0 ┆ true │
| # │ 1.0 ┆ true │ │ 2.5 ┆ false │
| # │ 2.5 ┆ false │ │ 5.0 ┆ true │
| # │ 5.0 ┆ true │ └─────┴───────┘
| # └─────┴───────┘ shape: (3, 2)
|
| set_tbl_formatting(format: 'TableFormatNames | None' = None, rounded_corners: 'bool | None' = False) -> 'type[Config]'
| Set table formatting style.
|
| Parameters
| ----------
| format : str
| * "ASCII_FULL": ASCII, with all borders and lines, including row dividers.
| * "ASCII_FULL_CONDENSED": Same as ASCII_FULL, but with dense row spacing.
| * "ASCII_NO_BORDERS": ASCII, no borders.
| * "ASCII_BORDERS_ONLY": ASCII, borders only.
| * "ASCII_BORDERS_ONLY_CONDENSED": ASCII, borders only, dense row spacing.
| * "ASCII_HORIZONTAL_ONLY": ASCII, horizontal lines only.
| * "ASCII_MARKDOWN": Markdown format (ascii ellipses for truncated values).
| * "MARKDOWN": Markdown format (utf8 ellipses for truncated values).
| * "UTF8_FULL": UTF8, with all borders and lines, including row dividers.
| * "UTF8_FULL_CONDENSED": Same as UTF8_FULL, but with dense row spacing.
| * "UTF8_NO_BORDERS": UTF8, no borders.
| * "UTF8_BORDERS_ONLY": UTF8, borders only.
| * "UTF8_HORIZONTAL_ONLY": UTF8, horizontal lines only.
| * "NOTHING": No borders or other lines.
|
| rounded_corners : bool
| Apply rounded corners to UTF8-styled tables (no-op for ASCII formats).
|
| Notes
| -----
| The UTF8 styles all use one or more of the semigraphic box-drawing characters
| found in the Unicode Box Drawing block, which are not ASCII compatible:
| https://en.wikipedia.org/wiki/Box-drawing_character#Box_Drawing
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"abc": [-2.5, 5.0], "mno": ["hello", "world"], "xyz": [True, False]}
| ... )
| >>> with pl.Config(
| ... tbl_formatting="MARKDOWN",
| ... tbl_hide_column_data_types=True,
| ... tbl_hide_dataframe_shape=True,
| ... ):
| ... print(df)
| | abc | mno | xyz |
| |------|-------|-------|
| | -2.5 | hello | true |
| | 5.0 | world | false |
|
| Raises
| ------
| ValueError: if format string not recognised.
|
| set_tbl_hide_column_data_types(active: 'bool | None' = True) -> 'type[Config]'
| Hide table column data types (i64, f64, str etc.).
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_hide_column_data_types(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ ┌─────┬───────┐
| # │ abc ┆ xyz │ │ abc ┆ xyz │
| # │ --- ┆ --- │ ╞═════╪═══════╡
| # │ f64 ┆ bool │ │ 1.0 ┆ true │
| # ╞═════╪═══════╡ >> │ 2.5 ┆ false │
| # │ 1.0 ┆ true │ │ 5.0 ┆ true │
| # │ 2.5 ┆ false │ └─────┴───────┘
| # │ 5.0 ┆ true │
| # └─────┴───────┘
|
| set_tbl_hide_column_names(active: 'bool | None' = True) -> 'type[Config]'
| Hide table column names.
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_hide_column_names(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ ┌─────┬───────┐
| # │ abc ┆ xyz │ │ f64 ┆ bool │
| # │ --- ┆ --- │ ╞═════╪═══════╡
| # │ f64 ┆ bool │ │ 1.0 ┆ true │
| # ╞═════╪═══════╡ >> │ 2.5 ┆ false │
| # │ 1.0 ┆ true │ │ 5.0 ┆ true │
| # │ 2.5 ┆ false │ └─────┴───────┘
| # │ 5.0 ┆ true │
| # └─────┴───────┘
|
| set_tbl_hide_dataframe_shape(active: 'bool | None' = True) -> 'type[Config]'
| Hide the DataFrame shape information when displaying tables.
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_hide_dataframe_shape(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) ┌─────┬───────┐
| # ┌─────┬───────┐ │ abc ┆ xyz │
| # │ abc ┆ xyz │ │ --- ┆ --- │
| # │ --- ┆ --- │ │ f64 ┆ bool │
| # │ f64 ┆ bool │ ╞═════╪═══════╡
| # ╞═════╪═══════╡ │ 1.0 ┆ true │
| # │ 1.0 ┆ true │ >> │ 2.5 ┆ false │
| # │ 2.5 ┆ false │ │ 5.0 ┆ true │
| # │ 5.0 ┆ true │ └─────┴───────┘
| # └─────┴───────┘
|
| set_tbl_hide_dtype_separator(active: 'bool | None' = True) -> 'type[Config]'
| Hide the '---' separator displayed between the column names and column types.
|
| See Also
| --------
| set_tbl_column_data_type_inline : Display the data type inline with the colname.
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_hide_dtype_separator(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ ┌─────┬───────┐
| # │ abc ┆ xyz │ │ abc ┆ xyz │
| # │ --- ┆ --- │ │ f64 ┆ bool │
| # │ f64 ┆ bool │ ╞═════╪═══════╡
| # ╞═════╪═══════╡ │ 1.0 ┆ true │
| # │ 1.0 ┆ true │ >> │ 2.5 ┆ false │
| # │ 2.5 ┆ false │ │ 5.0 ┆ true │
| # │ 5.0 ┆ true │ └─────┴───────┘
| # └─────┴───────┘
|
| set_tbl_rows(n: 'int | None') -> 'type[Config]'
| Set the max number of rows used to draw the table (both Dataframe and Series).
|
| Parameters
| ----------
| n : int
| Number of rows to display; if `n < 0` (eg: -1), display all
| rows (DataFrame) and all elements (Series).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"abc": [1.0, 2.5, 3.5, 5.0], "xyz": [True, False, True, False]}
| ... )
| >>> with pl.Config(tbl_rows=2):
| ... print(df)
| shape: (4, 2)
| ┌─────┬───────┐
| │ abc ┆ xyz │
| │ --- ┆ --- │
| │ f64 ┆ bool │
| ╞═════╪═══════╡
| │ 1.0 ┆ true │
| │ … ┆ … │
| │ 5.0 ┆ false │
| └─────┴───────┘
|
| set_tbl_width_chars(width: 'int | None') -> 'type[Config]'
| Set the maximum width of a table in characters.
|
| Parameters
| ----------
| width : int
| Maximum table width in characters; if n < 0 (eg: -1), display full width.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "id": ["SEQ1", "SEQ2"],
| ... "seq": ["ATGATAAAGGAG", "GCAACGCATATA"],
| ... }
| ... )
| >>> df
| shape: (2, 2)
| ┌──────┬──────────────┐
| │ id ┆ seq │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞══════╪══════════════╡
| │ SEQ1 ┆ ATGATAAAGGAG │
| │ SEQ2 ┆ GCAACGCATATA │
| └──────┴──────────────┘
| >>> pl.Config.set_tbl_width_chars(12) # doctest: +IGNORE_RESULT
| >>> df
| shape: (2, 2)
| ┌─────┬─────┐
| │ id ┆ seq │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞═════╪═════╡
| │ SEQ ┆ ATG │
| │ 1 ┆ ATA │
| │ ┆ AAG │
| │ ┆ GAG │
| │ SEQ ┆ GCA │
| │ 2 ┆ ACG │
| │ ┆ CAT │
| │ ┆ ATA │
| └─────┴─────┘
|
| set_thousands_separator(separator: 'str | bool | None' = None) -> 'type[Config]'
| Set the thousands grouping separator character.
|
| Parameters
| ----------
| separator : str, bool
| Set True to use the default "," (thousands) and "." (decimal) separators.
| Can also set a custom char, or set ``None`` to omit the separator.
|
| See Also
| --------
| set_decimal_separator : Set the decimal separator character.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "x": [1234567, -987654, 10101],
| ... "y": [1234.5, 100000.0, -7654321.25],
| ... }
| ... )
| >>> with pl.Config(
| ... tbl_cell_numeric_alignment="RIGHT",
| ... thousands_separator=True,
| ... float_precision=2,
| ... ):
| ... print(df)
| shape: (3, 2)
| ┌───────────┬───────────────┐
| │ x ┆ y │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═══════════╪═══════════════╡
| │ 1,234,567 ┆ 1,234.50 │
| │ -987,654 ┆ 100,000.00 │
| │ 10,101 ┆ -7,654,321.25 │
| └───────────┴───────────────┘
| >>> with pl.Config(
| ... tbl_cell_numeric_alignment="RIGHT",
| ... thousands_separator=".",
| ... decimal_separator=",",
| ... float_precision=2,
| ... ):
| ... print(df)
| shape: (3, 2)
| ┌───────────┬───────────────┐
| │ x ┆ y │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═══════════╪═══════════════╡
| │ 1.234.567 ┆ 1.234,50 │
| │ -987.654 ┆ 100.000,00 │
| │ 10.101 ┆ -7.654.321,25 │
| └───────────┴───────────────┘
|
| set_trim_decimal_zeros(active: 'bool | None' = True) -> 'type[Config]'
| Strip trailing zeros from Decimal data type values.
|
| Parameters
| ----------
| active : bool
| Enable stripping of trailing '0' characters from Decimal values.
|
| Examples
| --------
| >>> from decimal import Decimal as D
| >>> df = pl.DataFrame(
| ... data={"d": [D("1.01000"), D("-5.67890")]},
| ... schema={"d": pl.Decimal(scale=5)},
| ... )
| >>> with pl.Config(trim_decimal_zeros=False):
| ... print(df)
| shape: (2, 1)
| ┌───────────────┐
| │ d │
| │ --- │
| │ decimal[38,5] │
| ╞═══════════════╡
| │ 1.01000 │
| │ -5.67890 │
| └───────────────┘
| >>> with pl.Config(trim_decimal_zeros=True):
| ... print(df)
| shape: (2, 1)
| ┌───────────────┐
| │ d │
| │ --- │
| │ decimal[38,5] │
| ╞═══════════════╡
| │ 1.01 │
| │ -5.6789 │
| └───────────────┘
|
| set_verbose(active: 'bool | None' = True) -> 'type[Config]'
| Enable additional verbose/debug logging.
|
| Examples
| --------
| >>> pl.Config.set_verbose(True) # doctest: +SKIP
| >>> with pl.Config(verbose=True): # doctest: +SKIP
| ... do_polars_operations()
|
| state(*, if_set: 'bool' = False, env_only: 'bool' = False) -> 'dict[str, str | None]'
| Show the current state of all Config variables in the environment as a dict.
|
| Parameters
| ----------
| if_set
| By default this will show the state of all `Config` environment variables.
| change this to `True` to restrict the returned dictionary to include only
| those that have been set to a specific value.
| env_only
| Include only Config environment variables in the output; some options (such
| as "set_fmt_float") are set directly, not via an environment variable.
|
| Examples
| --------
| >>> set_state = pl.Config.state(if_set=True)
| >>> all_state = pl.Config.state()
|
| warn_unstable(active: 'bool | None' = True) -> 'type[Config]'
| Issue a warning when unstable functionality is used.
|
| Enabling this setting may help avoid functionality that is still evolving,
| potentially reducing maintenance burden from API changes and bugs.
|
| Examples
| --------
| >>> pl.Config.warn_unstable(True) # doctest: +SKIP
| >>> pl.col("a").qcut(5) # doctest: +SKIP
| UnstableWarning: `qcut` is considered unstable. It may be changed at any point without it being considered a breaking change.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_context_options': 'ConfigParameters | None', '_or...
|
| __hash__ = None
|
| ----------------------------------------------------------------------
| Methods inherited from contextlib.ContextDecorator:
|
| __call__(self, func)
| Call self as a function.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from contextlib.ContextDecorator:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.CredentialProvider
Python Library Documentation: class CredentialProvider in module polars.io.cloud.credential_provider._providers
class CredentialProvider(abc.ABC)
| Base class for credential providers.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Method resolution order:
| CredentialProvider
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| __call__(self) -> 'CredentialProviderFunctionReturn'
| Fetches the credentials.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'__call__'})
polars.CredentialProviderAWS
Python Library Documentation: class CredentialProviderAWS in module polars.io.cloud.credential_provider._providers class CredentialProviderAWS(CachingCredentialProvider) | CredentialProviderAWS(*, profile_name: 'str | None' = None, region_name: 'str | None' = None, assume_role: 'AWSAssumeRoleKWArgs | None' = None, _auto_init_unhandled_key: 'str | None' = None, _storage_options_has_endpoint_url: 'bool' = False) -> 'None' | | AWS Credential Provider. | | Using this requires the `boto3` Python package to be installed. | | .. warning:: | This functionality is considered **unstable**. It may be changed | at any point without it being considered a breaking change. | | Method resolution order: | CredentialProviderAWS | CachingCredentialProvider | CredentialProvider | abc.ABC | builtins.object | | Methods defined here: | | __init__(self, *, profile_name: 'str | None' = None, region_name: 'str | None' = None, assume_role: 'AWSAssumeRoleKWArgs | None' = None, _auto_init_unhandled_key: 'str | None' = None, _storage_options_has_endpoint_url: 'bool' = False) -> 'None' | Initialize a credential provider for AWS. | | Parameters | ---------- | profile_name : str | Profile name to use from credentials file. | assume_role : AWSAssumeRoleKWArgs | None | Configure a role to assume. These are passed as kwarg parameters to | [STS.client.assume_role()](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts/client/assume_role.html#STS.Client.assume_role) | | retrieve_credentials_impl(self) -> 'CredentialProviderFunctionReturn' | Fetch the credentials for the configured profile name. | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | EmptyCredentialError = <class 'polars.io.cloud.credential_provider._pr... | Raised when boto3 returns empty credentials. | | This generally indicates that no credentials could be found in the | environment. | | | __abstractmethods__ = frozenset() | | ---------------------------------------------------------------------- | Methods inherited from CachingCredentialProvider: | | __call__(self) -> 'CredentialProviderFunctionReturn' | Fetches the credentials. | | __repr__(self) -> 'str' | Return repr(self). | | clear_cached_credentials(self) -> 'None' | | ---------------------------------------------------------------------- | Data descriptors inherited from CredentialProvider: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
polars.CredentialProviderAzure
Python Library Documentation: class CredentialProviderAzure in module polars.io.cloud.credential_provider._providers class CredentialProviderAzure(CachingCredentialProvider) | CredentialProviderAzure(*, scopes: 'list[str] | None' = None, tenant_id: 'str | None' = None, credential: 'Any | None' = None, _storage_account: 'str | None' = None) -> 'None' | | Azure Credential Provider. | | Using this requires the `azure-identity` Python package to be installed. | | .. warning:: | This functionality is considered **unstable**. It may be changed | at any point without it being considered a breaking change. | | Method resolution order: | CredentialProviderAzure | CachingCredentialProvider | CredentialProvider | abc.ABC | builtins.object | | Methods defined here: | | __init__(self, *, scopes: 'list[str] | None' = None, tenant_id: 'str | None' = None, credential: 'Any | None' = None, _storage_account: 'str | None' = None) -> 'None' | Initialize a credential provider for Microsoft Azure. | | By default, this uses `azure.identity.DefaultAzureCredential()`. | | Parameters | ---------- | scopes | Scopes to pass to `get_token` | tenant_id | Azure tenant ID. | credential | Optionally pass an instantiated Azure credential class to use (e.g. | `azure.identity.DefaultAzureCredential`). The credential class must | have a `get_token()` method. | | retrieve_credentials_impl(self) -> 'CredentialProviderFunctionReturn' | Fetch the credentials. | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __abstractmethods__ = frozenset() | | ---------------------------------------------------------------------- | Methods inherited from CachingCredentialProvider: | | __call__(self) -> 'CredentialProviderFunctionReturn' | Fetches the credentials. | | __repr__(self) -> 'str' | Return repr(self). | | clear_cached_credentials(self) -> 'None' | | ---------------------------------------------------------------------- | Data descriptors inherited from CredentialProvider: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
polars.CredentialProviderFunction
Python Library Documentation: _UnionGenericAlias in module typing
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
polars.CredentialProviderFunctionReturn
Python Library Documentation: GenericAlias in module builtins class tuple(object) | tuple(iterable=(), /) | | Built-in immutable sequence. | | If no argument is given, the constructor returns an empty tuple. | If iterable is specified the tuple is initialized from iterable's items. | | If the argument is a tuple, the return value is the same object. | | Built-in subclasses: | asyncgen_hooks | UnraisableHookArgs | | Methods defined here: | | __add__(self, value, /) | Return self+value. | | __contains__(self, key, /) | Return bool(key in self). | | __eq__(self, value, /) | Return self==value. | | __ge__(self, value, /) | Return self>=value. | | __getattribute__(self, name, /) | Return getattr(self, name). | | __getitem__(self, key, /) | Return self[key]. | | __getnewargs__(self, /) | | __gt__(self, value, /) | Return self>value. | | __hash__(self, /) | Return hash(self). | | __iter__(self, /) | Implement iter(self). | | __le__(self, value, /) | Return self<=value. | | __len__(self, /) | Return len(self). | | __lt__(self, value, /) | Return self<value. | | __mul__(self, value, /) | Return self*value. | | __ne__(self, value, /) | Return self!=value. | | __repr__(self, /) | Return repr(self). | | __rmul__(self, value, /) | Return value*self. | | count(self, value, /) | Return number of occurrences of value. | | index(self, value, start=0, stop=9223372036854775807, /) | Return first index of value. | | Raises ValueError if the value is not present. | | ---------------------------------------------------------------------- | Class methods defined here: | | __class_getitem__(...) | See PEP 585 | | ---------------------------------------------------------------------- | Static methods defined here: | | __new__(*args, **kwargs) | Create and return a new object. See help(type) for accurate signature.
polars.CredentialProviderGCP
Python Library Documentation: class CredentialProviderGCP in module polars.io.cloud.credential_provider._providers class CredentialProviderGCP(CachingCredentialProvider) | CredentialProviderGCP(*, scopes: 'Any | None' = None, request: 'Any | None' = None, quota_project_id: 'Any | None' = None, default_scopes: 'Any | None' = None) -> 'None' | | GCP Credential Provider. | | Using this requires the `google-auth` Python package to be installed. | | .. warning:: | This functionality is considered **unstable**. It may be changed | at any point without it being considered a breaking change. | | Method resolution order: | CredentialProviderGCP | CachingCredentialProvider | CredentialProvider | abc.ABC | builtins.object | | Methods defined here: | | __init__(self, *, scopes: 'Any | None' = None, request: 'Any | None' = None, quota_project_id: 'Any | None' = None, default_scopes: 'Any | None' = None) -> 'None' | Initialize a credential provider for Google Cloud (GCP). | | Parameters | ---------- | Parameters are passed to `google.auth.default()` | | retrieve_credentials_impl(self) -> 'CredentialProviderFunctionReturn' | Fetch the credentials. | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __abstractmethods__ = frozenset() | | ---------------------------------------------------------------------- | Methods inherited from CachingCredentialProvider: | | __call__(self) -> 'CredentialProviderFunctionReturn' | Fetches the credentials. | | __repr__(self) -> 'str' | Return repr(self). | | clear_cached_credentials(self) -> 'None' | | ---------------------------------------------------------------------- | Data descriptors inherited from CredentialProvider: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
polars.DataFrame
Python Library Documentation: class DataFrame in module polars.dataframe.frame
class DataFrame(builtins.object)
| DataFrame(data: 'FrameInitTypes | None' = None, schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100, nan_to_null: 'bool' = False, height: 'int | None' = None) -> 'None'
|
| Two-dimensional data structure representing data as a table with rows and columns.
|
| Parameters
| ----------
| data : dict, Sequence, ndarray, Series, or pandas.DataFrame
| Two-dimensional data in various forms; dict input must contain Sequences,
| Generators, or a `range`. Sequence may contain Series or other Sequences.
| schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
| The schema of the resulting DataFrame. The schema may be declared in several
| ways:
|
| * As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
| * As a list of column names; in this case types are automatically inferred.
| * As a list of (name,type) pairs; this is equivalent to the dictionary form.
|
| The order of the schema determines the column order of the frame.
| When passing a dict, its insertion order is respected. To override specific
| column data types by name without changing column order, use
| ``schema_overrides`` instead.
|
| If you supply a list of column names that does not match the names in the
| underlying data, the names given here will overwrite them. The number
| of names given in the schema should match the underlying data dimensions.
|
| If set to `None` (default), the schema is inferred from the data.
| schema_overrides : dict, default None
| Support type specification or override of one or more columns; note that
| any dtypes inferred from the schema param will be overridden.
|
| The number of entries in the schema should match the underlying data
| dimensions, unless a sequence of dictionaries is being passed, in which case
| a *partial* schema can be declared to prevent specific fields from being loaded.
| strict : bool, default True
| Throw an error if any `data` value does not exactly match the given or inferred
| data type for that column. If set to `False`, values that do not match the data
| type are cast to that data type or, if casting is not possible, set to null
| instead.
| orient : {'col', 'row'}, default None
| Whether to interpret two-dimensional data as columns or as rows. If None,
| the orientation is inferred by matching the columns and data dimensions. If
| this does not yield conclusive results, column orientation is used.
| infer_schema_length : int or None
| The maximum number of rows to scan for schema inference. If set to `None`, the
| full data may be scanned *(this can be slow)*. This parameter only applies if
| the input data is a sequence or generator of rows; other input is read as-is.
| nan_to_null : bool, default False
| If the data comes from one or more numpy arrays, can optionally convert input
| data np.nan values to null instead. This is a no-op for all other input data.
| height : int or None, default None
| Allows constructing DataFrames with 0 width and a specified height. If
| passed with data, ensures the resulting DataFrame has this height.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Notes
| -----
| Polars explicitly does not support subclassing of its core data types. See
| the following GitHub issue for possible workarounds:
| https://github.com/pola-rs/polars/issues/2846#issuecomment-1711799869
|
| Examples
| --------
| Constructing a DataFrame from a dictionary:
|
| >>> data = {"a": [1, 2], "b": [3, 4]}
| >>> df = pl.DataFrame(data)
| >>> df
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
|
| Notice that the dtypes are automatically inferred as polars Int64:
|
| >>> df.dtypes
| [Int64, Int64]
|
| To specify a more detailed/specific frame schema you can supply the `schema`
| parameter with a dictionary of (name,dtype) pairs...
|
| >>> data = {"col1": [0, 2], "col2": [3, 7]}
| >>> df2 = pl.DataFrame(data, schema={"col1": pl.Float32, "col2": pl.Int64})
| >>> df2
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 0.0 ┆ 3 │
| │ 2.0 ┆ 7 │
| └──────┴──────┘
|
| ...a sequence of (name,dtype) pairs...
|
| >>> data = {"col1": [1, 2], "col2": [3, 4]}
| >>> df3 = pl.DataFrame(data, schema=[("col1", pl.Float32), ("col2", pl.Int64)])
| >>> df3
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 1.0 ┆ 3 │
| │ 2.0 ┆ 4 │
| └──────┴──────┘
|
| ...or a list of typed Series.
|
| >>> data = [
| ... pl.Series("col1", [1, 2], dtype=pl.Float32),
| ... pl.Series("col2", [3, 4], dtype=pl.Int64),
| ... ]
| >>> df4 = pl.DataFrame(data)
| >>> df4
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 1.0 ┆ 3 │
| │ 2.0 ┆ 4 │
| └──────┴──────┘
|
| Constructing a DataFrame from a numpy ndarray, specifying column names:
|
| >>> import numpy as np
| >>> data = np.array([(1, 2), (3, 4)], dtype=np.int64)
| >>> df5 = pl.DataFrame(data, schema=["a", "b"], orient="col")
| >>> df5
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
|
| Constructing a DataFrame from a list of lists, row orientation specified:
|
| >>> data = [[1, 2, 3], [4, 5, 6]]
| >>> df6 = pl.DataFrame(data, schema=["a", "b", "c"], orient="row")
| >>> df6
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| Methods defined here:
|
| __add__(self, other: 'DataFrame | Series | int | float | bool | str') -> 'DataFrame'
|
| __array__(self, dtype: 'np.dtype[Any] | None' = None, copy: 'bool | None' = None) -> 'np.ndarray[Any, Any]'
| Return a NumPy ndarray with the given data type.
|
| This method ensures a Polars DataFrame can be treated as a NumPy ndarray.
| It enables `np.asarray` and NumPy universal functions.
|
| See the NumPy documentation for more information:
| https://numpy.org/doc/stable/user/basics.interoperability.html#the-array-method
|
| __arrow_c_stream__(self, requested_schema: 'object | None' = None) -> 'object'
| Export a DataFrame via the Arrow PyCapsule Interface.
|
| https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html
|
| __bool__(self) -> 'NoReturn'
|
| __contains__(self, key: 'str') -> 'bool'
|
| __copy__(self) -> 'DataFrame'
|
| __dataframe__(self, nan_as_null: 'bool' = False, allow_copy: 'bool' = True) -> 'PolarsDataFrame'
| Convert to a dataframe object implementing the dataframe interchange protocol.
|
| .. deprecated:: 1.40.0
| Support for the Dataframe Interchange Protocol is deprecated.
|
| Parameters
| ----------
| nan_as_null
| Overwrite null values in the data with `NaN`.
|
| .. warning::
| This functionality has not been implemented and the parameter will be
| removed in a future version.
| Setting this to `True` will raise a `NotImplementedError`.
| allow_copy
| Allow memory to be copied to perform the conversion. If set to `False`,
| causes conversions that are not zero-copy to fail.
|
| Notes
| -----
| Details on the Python dataframe interchange protocol:
| https://data-apis.org/dataframe-protocol/latest/index.html
|
| Examples
| --------
| Convert a Polars DataFrame to a generic dataframe object and access some
| properties.
|
| >>> df = pl.DataFrame({"a": [1, 2], "b": [3.0, 4.0], "c": ["x", "y"]})
| >>> dfi = df.__dataframe__() # doctest: +SKIP
| >>> dfi.num_rows() # doctest: +SKIP
| 2
| >>> dfi.get_column(1).dtype # doctest: +SKIP
| (<DtypeKind.FLOAT: 2>, 64, 'g', '=')
|
| __deepcopy__(self, memo: 'None' = None) -> 'DataFrame'
|
| __eq__(self, other: 'object') -> 'DataFrame'
| Return self==value.
|
| __floordiv__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| __ge__(self, other: 'Any') -> 'DataFrame'
| Return self>=value.
|
| __getitem__(self, key: 'SingleIndexSelector | SingleColSelector | MultiColSelector | MultiIndexSelector | tuple[SingleIndexSelector, SingleColSelector] | tuple[SingleIndexSelector, MultiColSelector] | tuple[MultiIndexSelector, SingleColSelector] | tuple[MultiIndexSelector, MultiColSelector]') -> 'DataFrame | Series | Any'
| Get part of the DataFrame as a new DataFrame, Series, or scalar.
|
| Parameters
| ----------
| key
| Rows / columns to select. This is easiest to explain via example. Suppose
| we have a DataFrame with columns `'a'`, `'d'`, `'c'`, `'d'`. Here is what
| various types of `key` would do:
|
| - `df[0, 'a']` extracts the first element of column `'a'` and returns a
| scalar.
| - `df[0]` extracts the first row and returns a Dataframe.
| - `df['a']` extracts column `'a'` and returns a Series.
| - `df[0:2]` extracts the first two rows and returns a Dataframe.
| - `df[0:2, 'a']` extracts the first two rows from column `'a'` and returns
| a Series.
| - `df[0:2, 0]` extracts the first two rows from the first column and returns
| a Series.
| - `df[[0, 1], [0, 1, 2]]` extracts the first two rows and the first three
| columns and returns a Dataframe.
| - `df[0: 2, ['a', 'c']]` extracts the first two rows from columns `'a'` and
| `'c'` and returns a Dataframe.
| - `df[:, 0: 2]` extracts all rows from the first two columns and returns a
| Dataframe.
| - `df[:, 'a': 'c']` extracts all rows and all columns positioned between
| `'a'` and `'c'` *inclusive* and returns a Dataframe. In our example,
| that would extract columns `'a'`, `'d'`, and `'c'`.
|
| Returns
| -------
| DataFrame, Series, or scalar, depending on `key`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"a": [1, 2, 3], "d": [4, 5, 6], "c": [1, 3, 2], "b": [7, 8, 9]}
| ... )
| >>> df[0]
| shape: (1, 4)
| ┌─────┬─────┬─────┬─────┐
| │ a ┆ d ┆ c ┆ b │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╡
| │ 1 ┆ 4 ┆ 1 ┆ 7 │
| └─────┴─────┴─────┴─────┘
| >>> df[0, "a"]
| 1
| >>> df["a"]
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
| >>> df[0:2]
| shape: (2, 4)
| ┌─────┬─────┬─────┬─────┐
| │ a ┆ d ┆ c ┆ b │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╡
| │ 1 ┆ 4 ┆ 1 ┆ 7 │
| │ 2 ┆ 5 ┆ 3 ┆ 8 │
| └─────┴─────┴─────┴─────┘
| >>> df[0:2, "a"]
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 2
| ]
| >>> df[0:2, 0]
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 2
| ]
| >>> df[[0, 1], [0, 1, 2]]
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ d ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 4 ┆ 1 │
| │ 2 ┆ 5 ┆ 3 │
| └─────┴─────┴─────┘
| >>> df[0:2, ["a", "c"]]
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ c │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 3 │
| └─────┴─────┘
| >>> df[:, 0:2]
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ d │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| │ 3 ┆ 6 │
| └─────┴─────┘
| >>> df[:, "a":"c"]
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ d ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 4 ┆ 1 │
| │ 2 ┆ 5 ┆ 3 │
| │ 3 ┆ 6 ┆ 2 │
| └─────┴─────┴─────┘
|
| __getstate__(self) -> 'bytes'
| Helper for pickle.
|
| __gt__(self, other: 'Any') -> 'DataFrame'
| Return self>value.
|
| __init__(self, data: 'FrameInitTypes | None' = None, schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100, nan_to_null: 'bool' = False, height: 'int | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> 'Iterator[Series]'
|
| __le__(self, other: 'Any') -> 'DataFrame'
| Return self<=value.
|
| __len__(self) -> 'int'
|
| __lt__(self, other: 'Any') -> 'DataFrame'
| Return self<value.
|
| __mod__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| __mul__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| __ne__(self, other: 'object') -> 'DataFrame'
| Return self!=value.
|
| __radd__(self, other: 'DataFrame | Series | int | float | bool | str') -> 'DataFrame'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __reversed__(self) -> 'Iterator[Series]'
|
| __rmul__(self, other: 'int | float') -> 'DataFrame'
|
| __setitem__(self, key: 'str | Sequence[str] | tuple[Any, str | int]', value: 'Any') -> 'None'
| Modify DataFrame elements in place, using assignment syntax.
|
| Parameters
| ----------
| key : str | Sequence[int] | Sequence[str] | tuple[Any, str | int]
| Specifies the location(s) within the DataFrame to assign new values.
| The behavior varies based on the type of `key`:
|
| - Str: `df["a"] = value`:
| Not supported. Raises a `TypeError`. Use `df.with_columns(...)`
| to add or modify columns.
|
| - Sequence[str]: `df[["a", "b"]] = value`:
| Assigns multiple columns at once. `value` must be a 2D array-like
| structure with the same number of columns as the list
| of column names provided.
|
| - tuple[Any, str | int]: `df[row_idx, "a"] = value`:
| Assigns a new value to a specific element in the DataFrame, where
| `row_idx` specifies the row and `"a"` specifies the column.
|
| - `df[row_idx, col_idx] = value`:
| Similar to the above, but `col_idx` is the integer index of the column.
|
| value : Any
| The new value(s) to assign. The expected structure of `value` depends on the
| form of `key`:
|
| - For multiple column assignment (`df[["a", "b"]] = value`), `value` should
| be a 2D array-like object with shape (n_rows, n_columns).
|
| - For single element assignment (`df[row_idx, "a"] = value`), `value` should
| be a scalar.
|
| Raises
| ------
| TypeError
| If an unsupported assignment is attempted, such as assigning a Series
| directly to a column using `df["a"] = series`.
|
| ValueError
| If the shape of `value` does not match the expected shape based on `key`.
|
| Examples
| --------
| Sequence[str] : `df[["a", "b"]] = value`:
|
| >>> import numpy as np
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
| >>> df[["a", "b"]] = np.array([[10, 40], [20, 50], [30, 60]])
| >>> df
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 10 ┆ 40 │
| │ 20 ┆ 50 │
| │ 30 ┆ 60 │
| └─────┴─────┘
|
| tuple[Any, str | int] : `df[row_idx, "a"] = value`:
|
| >>> df[1, "a"] = 100
| >>> df
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 10 ┆ 40 │
| │ 100 ┆ 50 │
| │ 30 ┆ 60 │
| └─────┴─────┘
|
| `df[row_idx, col_idx] = value`:
|
| >>> df[0, 1] = 30
| >>> df
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 10 ┆ 30 │
| │ 100 ┆ 50 │
| │ 30 ┆ 60 │
| └─────┴─────┘
|
| __setstate__(self, state: 'bytes') -> 'None'
|
| __str__(self) -> 'str'
| Return str(self).
|
| __sub__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| __truediv__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| approx_n_unique(self) -> 'DataFrame'
| Approximate count of unique values.
|
| .. deprecated:: 0.20.11
| Use the `select(pl.all().approx_n_unique())` method instead.
|
| This is done using the HyperLogLog++ algorithm for cardinality estimation.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> df.approx_n_unique() # doctest: +SKIP
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| bottom_k(self, k: 'int', *, by: 'IntoExpr | Iterable[IntoExpr]', reverse: 'bool | Sequence[bool]' = False) -> 'DataFrame'
| Return the `k` smallest rows.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| k
| Number of rows to return.
| by
| Column(s) used to determine the bottom rows.
| Accepts expression input. Strings are parsed as column names.
| reverse
| Consider the `k` largest elements of the `by` column(s) (instead of the `k`
| smallest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [2, 1, 1, 3, 2, 1],
| ... }
| ... )
|
| Get the rows which contain the 4 smallest values in column b.
|
| >>> df.bottom_k(4, by="b")
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 1 │
| │ a ┆ 1 │
| │ c ┆ 1 │
| │ a ┆ 2 │
| └─────┴─────┘
|
| Get the rows which contain the 4 smallest values when sorting on column a and b.
|
| >>> df.bottom_k(4, by=["a", "b"])
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ a ┆ 2 │
| │ b ┆ 1 │
| │ b ┆ 2 │
| └─────┴─────┘
|
| cast(self, dtypes: 'Mapping[ColumnNameOrSelector | PolarsDataType, PolarsDataType | PythonDataType] | PolarsDataType | Schema', *, strict: 'bool' = True) -> 'DataFrame'
| Cast DataFrame column(s) to the specified dtype(s).
|
| Parameters
| ----------
| dtypes
| Mapping of column names (or selector) to dtypes, or a single dtype
| to which all columns will be cast.
| strict
| Raise if cast is invalid on rows after predicates are pushed down.
| If `False`, invalid casts will produce null values.
|
| Examples
| --------
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": [date(2020, 1, 2), date(2021, 3, 4), date(2022, 5, 6)],
| ... }
| ... )
|
| Cast specific frame columns to the specified dtypes:
|
| >>> df.cast({"foo": pl.Float32, "bar": pl.UInt8})
| shape: (3, 3)
| ┌─────┬─────┬────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f32 ┆ u8 ┆ date │
| ╞═════╪═════╪════════════╡
| │ 1.0 ┆ 6 ┆ 2020-01-02 │
| │ 2.0 ┆ 7 ┆ 2021-03-04 │
| │ 3.0 ┆ 8 ┆ 2022-05-06 │
| └─────┴─────┴────────────┘
|
| Cast all frame columns matching one dtype (or dtype group) to another dtype:
|
| >>> df.cast({pl.Date: pl.Datetime})
| shape: (3, 3)
| ┌─────┬─────┬─────────────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ datetime[μs] │
| ╞═════╪═════╪═════════════════════╡
| │ 1 ┆ 6.0 ┆ 2020-01-02 00:00:00 │
| │ 2 ┆ 7.0 ┆ 2021-03-04 00:00:00 │
| │ 3 ┆ 8.0 ┆ 2022-05-06 00:00:00 │
| └─────┴─────┴─────────────────────┘
|
| Use selectors to define the columns being cast:
|
| >>> import polars.selectors as cs
| >>> df.cast({cs.numeric(): pl.UInt32, cs.temporal(): pl.String})
| shape: (3, 3)
| ┌─────┬─────┬────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ str │
| ╞═════╪═════╪════════════╡
| │ 1 ┆ 6 ┆ 2020-01-02 │
| │ 2 ┆ 7 ┆ 2021-03-04 │
| │ 3 ┆ 8 ┆ 2022-05-06 │
| └─────┴─────┴────────────┘
|
| Cast all frame columns to the specified dtype:
|
| >>> df.cast(pl.String).to_dict(as_series=False)
| {'foo': ['1', '2', '3'],
| 'bar': ['6.0', '7.0', '8.0'],
| 'ham': ['2020-01-02', '2021-03-04', '2022-05-06']}
|
| clear(self, n: 'int' = 0) -> 'DataFrame'
| Create an empty (n=0) or `n`-row null-filled (n>0) copy of the DataFrame.
|
| Returns a `n`-row null-filled DataFrame with an identical schema.
| `n` can be greater than the current number of rows in the DataFrame.
|
| Parameters
| ----------
| n
| Number of (null-filled) rows to return in the cleared frame.
|
| See Also
| --------
| clone : Cheap deepcopy/clone.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> df.clear()
| shape: (0, 3)
| ┌─────┬─────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞═════╪═════╪══════╡
| └─────┴─────┴──────┘
|
| >>> df.clear(n=2)
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ null ┆ null ┆ null │
| └──────┴──────┴──────┘
|
| clone(self) -> 'DataFrame'
| Create a copy of this DataFrame.
|
| This is a cheap operation that does not copy data.
|
| See Also
| --------
| clear : Create an empty copy of the current DataFrame, with identical
| schema but no data.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> df.clone()
| shape: (4, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true │
| │ 2 ┆ 4.0 ┆ true │
| │ 3 ┆ 10.0 ┆ false │
| │ 4 ┆ 13.0 ┆ true │
| └─────┴──────┴───────┘
|
| collect_schema(self) -> 'Schema'
| Get an ordered mapping of column names to their data type.
|
| This is an alias for the :attr:`schema` property.
|
| See Also
| --------
| schema
|
| Notes
| -----
| This method is included to facilitate writing code that is generic for both
| DataFrame and LazyFrame.
|
| Examples
| --------
| Determine the schema.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.collect_schema()
| Schema({'foo': Int64, 'bar': Float64, 'ham': String})
|
| Access various properties of the schema using the :class:`Schema` object.
|
| >>> schema = df.collect_schema()
| >>> schema["bar"]
| Float64
| >>> schema.names()
| ['foo', 'bar', 'ham']
| >>> schema.dtypes()
| [Int64, Float64, String]
| >>> schema.len()
| 3
|
| corr(self, *, label: 'str | None' = None, **kwargs: 'Any') -> 'DataFrame'
| Return pairwise Pearson product-moment correlation coefficients between columns.
|
| See numpy `corrcoef` for more information:
| https://numpy.org/doc/stable/reference/generated/numpy.corrcoef.html
|
| Parameters
| ----------
| label
| If given, a new column that contains the labels (column names)
| associated with each row is added, with this name.
| **kwargs
| Keyword arguments that are passed to `numpy.corrcoef`.
|
| Notes
| -----
| This functionality requires `numpy` to be installed.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [3, 2, 1], "ham": [7, 8, 9]})
| >>> df.corr()
| shape: (3, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪══════╡
| │ 1.0 ┆ -1.0 ┆ 1.0 │
| │ -1.0 ┆ 1.0 ┆ -1.0 │
| │ 1.0 ┆ -1.0 ┆ 1.0 │
| └──────┴──────┴──────┘
| >>> df.corr(label="cols")
| shape: (3, 4)
| ┌──────┬──────┬──────┬──────┐
| │ cols ┆ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪══════╪══════╡
| │ foo ┆ 1.0 ┆ -1.0 ┆ 1.0 │
| │ bar ┆ -1.0 ┆ 1.0 ┆ -1.0 │
| │ ham ┆ 1.0 ┆ -1.0 ┆ 1.0 │
| └──────┴──────┴──────┴──────┘
|
| count(self) -> 'DataFrame'
| Return the number of non-null elements for each column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"a": [1, 2, 3, 4], "b": [1, 2, 1, None], "c": [None, None, None, None]}
| ... )
| >>> df.count()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 4 ┆ 3 ┆ 0 │
| └─────┴─────┴─────┘
|
| describe(self, percentiles: 'Sequence[float] | float | None' = (0.25, 0.5, 0.75), *, interpolation: 'QuantileMethod' = 'nearest') -> 'DataFrame'
| Summary statistics for a DataFrame.
|
| Parameters
| ----------
| percentiles
| One or more percentiles to include in the summary statistics.
| All values must be in the range `[0, 1]`.
|
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method used when calculating percentiles.
|
| Notes
| -----
| The median is included by default as the 50% percentile.
|
| Warnings
| --------
| We do not guarantee the output of `describe` to be stable. It will show
| statistics that we deem informative, and may be updated in the future.
| Using `describe` programmatically (versus interactive exploration) is
| not recommended for this reason.
|
| See Also
| --------
| glimpse
|
| Examples
| --------
| >>> from datetime import date, time
| >>> df = pl.DataFrame(
| ... {
| ... "float": [1.0, 2.8, 3.0],
| ... "int": [40, 50, None],
| ... "bool": [True, False, True],
| ... "str": ["zz", "xx", "yy"],
| ... "date": [date(2020, 1, 1), date(2021, 7, 5), date(2022, 12, 31)],
| ... "time": [time(10, 20, 30), time(14, 45, 50), time(23, 15, 10)],
| ... }
| ... )
|
| Show default frame statistics:
|
| >>> df.describe()
| shape: (9, 7)
| ┌────────────┬──────────┬──────────┬──────────┬──────┬─────────────────────┬──────────┐
| │ statistic ┆ float ┆ int ┆ bool ┆ str ┆ date ┆ time │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞════════════╪══════════╪══════════╪══════════╪══════╪═════════════════════╪══════════╡
| │ count ┆ 3.0 ┆ 2.0 ┆ 3.0 ┆ 3 ┆ 3 ┆ 3 │
| │ null_count ┆ 0.0 ┆ 1.0 ┆ 0.0 ┆ 0 ┆ 0 ┆ 0 │
| │ mean ┆ 2.266667 ┆ 45.0 ┆ 0.666667 ┆ null ┆ 2021-07-02 16:00:00 ┆ 16:07:10 │
| │ std ┆ 1.101514 ┆ 7.071068 ┆ null ┆ null ┆ null ┆ null │
| │ min ┆ 1.0 ┆ 40.0 ┆ 0.0 ┆ xx ┆ 2020-01-01 ┆ 10:20:30 │
| │ 25% ┆ 2.8 ┆ 40.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 50% ┆ 2.8 ┆ 50.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 75% ┆ 3.0 ┆ 50.0 ┆ null ┆ null ┆ 2022-12-31 ┆ 23:15:10 │
| │ max ┆ 3.0 ┆ 50.0 ┆ 1.0 ┆ zz ┆ 2022-12-31 ┆ 23:15:10 │
| └────────────┴──────────┴──────────┴──────────┴──────┴─────────────────────┴──────────┘
|
| Customize which percentiles are displayed, applying linear interpolation:
|
| >>> with pl.Config(tbl_rows=12):
| ... df.describe(
| ... percentiles=[0.1, 0.3, 0.5, 0.7, 0.9],
| ... interpolation="linear",
| ... )
| shape: (11, 7)
| ┌────────────┬──────────┬──────────┬──────────┬──────┬─────────────────────┬──────────┐
| │ statistic ┆ float ┆ int ┆ bool ┆ str ┆ date ┆ time │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞════════════╪══════════╪══════════╪══════════╪══════╪═════════════════════╪══════════╡
| │ count ┆ 3.0 ┆ 2.0 ┆ 3.0 ┆ 3 ┆ 3 ┆ 3 │
| │ null_count ┆ 0.0 ┆ 1.0 ┆ 0.0 ┆ 0 ┆ 0 ┆ 0 │
| │ mean ┆ 2.266667 ┆ 45.0 ┆ 0.666667 ┆ null ┆ 2021-07-02 16:00:00 ┆ 16:07:10 │
| │ std ┆ 1.101514 ┆ 7.071068 ┆ null ┆ null ┆ null ┆ null │
| │ min ┆ 1.0 ┆ 40.0 ┆ 0.0 ┆ xx ┆ 2020-01-01 ┆ 10:20:30 │
| │ 10% ┆ 1.36 ┆ 41.0 ┆ null ┆ null ┆ 2020-04-20 ┆ 11:13:34 │
| │ 30% ┆ 2.08 ┆ 43.0 ┆ null ┆ null ┆ 2020-11-26 ┆ 12:59:42 │
| │ 50% ┆ 2.8 ┆ 45.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 70% ┆ 2.88 ┆ 47.0 ┆ null ┆ null ┆ 2022-02-07 ┆ 18:09:34 │
| │ 90% ┆ 2.96 ┆ 49.0 ┆ null ┆ null ┆ 2022-09-13 ┆ 21:33:18 │
| │ max ┆ 3.0 ┆ 50.0 ┆ 1.0 ┆ zz ┆ 2022-12-31 ┆ 23:15:10 │
| └────────────┴──────────┴──────────┴──────────┴──────┴─────────────────────┴──────────┘
|
| drop(self, *columns: 'ColumnNameOrSelector | Iterable[ColumnNameOrSelector]', strict: 'bool' = True) -> 'DataFrame'
| Remove columns from the dataframe.
|
| Parameters
| ----------
| *columns
| Names of the columns that should be removed from the dataframe.
| Accepts column selector input.
| strict
| Validate that all column names exist in the current schema,
| and throw an exception if any do not.
|
| Examples
| --------
| Drop a single column by passing the name of that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.drop("ham")
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 6.0 │
| │ 2 ┆ 7.0 │
| │ 3 ┆ 8.0 │
| └─────┴─────┘
|
| Drop multiple columns by passing a list of column names.
|
| >>> df.drop(["bar", "ham"])
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Drop multiple columns by passing a selector.
|
| >>> import polars.selectors as cs
| >>> df.drop(cs.numeric())
| shape: (3, 1)
| ┌─────┐
| │ ham │
| │ --- │
| │ str │
| ╞═════╡
| │ a │
| │ b │
| │ c │
| └─────┘
|
| Use positional arguments to drop multiple columns.
|
| >>> df.drop("foo", "ham")
| shape: (3, 1)
| ┌─────┐
| │ bar │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 6.0 │
| │ 7.0 │
| │ 8.0 │
| └─────┘
|
| drop_in_place(self, name: 'str') -> 'Series'
| Drop a single column in-place and return the dropped column.
|
| Parameters
| ----------
| name
| Name of the column to drop.
|
| Returns
| -------
| Series
| The dropped column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.drop_in_place("ham")
| shape: (3,)
| Series: 'ham' [str]
| [
| "a"
| "b"
| "c"
| ]
|
| drop_nans(self, subset: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None) -> 'DataFrame'
| Drop all rows that contain one or more NaN values.
|
| The original order of the remaining rows is preserved.
|
| Parameters
| ----------
| subset
| Column name(s) for which NaN values are considered; if set to `None`
| (default), use all columns (note that only floating-point columns
| can contain NaNs).
|
| See Also
| --------
| drop_nulls
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To drop null values, use :func:`drop_nulls`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [-20.5, float("nan"), 80.0],
| ... "bar": [float("nan"), 110.0, 25.5],
| ... "ham": ["xxx", "yyy", None],
| ... }
| ... )
|
| The default behavior of this method is to drop rows where any single
| value in the row is NaN:
|
| >>> df.drop_nans()
| shape: (1, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════╪══════╪══════╡
| │ 80.0 ┆ 25.5 ┆ null │
| └──────┴──────┴──────┘
|
| This behaviour can be constrained to consider only a subset of columns, as
| defined by name, or with a selector. For example, dropping rows only if
| there is a NaN in the "bar" column:
|
| >>> df.drop_nans(subset=["bar"])
| shape: (2, 3)
| ┌──────┬───────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════╪═══════╪══════╡
| │ NaN ┆ 110.0 ┆ yyy │
| │ 80.0 ┆ 25.5 ┆ null │
| └──────┴───────┴──────┘
|
| Dropping a row only if *all* values are NaN requires a different formulation:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [float("nan"), float("nan"), float("nan"), float("nan")],
| ... "b": [10.0, 2.5, float("nan"), 5.25],
| ... "c": [65.75, float("nan"), float("nan"), 10.5],
| ... }
| ... )
| >>> df.filter(~pl.all_horizontal(pl.all().is_nan()))
| shape: (3, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞═════╪══════╪═══════╡
| │ NaN ┆ 10.0 ┆ 65.75 │
| │ NaN ┆ 2.5 ┆ NaN │
| │ NaN ┆ 5.25 ┆ 10.5 │
| └─────┴──────┴───────┘
|
| drop_nulls(self, subset: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None) -> 'DataFrame'
| Drop all rows that contain one or more null values.
|
| The original order of the remaining rows is preserved.
|
| Parameters
| ----------
| subset
| Column name(s) for which null values are considered.
| If set to `None` (default), use all columns.
|
| See Also
| --------
| drop_nans
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To drop NaN values, use :func:`drop_nans`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, None, 8],
| ... "ham": ["a", "b", None],
| ... }
| ... )
|
| The default behavior of this method is to drop rows where any single
| value of the row is null.
|
| >>> df.drop_nulls()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| This behaviour can be constrained to consider only a subset of columns, as
| defined by name or with a selector. For example, dropping rows if there is
| a null in any of the integer columns:
|
| >>> import polars.selectors as cs
| >>> df.drop_nulls(subset=cs.integer())
| shape: (2, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 6 ┆ a │
| │ 3 ┆ 8 ┆ null │
| └─────┴─────┴──────┘
|
| Below are some additional examples that show how to drop null
| values based on other conditions.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, None, None, None],
| ... "b": [1, 2, None, 1],
| ... "c": [1, None, None, 1],
| ... }
| ... )
| >>> df
| shape: (4, 3)
| ┌──────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ null ┆ i64 ┆ i64 │
| ╞══════╪══════╪══════╡
| │ null ┆ 1 ┆ 1 │
| │ null ┆ 2 ┆ null │
| │ null ┆ null ┆ null │
| │ null ┆ 1 ┆ 1 │
| └──────┴──────┴──────┘
|
| Drop a row only if all values are null:
|
| >>> df.filter(~pl.all_horizontal(pl.all().is_null()))
| shape: (3, 3)
| ┌──────┬─────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ null ┆ i64 ┆ i64 │
| ╞══════╪═════╪══════╡
| │ null ┆ 1 ┆ 1 │
| │ null ┆ 2 ┆ null │
| │ null ┆ 1 ┆ 1 │
| └──────┴─────┴──────┘
|
| Drop a column if all values are null:
|
| >>> df[[s.name for s in df if not (s.null_count() == df.height)]]
| shape: (4, 2)
| ┌──────┬──────┐
| │ b ┆ c │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ 1 ┆ 1 │
| │ 2 ┆ null │
| │ null ┆ null │
| │ 1 ┆ 1 │
| └──────┴──────┘
|
| equals(self, other: 'DataFrame', *, null_equal: 'bool' = True) -> 'bool'
| Check whether the DataFrame is equal to another DataFrame.
|
| Parameters
| ----------
| other
| DataFrame to compare with.
| null_equal
| Consider null values as equal.
|
| See Also
| --------
| polars.testing.assert_frame_equal
|
| Examples
| --------
| >>> df1 = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df2 = pl.DataFrame(
| ... {
| ... "foo": [3, 2, 1],
| ... "bar": [8.0, 7.0, 6.0],
| ... "ham": ["c", "b", "a"],
| ... }
| ... )
| >>> df1.equals(df1)
| True
| >>> df1.equals(df2)
| False
|
| estimated_size(self, unit: 'SizeUnit' = 'b') -> 'int | float'
| Return an estimation of the total (heap) allocated size of the `DataFrame`.
|
| Estimated size is given in the specified unit (bytes by default).
|
| This estimation is the sum of the size of its buffers, validity, including
| nested arrays. Multiple arrays may share buffers and bitmaps. Therefore, the
| size of 2 arrays is not the sum of the sizes computed from this function. In
| particular, [`StructArray`]'s size is an upper bound.
|
| When an array is sliced, its allocated size remains constant because the buffer
| unchanged. However, this function will yield a smaller number. This is because
| this function returns the visible size of the buffer, not its total capacity.
|
| FFI buffers are included in this estimation.
|
| Notes
| -----
| For data with Object dtype, the estimated size only reports the pointer
| size, which is a huge underestimation.
|
| Parameters
| ----------
| unit : {'b', 'kb', 'mb', 'gb', 'tb'}
| Scale the returned size to the given unit.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "x": list(reversed(range(1_000_000))),
| ... "y": [v / 1000 for v in range(1_000_000)],
| ... "z": [str(v) for v in range(1_000_000)],
| ... },
| ... schema=[("x", pl.UInt32), ("y", pl.Float64), ("z", pl.String)],
| ... )
| >>> df.estimated_size()
| 17888890
| >>> df.estimated_size("mb")
| 17.0601749420166
|
| explode(self, columns: 'ColumnNameOrSelector | Iterable[ColumnNameOrSelector]', *more_columns: 'ColumnNameOrSelector', empty_as_null: 'bool' = <object object at 0x00000280D5528F20>, keep_nulls: 'bool' = True) -> 'DataFrame'
| Explode the dataframe to long format by exploding the given columns.
|
| Parameters
| ----------
| columns
| Column names, expressions, or a selector defining them. The underlying
| columns being exploded must be of the `List` or `Array` data type.
| *more_columns
| Additional names of columns to explode, specified as positional arguments.
| empty_as_null
| Explode an empty list/array into a `null`.
| keep_nulls
| Explode a `null` list/array into a `null`.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "letters": ["a", "a", "b", "c"],
| ... "numbers": [[1], [2, 3], [4, 5], [6, 7, 8]],
| ... }
| ... )
| >>> df
| shape: (4, 2)
| ┌─────────┬───────────┐
| │ letters ┆ numbers │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═════════╪═══════════╡
| │ a ┆ [1] │
| │ a ┆ [2, 3] │
| │ b ┆ [4, 5] │
| │ c ┆ [6, 7, 8] │
| └─────────┴───────────┘
| >>> df.explode("numbers", empty_as_null=False)
| shape: (8, 2)
| ┌─────────┬─────────┐
| │ letters ┆ numbers │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════════╡
| │ a ┆ 1 │
| │ a ┆ 2 │
| │ a ┆ 3 │
| │ b ┆ 4 │
| │ b ┆ 5 │
| │ c ┆ 6 │
| │ c ┆ 7 │
| │ c ┆ 8 │
| └─────────┴─────────┘
|
| extend(self, other: 'DataFrame') -> 'DataFrame'
| Extend the memory backed by this `DataFrame` with the values from `other`.
|
| Different from `vstack` which adds the chunks from `other` to the chunks of
| this `DataFrame`, `extend` appends the data from `other` to the underlying
| memory locations and thus may cause a reallocation.
|
| If this does not cause a reallocation, the resulting data structure will not
| have any extra chunks and thus will yield faster queries.
|
| Prefer `extend` over `vstack` when you want to do a query after a single
| append. For instance, during online operations where you add `n` rows and rerun
| a query.
|
| Prefer `vstack` over `extend` when you want to append many times before
| doing a query. For instance, when you read in multiple files and want to store
| them in a single `DataFrame`. In the latter case, finish the sequence of
| `vstack` operations with a `rechunk`.
|
| Parameters
| ----------
| other
| DataFrame to vertically add.
|
| Warnings
| --------
| This method modifies the dataframe in-place. The dataframe is returned for
| convenience only.
|
| See Also
| --------
| vstack
|
| Examples
| --------
| >>> df1 = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df2 = pl.DataFrame({"foo": [10, 20, 30], "bar": [40, 50, 60]})
| >>> df1.extend(df2)
| shape: (6, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| │ 3 ┆ 6 │
| │ 10 ┆ 40 │
| │ 20 ┆ 50 │
| │ 30 ┆ 60 │
| └─────┴─────┘
|
| fill_nan(self, value: 'Expr | int | float | None') -> 'DataFrame'
| Fill floating point NaN values by an Expression evaluation.
|
| Parameters
| ----------
| value
| Value used to fill NaN values.
|
| Returns
| -------
| DataFrame
| DataFrame with NaN values replaced by the given value.
|
| See Also
| --------
| fill_null
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To fill null values, use :func:`fill_null`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1.5, 2, float("nan"), 4],
| ... "b": [0.5, 4, float("nan"), 13],
| ... }
| ... )
| >>> df.fill_nan(99)
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪══════╡
| │ 1.5 ┆ 0.5 │
| │ 2.0 ┆ 4.0 │
| │ 99.0 ┆ 99.0 │
| │ 4.0 ┆ 13.0 │
| └──────┴──────┘
|
| fill_null(self, value: 'Any | Expr | None' = None, strategy: 'FillNullStrategy | None' = None, limit: 'int | None' = None, *, matches_supertype: 'bool' = True) -> 'DataFrame'
| Fill null values using the specified value or strategy.
|
| Parameters
| ----------
| value
| Value used to fill null values.
| strategy : {None, 'forward', 'backward', 'min', 'max', 'mean', 'zero', 'one'}
| Strategy used to fill null values.
| limit
| Number of consecutive null values to fill when using the 'forward' or
| 'backward' strategy.
| matches_supertype
| Fill all matching supertype of the fill `value`.
|
| Returns
| -------
| DataFrame
| DataFrame with None values replaced by the filling strategy.
|
| See Also
| --------
| fill_nan
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To fill NaN values, use :func:`fill_nan`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 4],
| ... "b": [0.5, 4, None, 13],
| ... }
| ... )
| >>> df.fill_null(99)
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 99 ┆ 99.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
| >>> df.fill_null(strategy="forward")
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 2 ┆ 4.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| >>> df.fill_null(strategy="max")
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 4 ┆ 13.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| >>> df.fill_null(strategy="zero")
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 0 ┆ 0.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| filter(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]', **constraints: 'Any') -> 'DataFrame'
| Filter rows, retaining those that match the given predicate expression(s).
|
| The original order of the remaining rows is preserved.
|
| Only rows where the predicate resolves as True are retained; when the
| predicate result is False (or null), the row is discarded.
|
| Parameters
| ----------
| predicates
| Expression(s) that evaluate to a boolean Series.
| constraints
| Column filters; use `name = value` to filter columns by the supplied value.
| Each constraint will behave the same as `pl.col(name).eq(value)`, and
| be implicitly joined with the other filter conditions using `&`.
|
| Notes
| -----
| If you are transitioning from Pandas, and performing filter operations based on
| the comparison of two or more columns, please note that in Polars any comparison
| involving `null` values will result in a `null` result, *not* boolean True or
| False. As a result, these rows will not be retained. Ensure that null values
| are handled appropriately to avoid unexpected behaviour (see examples below).
|
| See Also
| --------
| remove
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, None, 4, None, 0],
| ... "bar": [6, 7, 8, None, None, 9, 0],
| ... "ham": ["a", "b", "c", None, "d", "e", "f"],
| ... }
| ... )
|
| Filter rows matching a condition:
|
| >>> df.filter(pl.col("foo") > 1)
| shape: (3, 3)
| ┌─────┬──────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪══════╪═════╡
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ null ┆ d │
| └─────┴──────┴─────┘
|
| Filter on multiple conditions, combined with and/or operators:
|
| >>> df.filter(
| ... (pl.col("foo") < 3) & (pl.col("ham") == "a"),
| ... )
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| >>> df.filter(
| ... (pl.col("foo") == 1) | (pl.col("ham") == "c"),
| ... )
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Provide multiple filters using `*args` syntax:
|
| >>> df.filter(
| ... pl.col("foo") <= 2,
| ... ~pl.col("ham").is_in(["b", "c"]),
| ... )
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 0 ┆ 0 ┆ f │
| └─────┴─────┴─────┘
|
| Provide multiple filters using `**kwargs` syntax:
|
| >>> df.filter(foo=2, ham="b")
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ 7 ┆ b │
| └─────┴─────┴─────┘
|
| Filter by comparing two columns against each other:
|
| >>> df.filter(
| ... pl.col("foo") == pl.col("bar"),
| ... )
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 0 ┆ 0 ┆ f │
| └─────┴─────┴─────┘
|
| >>> df.filter(
| ... pl.col("foo") != pl.col("bar"),
| ... )
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Notice how the row with `None` values is filtered out. In order to keep the
| same behavior as pandas, use:
|
| >>> df.filter(
| ... pl.col("foo").ne_missing(pl.col("bar")),
| ... )
| shape: (5, 3)
| ┌──────┬──────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ null ┆ d │
| │ null ┆ 9 ┆ e │
| └──────┴──────┴─────┘
|
| fold(self, operation: 'Callable[[Series, Series], Series]') -> 'Series'
| Apply a horizontal reduction on a DataFrame.
|
| This can be used to effectively determine aggregations on a row level, and can
| be applied to any DataType that can be supercast (cast to a similar parent
| type).
|
| An example of the supercast rules when applying an arithmetic operation on two
| DataTypes are for instance:
|
| - Int8 + String = String
| - Float32 + Int64 = Float32
| - Float32 + Float64 = Float64
|
| Examples
| --------
| A horizontal sum operation:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [2, 1, 3],
| ... "b": [1, 2, 3],
| ... "c": [1.0, 2.0, 3.0],
| ... }
| ... )
| >>> df.fold(lambda s1, s2: s1 + s2)
| shape: (3,)
| Series: 'a' [f64]
| [
| 4.0
| 5.0
| 9.0
| ]
|
| A horizontal minimum operation:
|
| >>> df = pl.DataFrame({"a": [2, 1, 3], "b": [1, 2, 3], "c": [1.0, 2.0, 3.0]})
| >>> df.fold(lambda s1, s2: s1.zip_with(s1 < s2, s2))
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.0
| 1.0
| 3.0
| ]
|
| A horizontal string concatenation:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["foo", "bar", None],
| ... "b": [1, 2, 3],
| ... "c": [1.0, 2.0, 3.0],
| ... }
| ... )
| >>> df.fold(lambda s1, s2: s1 + s2)
| shape: (3,)
| Series: 'a' [str]
| [
| "foo11.0"
| "bar22.0"
| null
| ]
|
| A horizontal boolean or, similar to a row-wise .any():
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [False, False, True],
| ... "b": [False, True, False],
| ... }
| ... )
| >>> df.fold(lambda s1, s2: s1 | s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| true
| true
| ]
|
| Parameters
| ----------
| operation
| function that takes two `Series` and returns a `Series`.
|
| gather(self, indices: 'int | Sequence[int] | IntoExpr | Series | np.ndarray[Any, Any]', *, null_on_oob: 'bool' = False) -> 'DataFrame'
| Selects rows from this DataFrame at the given indices.
|
| .. warning::
| This functionality is experimental. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| indices
| The indices of the rows to select.
|
| null_on_oob
| If true when an index is out-of-bounds a null row will be generated
| instead of raising an error.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [2, 1, 0], "s": ["foo", "bar", "baz"]})
| >>> df.gather([2, 0, 0])
| shape: (3, 2)
| ┌─────┬─────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 0 ┆ baz │
| │ 2 ┆ foo │
| │ 2 ┆ foo │
| └─────┴─────┘
|
| >>> df.gather([0, 10, 1], null_on_oob=True)
| shape: (3, 2)
| ┌──────┬──────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞══════╪══════╡
| │ 2 ┆ foo │
| │ null ┆ null │
| │ 1 ┆ bar │
| └──────┴──────┘
|
| gather_every(self, n: 'int', offset: 'int' = 0) -> 'DataFrame'
| Take every nth row in the DataFrame and return as a new DataFrame.
|
| Parameters
| ----------
| n
| Gather every *n*-th row.
| offset
| Starting index.
|
| Examples
| --------
| >>> s = pl.DataFrame({"a": [1, 2, 3, 4], "b": [5, 6, 7, 8]})
| >>> s.gather_every(2)
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 5 │
| │ 3 ┆ 7 │
| └─────┴─────┘
|
| >>> s.gather_every(2, offset=1)
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 6 │
| │ 4 ┆ 8 │
| └─────┴─────┘
|
| get_column(self, name: 'str', *, default: 'Any | NoDefault' = <no_default>) -> 'Series | Any'
| Get a single column by name.
|
| Parameters
| ----------
| name
| String name of the column to retrieve.
| default
| Value to return if the column does not exist; if not explicitly set and
| the column is not present a `ColumnNotFoundError` exception is raised.
|
| Returns
| -------
| Series (or arbitrary default value, if specified).
|
| See Also
| --------
| to_series
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df.get_column("foo")
| shape: (3,)
| Series: 'foo' [i64]
| [
| 1
| 2
| 3
| ]
|
| Missing column handling; can optionally provide an arbitrary default value
| to the method (otherwise a `ColumnNotFoundError` exception is raised).
|
| >>> df.get_column("baz", default=pl.Series("baz", ["?", "?", "?"]))
| shape: (3,)
| Series: 'baz' [str]
| [
| "?"
| "?"
| "?"
| ]
| >>> res = df.get_column("baz", default=None)
| >>> res is None
| True
|
| get_column_index(self, name: 'str') -> 'int'
| Find the index of a column by name.
|
| Parameters
| ----------
| name
| Name of the column to find.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"foo": [1, 2, 3], "bar": [6, 7, 8], "ham": ["a", "b", "c"]}
| ... )
| >>> df.get_column_index("ham")
| 2
| >>> df.get_column_index("sandwich") # doctest: +SKIP
| ColumnNotFoundError: sandwich
|
| get_columns(self) -> 'list[Series]'
| Get the DataFrame as a List of Series.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df.get_columns()
| [shape: (3,)
| Series: 'foo' [i64]
| [
| 1
| 2
| 3
| ], shape: (3,)
| Series: 'bar' [i64]
| [
| 4
| 5
| 6
| ]]
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> df.get_columns()
| [shape: (4,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| ], shape: (4,)
| Series: 'b' [f64]
| [
| 0.5
| 4.0
| 10.0
| 13.0
| ], shape: (4,)
| Series: 'c' [bool]
| [
| true
| true
| false
| true
| ]]
|
| glimpse(self, *, max_items_per_column: 'int' = 10, max_colname_length: 'int' = 50, return_type: "Literal['frame', 'self', 'string'] | None" = None) -> 'str | DataFrame | None'
| Return a dense preview of the DataFrame.
|
| The formatting shows one line per column so that wide dataframes display
| cleanly. Each line shows the column name, the data type, and the first
| few values.
|
| .. versionchanged:: 1.35.0
| The `return_as_string` parameter was renamed `return_type` and now accepts
| string values `'string'` and `'frame'` instead of boolean True or False.
|
| Parameters
| ----------
| max_items_per_column
| Maximum number of items to show per column.
| max_colname_length
| Maximum length of the displayed column names; values that exceed
| this value are truncated with a trailing ellipsis.
| return_type
| Modify the return format:
|
| - `None` (default): Print the glimpse output to stdout, returning `None`.
| - `"self"`: Print the glimpse output to stdout, returning the *original* frame.
| - `"frame"`: Return the glimpse output as a new DataFrame.
| - `"string"`: Return the glimpse output as a string.
|
| See Also
| --------
| describe, head, tail
|
| Examples
| --------
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1.0, 2.8, 3.0],
| ... "b": [4, 5, None],
| ... "c": [True, False, True],
| ... "d": [None, "b", "c"],
| ... "e": ["usd", "eur", None],
| ... "f": [date(2020, 1, 1), date(2021, 1, 2), date(2022, 1, 1)],
| ... }
| ... )
|
| Print glimpse-formatted output to stdout, returning `None`:
|
| >>> res = df.glimpse()
| Rows: 3
| Columns: 6
| $ a <f64> 1.0, 2.8, 3.0
| $ b <i64> 4, 5, null
| $ c <bool> True, False, True
| $ d <str> null, 'b', 'c'
| $ e <str> 'usd', 'eur', null
| $ f <date> 2020-01-01, 2021-01-02, 2022-01-01
| >>> res is None
| True
|
| Return the glimpse output as a string:
|
| >>> res = df.glimpse(return_type="string")
| >>> isinstance(res, str)
| True
|
| Return the glimpse output as a DataFrame:
|
| >>> df.glimpse(return_type="frame")
| shape: (6, 3)
| ┌────────┬───────┬─────────────────────────────────┐
| │ column ┆ dtype ┆ values │
| │ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ list[str] │
| ╞════════╪═══════╪═════════════════════════════════╡
| │ a ┆ f64 ┆ ["1.0", "2.8", "3.0"] │
| │ b ┆ i64 ┆ ["4", "5", null] │
| │ c ┆ bool ┆ ["True", "False", "True"] │
| │ d ┆ str ┆ [null, "'b'", "'c'"] │
| │ e ┆ str ┆ ["'usd'", "'eur'", null] │
| │ f ┆ date ┆ ["2020-01-01", "2021-01-02", "… │
| └────────┴───────┴─────────────────────────────────┘
|
| Print glimpse-formatted output to stdout, returning the *original* frame:
|
| >>> res = df.glimpse(return_type="self")
| Rows: 3
| Columns: 6
| $ a <f64> 1.0, 2.8, 3.0
| $ b <i64> 4, 5, null
| $ c <bool> True, False, True
| $ d <str> null, 'b', 'c'
| $ e <str> 'usd', 'eur', null
| $ f <date> 2020-01-01, 2021-01-02, 2022-01-01
| >>> res
| shape: (3, 6)
| ┌─────┬──────┬───────┬──────┬──────┬────────────┐
| │ a ┆ b ┆ c ┆ d ┆ e ┆ f │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ i64 ┆ bool ┆ str ┆ str ┆ date │
| ╞═════╪══════╪═══════╪══════╪══════╪════════════╡
| │ 1.0 ┆ 4 ┆ true ┆ null ┆ usd ┆ 2020-01-01 │
| │ 2.8 ┆ 5 ┆ false ┆ b ┆ eur ┆ 2021-01-02 │
| │ 3.0 ┆ null ┆ true ┆ c ┆ null ┆ 2022-01-01 │
| └─────┴──────┴───────┴──────┴──────┴────────────┘
|
| group_by(self, *by: 'IntoExpr | Iterable[IntoExpr]', maintain_order: 'bool' = False, **named_by: 'IntoExpr') -> 'GroupBy'
| Start a group by operation.
|
| Parameters
| ----------
| *by
| Column(s) to group by. Accepts expression input. Strings are parsed as
| column names.
| maintain_order
| Ensure that the order of the groups is consistent with the input data.
| This is slower than a default group by.
| Settings this to `True` blocks the possibility
| to run on the streaming engine.
|
| .. note::
| Within each group, the order of rows is always preserved, regardless
| of this argument.
| **named_by
| Additional columns to group by, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| GroupBy
| Object which can be used to perform aggregations.
|
| Examples
| --------
| Group by one column and call `agg` to compute the grouped sum of another
| column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "c"],
| ... "b": [1, 2, 1, 3, 3],
| ... "c": [5, 4, 3, 2, 1],
| ... }
| ... )
| >>> df.group_by("a").agg(pl.col("b").sum()) # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 2 │
| │ b ┆ 5 │
| │ c ┆ 3 │
| └─────┴─────┘
|
| Set `maintain_order=True` to ensure the order of the groups is consistent with
| the input.
|
| >>> df.group_by("a", maintain_order=True).agg(pl.col("c"))
| shape: (3, 2)
| ┌─────┬───────────┐
| │ a ┆ c │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═════╪═══════════╡
| │ a ┆ [5, 3] │
| │ b ┆ [4, 2] │
| │ c ┆ [1] │
| └─────┴───────────┘
|
| Group by multiple columns by passing a list of column names.
|
| >>> df.group_by(["a", "b"]).agg(pl.max("c")) # doctest: +IGNORE_RESULT
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘
|
| Or use positional arguments to group by multiple columns in the same way.
| Expressions are also accepted.
|
| >>> df.group_by("a", pl.col("b") // 2).agg(pl.col("c").mean()) # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ f64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 0 ┆ 4.0 │
| │ b ┆ 1 ┆ 3.0 │
| │ c ┆ 1 ┆ 1.0 │
| └─────┴─────┴─────┘
|
| The `GroupBy` object returned by this method is iterable, returning the name
| and data of each group.
|
| >>> for name, data in df.group_by("a"): # doctest: +SKIP
| ... print(name)
| ... print(data)
| ('a',)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ a ┆ 1 ┆ 3 │
| └─────┴─────┴─────┘
| ('b',)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| └─────┴─────┴─────┘
| ('c',)
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘
|
| group_by_dynamic(self, index_column: 'IntoExpr', *, every: 'str | timedelta', period: 'str | timedelta | None' = None, offset: 'str | timedelta | None' = None, include_boundaries: 'bool' = False, closed: 'ClosedInterval' = 'left', label: 'Label' = 'left', group_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, start_by: 'StartBy' = 'window') -> 'DynamicGroupBy'
| Group based on a time value (or index value of type Int32, Int64).
|
| Time windows are calculated and rows are assigned to windows. Different from a
| normal group by is that a row can be member of multiple groups.
| By default, the windows look like:
|
| - [start, start + period)
| - [start + every, start + every + period)
| - [start + 2*every, start + 2*every + period)
| - ...
|
| where `start` is determined by `start_by`, `offset`, `every`, and the earliest
| datapoint. See the `start_by` argument description for details.
|
| .. warning::
| The index column must be sorted in ascending order. If `group_by` is passed, then
| the index column must be sorted in ascending order within each group.
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order (or, if `group_by` is specified,
| then it must be sorted in ascending order within each group).
|
| In case of a dynamic group by on indices, dtype needs to be one of
| {Int32, Int64}. Note that Int32 gets temporarily cast to Int64, so if
| performance matters use an Int64 column.
| every
| interval of the window
| period
| length of the window, if None it will equal 'every'
| offset
| offset of the window, does not take effect if `start_by` is 'datapoint'.
| Defaults to zero.
| include_boundaries
| Add the lower and upper bound of the window to the "_lower_boundary" and
| "_upper_boundary" columns. This will impact performance because it's harder to
| parallelize
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
| label : {'left', 'right', 'datapoint'}
| Define which label to use for the window:
|
| - 'left': lower boundary of the window
| - 'right': upper boundary of the window
| - 'datapoint': the first value of the index column in the given window.
| If you don't need the label to be at one of the boundaries, choose this
| option for maximum performance
| group_by
| Also group by this column/these columns
| start_by : {'window', 'datapoint', 'monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday', 'sunday'}
| The strategy to determine the start of the first window by.
|
| * 'window': Start by taking the earliest timestamp, truncating it with
| `every`, and then adding `offset`.
| Note that weekly windows start on Monday.
| * 'datapoint': Start from the first encountered data point.
| * a day of the week (only takes effect if `every` contains `'w'`):
|
| * 'monday': Start the window on the Monday before the first data point.
| * 'tuesday': Start the window on the Tuesday before the first data point.
| * ...
| * 'sunday': Start the window on the Sunday before the first data point.
|
| The resulting window is then shifted back until the earliest datapoint
| is in or in front of it.
|
| Returns
| -------
| DynamicGroupBy
| Object you can call `.agg` on to aggregate by groups, the result
| of which will be sorted by `index_column` (but note that if `group_by` columns are
| passed, it will only be sorted within each group).
|
| See Also
| --------
| rolling
|
| Notes
| -----
| 1) If you're coming from pandas, then
|
| .. code-block:: python
|
| # polars
| df.group_by_dynamic("ts", every="1d").agg(pl.col("value").sum())
|
| is equivalent to
|
| .. code-block:: python
|
| # pandas
| df.set_index("ts").resample("D")["value"].sum().reset_index()
|
| though note that, unlike pandas, polars doesn't add extra rows for empty
| windows. If you need `index_column` to be evenly spaced, then please combine
| with :func:`DataFrame.upsample`.
|
| 2) The `every`, `period` and `offset` arguments are created with
| the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them (except in `every`):
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| In case of a group_by_dynamic on an integer column, the windows are defined by:
|
| - "1i" # length 1
| - "10i" # length 10
|
| Examples
| --------
| >>> from datetime import datetime
| >>> df = pl.DataFrame(
| ... {
| ... "time": pl.datetime_range(
| ... start=datetime(2021, 12, 16),
| ... end=datetime(2021, 12, 16, 3),
| ... interval="30m",
| ... eager=True,
| ... ),
| ... "n": range(7),
| ... }
| ... )
| >>> df
| shape: (7, 2)
| ┌─────────────────────┬─────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ i64 │
| ╞═════════════════════╪═════╡
| │ 2021-12-16 00:00:00 ┆ 0 │
| │ 2021-12-16 00:30:00 ┆ 1 │
| │ 2021-12-16 01:00:00 ┆ 2 │
| │ 2021-12-16 01:30:00 ┆ 3 │
| │ 2021-12-16 02:00:00 ┆ 4 │
| │ 2021-12-16 02:30:00 ┆ 5 │
| │ 2021-12-16 03:00:00 ┆ 6 │
| └─────────────────────┴─────┘
|
| Group by windows of 1 hour.
|
| >>> df.group_by_dynamic("time", every="1h", closed="right").agg(pl.col("n"))
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-15 23:00:00 ┆ [0] │
| │ 2021-12-16 00:00:00 ┆ [1, 2] │
| │ 2021-12-16 01:00:00 ┆ [3, 4] │
| │ 2021-12-16 02:00:00 ┆ [5, 6] │
| └─────────────────────┴───────────┘
|
| The window boundaries can also be added to the aggregation result
|
| >>> df.group_by_dynamic(
| ... "time", every="1h", include_boundaries=True, closed="right"
| ... ).agg(pl.col("n").mean())
| shape: (4, 4)
| ┌─────────────────────┬─────────────────────┬─────────────────────┬─────┐
| │ _lower_boundary ┆ _upper_boundary ┆ time ┆ n │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ f64 │
| ╞═════════════════════╪═════════════════════╪═════════════════════╪═════╡
| │ 2021-12-15 23:00:00 ┆ 2021-12-16 00:00:00 ┆ 2021-12-15 23:00:00 ┆ 0.0 │
| │ 2021-12-16 00:00:00 ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 00:00:00 ┆ 1.5 │
| │ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ 3.5 │
| │ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ 5.5 │
| └─────────────────────┴─────────────────────┴─────────────────────┴─────┘
|
| When closed="left", the window excludes the right end of interval:
| [lower_bound, upper_bound)
|
| >>> df.group_by_dynamic("time", every="1h", closed="left").agg(pl.col("n"))
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-16 00:00:00 ┆ [0, 1] │
| │ 2021-12-16 01:00:00 ┆ [2, 3] │
| │ 2021-12-16 02:00:00 ┆ [4, 5] │
| │ 2021-12-16 03:00:00 ┆ [6] │
| └─────────────────────┴───────────┘
|
| When closed="both" the time values at the window boundaries belong to 2 groups.
|
| >>> df.group_by_dynamic("time", every="1h", closed="both").agg(pl.col("n"))
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-16 00:00:00 ┆ [0, 1, 2] │
| │ 2021-12-16 01:00:00 ┆ [2, 3, 4] │
| │ 2021-12-16 02:00:00 ┆ [4, 5, 6] │
| │ 2021-12-16 03:00:00 ┆ [6] │
| └─────────────────────┴───────────┘
|
| Dynamic group bys can also be combined with grouping on normal keys
|
| >>> df = df.with_columns(groups=pl.Series(["a", "a", "a", "b", "b", "a", "a"]))
| >>> df
| shape: (7, 3)
| ┌─────────────────────┬─────┬────────┐
| │ time ┆ n ┆ groups │
| │ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ str │
| ╞═════════════════════╪═════╪════════╡
| │ 2021-12-16 00:00:00 ┆ 0 ┆ a │
| │ 2021-12-16 00:30:00 ┆ 1 ┆ a │
| │ 2021-12-16 01:00:00 ┆ 2 ┆ a │
| │ 2021-12-16 01:30:00 ┆ 3 ┆ b │
| │ 2021-12-16 02:00:00 ┆ 4 ┆ b │
| │ 2021-12-16 02:30:00 ┆ 5 ┆ a │
| │ 2021-12-16 03:00:00 ┆ 6 ┆ a │
| └─────────────────────┴─────┴────────┘
| >>> df.group_by_dynamic(
| ... "time",
| ... every="1h",
| ... closed="both",
| ... group_by="groups",
| ... include_boundaries=True,
| ... ).agg(pl.col("n"))
| shape: (6, 5)
| ┌────────┬─────────────────────┬─────────────────────┬─────────────────────┬───────────┐
| │ groups ┆ _lower_boundary ┆ _upper_boundary ┆ time ┆ n │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ list[i64] │
| ╞════════╪═════════════════════╪═════════════════════╪═════════════════════╪═══════════╡
| │ a ┆ 2021-12-16 00:00:00 ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 00:00:00 ┆ [0, 1, 2] │
| │ a ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ [2] │
| │ a ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ [5, 6] │
| │ a ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 04:00:00 ┆ 2021-12-16 03:00:00 ┆ [6] │
| │ b ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ [3, 4] │
| │ b ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ [4] │
| └────────┴─────────────────────┴─────────────────────┴─────────────────────┴───────────┘
|
| Dynamic group by on an index column
|
| >>> df = pl.DataFrame(
| ... {
| ... "idx": pl.int_range(0, 6, eager=True),
| ... "A": ["A", "A", "B", "B", "B", "C"],
| ... }
| ... )
| >>> (
| ... df.group_by_dynamic(
| ... "idx",
| ... every="2i",
| ... period="3i",
| ... include_boundaries=True,
| ... closed="right",
| ... ).agg(pl.col("A").alias("A_agg_list"))
| ... )
| shape: (4, 4)
| ┌─────────────────┬─────────────────┬─────┬─────────────────┐
| │ _lower_boundary ┆ _upper_boundary ┆ idx ┆ A_agg_list │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ list[str] │
| ╞═════════════════╪═════════════════╪═════╪═════════════════╡
| │ -2 ┆ 1 ┆ -2 ┆ ["A", "A"] │
| │ 0 ┆ 3 ┆ 0 ┆ ["A", "B", "B"] │
| │ 2 ┆ 5 ┆ 2 ┆ ["B", "B", "C"] │
| │ 4 ┆ 7 ┆ 4 ┆ ["C"] │
| └─────────────────┴─────────────────┴─────┴─────────────────┘
|
| hash_rows(self, seed: 'int' = 0, seed_1: 'int | None' = None, seed_2: 'int | None' = None, seed_3: 'int | None' = None) -> 'Series'
| Hash and combine the rows in this DataFrame.
|
| The hash value is of type `UInt64`.
|
| Parameters
| ----------
| seed
| Random seed parameter. Defaults to 0.
| seed_1
| Random seed parameter. Defaults to `seed` if not set.
| seed_2
| Random seed parameter. Defaults to `seed` if not set.
| seed_3
| Random seed parameter. Defaults to `seed` if not set.
|
| Notes
| -----
| This implementation of `hash_rows` does not guarantee stable results
| across different Polars versions. Its stability is only guaranteed within a
| single version.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, None, 3, 4],
| ... "ham": ["a", "b", None, "d"],
| ... }
| ... )
| >>> df.hash_rows(seed=42) # doctest: +IGNORE_RESULT
| shape: (4,)
| Series: '' [u64]
| [
| 10783150408545073287
| 1438741209321515184
| 10047419486152048166
| 2047317070637311557
| ]
|
| head(self, n: 'int' = 5) -> 'DataFrame'
| Get the first `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return. If a negative value is passed, return all rows
| except the last `abs(n)`.
|
| See Also
| --------
| tail, glimpse, slice
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> df.head(3)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Pass a negative value to get all rows `except` the last `abs(n)`.
|
| >>> df.head(-3)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| └─────┴─────┴─────┘
|
| hstack(self, columns: 'list[Series] | DataFrame', *, in_place: 'bool' = False) -> 'DataFrame'
| Return a new DataFrame grown horizontally by stacking multiple Series to it.
|
| Parameters
| ----------
| columns
| Series to stack.
| in_place
| Modify in place.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> x = pl.Series("apple", [10, 20, 30])
| >>> df.hstack([x])
| shape: (3, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6 ┆ a ┆ 10 │
| │ 2 ┆ 7 ┆ b ┆ 20 │
| │ 3 ┆ 8 ┆ c ┆ 30 │
| └─────┴─────┴─────┴───────┘
|
| insert_column(self, index: 'int', column: 'IntoExprColumn') -> 'DataFrame'
| Insert a Series (or expression) at a certain column index.
|
| This operation is in place.
|
| Parameters
| ----------
| index
| Index at which to insert the new column.
| column
| `Series` or expression to insert.
|
| Examples
| --------
| Insert a new Series column at the given index:
|
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> s = pl.Series("baz", [97, 98, 99])
| >>> df.insert_column(1, s)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ baz ┆ bar │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 97 ┆ 4 │
| │ 2 ┆ 98 ┆ 5 │
| │ 3 ┆ 99 ┆ 6 │
| └─────┴─────┴─────┘
|
| Insert a new expression column at the given index:
|
| >>> df = pl.DataFrame(
| ... {"a": [2, 4, 2], "b": [0.5, 4, 10], "c": ["xx", "yy", "zz"]}
| ... )
| >>> expr = (pl.col("b") / pl.col("a")).alias("b_div_a")
| >>> df.insert_column(2, expr)
| shape: (3, 4)
| ┌─────┬──────┬─────────┬─────┐
| │ a ┆ b ┆ b_div_a ┆ c │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 ┆ str │
| ╞═════╪══════╪═════════╪═════╡
| │ 2 ┆ 0.5 ┆ 0.25 ┆ xx │
| │ 4 ┆ 4.0 ┆ 1.0 ┆ yy │
| │ 2 ┆ 10.0 ┆ 5.0 ┆ zz │
| └─────┴──────┴─────────┴─────┘
|
| interpolate(self) -> 'DataFrame'
| Interpolate intermediate values. The interpolation method is linear.
|
| Nulls at the beginning and end of the series remain null.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, None, 9, 10],
| ... "bar": [6, 7, 9, None],
| ... "baz": [1, None, None, 9],
| ... }
| ... )
| >>> df.interpolate()
| shape: (4, 3)
| ┌──────┬──────┬──────────┐
| │ foo ┆ bar ┆ baz │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪══════════╡
| │ 1.0 ┆ 6.0 ┆ 1.0 │
| │ 5.0 ┆ 7.0 ┆ 3.666667 │
| │ 9.0 ┆ 9.0 ┆ 6.333333 │
| │ 10.0 ┆ null ┆ 9.0 │
| └──────┴──────┴──────────┘
|
| is_duplicated(self) -> 'Series'
| Get a mask of all duplicated rows in this DataFrame.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 1],
| ... "b": ["x", "y", "z", "x"],
| ... }
| ... )
| >>> df.is_duplicated()
| shape: (4,)
| Series: '' [bool]
| [
| true
| false
| false
| true
| ]
|
| This mask can be used to visualize the duplicated lines like this:
|
| >>> df.filter(df.is_duplicated())
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ x │
| │ 1 ┆ x │
| └─────┴─────┘
|
| is_empty(self) -> 'bool'
| Returns `True` if the DataFrame contains no rows.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df.is_empty()
| False
| >>> df.filter(pl.col("foo") > 99).is_empty()
| True
|
| is_sorted(self, by: 'str | Iterable[str]', *more_by: 'str', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False) -> 'bool'
| Check whether the DataFrame is sorted by the given columns.
|
| Parameters
| ----------
| by
| Column name(s) to check.
| *more_by
| Additional column names.
| descending
| Sort in descending order. When sorting by multiple columns, can be
| specified per column by passing a sequence of booleans.
| nulls_last
| Place null values last. When sorting by multiple columns, can be
| specified per column by passing a sequence of booleans.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [5, 4, 3]})
| >>> df.is_sorted("a")
| True
| >>> df.is_sorted("b", descending=True)
| True
| >>> df.is_sorted("a", "b")
| True
|
| is_unique(self) -> 'Series'
| Get a mask of all unique rows in this DataFrame.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 1],
| ... "b": ["x", "y", "z", "x"],
| ... }
| ... )
| >>> df.is_unique()
| shape: (4,)
| Series: '' [bool]
| [
| false
| true
| true
| false
| ]
|
| This mask can be used to visualize the unique lines like this:
|
| >>> df.filter(df.is_unique())
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 2 ┆ y │
| │ 3 ┆ z │
| └─────┴─────┘
|
| item(self, row: 'int | None' = None, column: 'int | str | None' = None) -> 'Any'
| Return the DataFrame as a scalar, or return the element at the given row/column.
|
| Parameters
| ----------
| row
| Optional row index.
| column
| Optional column index or name.
|
| See Also
| --------
| row : Get the values of a single row, either by index or by predicate.
|
| Notes
| -----
| If row/col not provided, this is equivalent to `df[0,0]`, with a check that
| the shape is (1,1). With row/col, this is equivalent to `df[row,col]`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
| >>> df.select((pl.col("a") * pl.col("b")).sum()).item()
| 32
| >>> df.item(1, 1)
| 5
| >>> df.item(2, "b")
| 6
|
| iter_columns(self) -> 'Iterator[Series]'
| Returns an iterator over the columns of this DataFrame.
|
| Yields
| ------
| Series
|
| Notes
| -----
| Consider whether you can use :func:`all` instead.
| If you can, it will be more efficient.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> [s.name for s in df.iter_columns()]
| ['a', 'b']
|
| If you're using this to modify a dataframe's columns, e.g.
|
| >>> # Do NOT do this
| >>> pl.DataFrame(column * 2 for column in df.iter_columns())
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 4 │
| │ 6 ┆ 8 │
| │ 10 ┆ 12 │
| └─────┴─────┘
|
| then consider whether you can use :func:`all` instead:
|
| >>> df.select(pl.all() * 2)
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 4 │
| │ 6 ┆ 8 │
| │ 10 ┆ 12 │
| └─────┴─────┘
|
| iter_rows(self, *, named: 'bool' = False, buffer_size: 'int' = 512) -> 'Iterator[tuple[Any, ...]] | Iterator[dict[str, Any]]'
| Returns an iterator over the DataFrame of rows of python-native values.
|
| Parameters
| ----------
| named
| Return dictionaries instead of tuples. The dictionaries are a mapping of
| column name to row value. This is more expensive than returning a regular
| tuple, but allows for accessing values by column name.
| buffer_size
| Determines the number of rows that are buffered internally while iterating
| over the data; you should only modify this in very specific cases where the
| default value is determined not to be a good fit to your access pattern, as
| the speedup from using the buffer is significant (~2-4x). Setting this
| value to zero disables row buffering (not recommended).
|
| Notes
| -----
| If you have `ns`-precision temporal values you should be aware that Python
| natively only supports up to `μs`-precision; `ns`-precision values will be
| truncated to microseconds on conversion to Python. If this matters to your
| use-case you should export to a different format (such as Arrow or NumPy).
|
| Warnings
| --------
| Row iteration is not optimal as the underlying data is stored in columnar form;
| where possible, prefer export via one of the dedicated export/output methods
| that deals with columnar data.
|
| Yields
| ------
| iterator of tuples (default) or dictionaries (if named) of python row values
|
| See Also
| --------
| rows : Materialises all frame data as a list of rows (potentially expensive).
| rows_by_key : Materialises frame data as a key-indexed dictionary.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> [row[0] for row in df.iter_rows()]
| [1, 3, 5]
| >>> [row["b"] for row in df.iter_rows(named=True)]
| [2, 4, 6]
|
| iter_slices(self, n_rows: 'int' = 10000) -> 'Iterator[DataFrame]'
| Returns a non-copying iterator of slices over the underlying DataFrame.
|
| Parameters
| ----------
| n_rows
| Determines the number of rows contained in each DataFrame slice.
|
| Examples
| --------
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... data={
| ... "a": range(17_500),
| ... "b": date(2023, 1, 1),
| ... "c": "klmnoopqrstuvwxyz",
| ... },
| ... schema_overrides={"a": pl.Int32},
| ... )
| >>> for idx, frame in enumerate(df.iter_slices()):
| ... print(f"{type(frame).__name__}:[{idx}]:{len(frame)}")
| DataFrame:[0]:10000
| DataFrame:[1]:7500
|
| Using `iter_slices` is an efficient way to chunk-iterate over DataFrames and
| any supported frame export/conversion types; for example, as RecordBatches:
|
| >>> for frame in df.iter_slices(n_rows=15_000):
| ... record_batch = frame.to_arrow().to_batches()[0]
| ... print(f"{record_batch.schema}\n<< {len(record_batch)}")
| a: int32
| b: date32[day]
| c: large_string
| << 15000
| a: int32
| b: date32[day]
| c: large_string
| << 2500
|
| See Also
| --------
| iter_rows : Row iterator over frame data (does not materialise all rows).
| partition_by : Split into multiple DataFrames, partitioned by groups.
|
| join(self, other: 'DataFrame', on: 'str | Expr | Sequence[str | Expr] | None' = None, how: 'JoinStrategy' = 'inner', *, left_on: 'str | Expr | Sequence[str | Expr] | None' = None, right_on: 'str | Expr | Sequence[str | Expr] | None' = None, suffix: 'str' = '_right', validate: 'JoinValidation' = 'm:m', nulls_equal: 'bool' = False, coalesce: 'bool | None' = None, maintain_order: 'MaintainOrderJoin | None' = None, build_side: 'JoinBuildSide' = 'auto') -> 'DataFrame'
| Join in SQL-like fashion.
|
| .. versionchanged:: 1.24
| The `join_nulls` parameter was renamed `nulls_equal`.
|
| Parameters
| ----------
| other
| DataFrame to join with.
| on
| Name(s) of the join columns in both DataFrames. If set, `left_on` and
| `right_on` should be None. This should not be specified if `how='cross'`.
| how : {'inner', 'left', 'right', 'full', 'semi', 'anti', 'cross'}
| Join strategy.
|
| .. list-table ::
| :header-rows: 0
|
| * - **inner**
| - *(Default)* Returns rows that have matching values in both tables.
| * - **left**
| - Returns all rows from the left table, and the matched rows from
| the right table.
| * - **right**
| - Returns all rows from the right table, and the matched rows from
| the left table.
| * - **full**
| - Returns all rows from both tables, joining matching rows and
| filling non-matches with null values.
| * - **cross**
| - Returns the Cartesian product of rows from both tables
| * - **semi**
| - Returns rows from the left table that have a match in the right
| table. Does not return columns from the right table.
| * - **anti**
| - Returns rows from the left table that have no match in the right
| table. Does not return columns from the right table.
|
| left_on
| Name(s) of the left join column(s).
| right_on
| Name(s) of the right join column(s).
| suffix
| Suffix to append to columns with a duplicate name.
| validate: {'m:m', 'm:1', '1:m', '1:1'}
| Checks if join is of specified type.
|
| .. list-table ::
| :header-rows: 0
|
| * - **m:m**
| - *(Default)* Many-to-many (default). Does not result in checks.
| * - **1:1**
| - One-to-one. Checks if join keys are unique in both left and
| right datasets.
| * - **1:m**
| - One-to-many. Checks if join keys are unique in left dataset.
| * - **m:1**
| - Many-to-one. Check if join keys are unique in right dataset.
|
| .. note::
| This is currently not supported by the streaming engine.
|
| nulls_equal
| Join on null values. By default null values will never produce matches.
| coalesce
| Coalescing behavior (merging of join columns).
|
| .. list-table ::
| :header-rows: 0
|
| * - **None**
| - *(Default)* Coalesce unless `how='full'` is specified.
| * - **True**
| - Always coalesce join columns.
| * - **False**
| - Never coalesce join columns.
|
| .. note::
| Joining on any other expressions than `col`
| will turn off coalescing.
| maintain_order : {'none', 'left', 'right', 'left_right', 'right_left'}
| Which DataFrame row order to preserve, if any.
| Do not rely on any observed ordering without explicitly setting this
| parameter, as your code may break in a future release.
| Not specifying any ordering can improve performance.
|
| .. list-table ::
| :header-rows: 0
|
| * - **none**
| - *(Default)* No specific ordering is desired. The ordering might
| differ across Polars versions or even between different runs.
| * - **left**
| - Preserves the order of the left DataFrame.
| * - **right**
| - Preserves the order of the right DataFrame.
| * - **left_right**
| - First preserves the order of the left DataFrame, then the right.
| * - **right_left**
| - First preserves the order of the right DataFrame, then the left.
| build_side: {'auto', 'prefer_left', 'prefer_right', 'force_left', 'force_right'}
| Which side of the join will be used as the build side. This side will be
| likely be held in memory as a hash table. Note that unless a `force_`
| variant is chosen, the chosen side might differ across Polars versions or
| even between different runs.
|
| .. list-table ::
| :header-rows: 0
|
| * - **auto**
| - *(Default)* Let Polars figure out the build side.
| * - **prefer_left**
| - Unless there's a very good reason to believe that the right side is
| smaller, use the left side.
| * - **prefer_right**
| - Unless there's a very good reason to believe that the left side is
| smaller, use the right side.
| * - **force_left**
| - Always use the left side.
| * - **force_right**
| - Always use the right side.
|
| .. warning::
| This functionality is considered **experimental**. It may be removed or
| changed at any point without it being considered a breaking change.
|
|
| See Also
| --------
| join_asof
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> other_df = pl.DataFrame(
| ... {
| ... "apple": ["x", "y", "z"],
| ... "ham": ["a", "b", "d"],
| ... }
| ... )
| >>> df.join(other_df, on="ham")
| shape: (2, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| └─────┴─────┴─────┴───────┘
|
| >>> df.join(other_df, on="ham", how="full")
| shape: (4, 5)
| ┌──────┬──────┬──────┬───────┬───────────┐
| │ foo ┆ bar ┆ ham ┆ apple ┆ ham_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞══════╪══════╪══════╪═══════╪═══════════╡
| │ 1 ┆ 6.0 ┆ a ┆ x ┆ a │
| │ 2 ┆ 7.0 ┆ b ┆ y ┆ b │
| │ null ┆ null ┆ null ┆ z ┆ d │
| │ 3 ┆ 8.0 ┆ c ┆ null ┆ null │
| └──────┴──────┴──────┴───────┴───────────┘
|
| >>> df.join(other_df, on="ham", how="full", coalesce=True)
| shape: (4, 4)
| ┌──────┬──────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞══════╪══════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| │ null ┆ null ┆ d ┆ z │
| │ 3 ┆ 8.0 ┆ c ┆ null │
| └──────┴──────┴─────┴───────┘
|
| >>> df.join(other_df, on="ham", how="left")
| shape: (3, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| │ 3 ┆ 8.0 ┆ c ┆ null │
| └─────┴─────┴─────┴───────┘
|
| >>> df.join(other_df, on="ham", how="semi")
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 7.0 ┆ b │
| └─────┴─────┴─────┘
|
| >>> df.join(other_df, on="ham", how="anti")
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| >>> df.join(other_df, how="cross")
| shape: (9, 5)
| ┌─────┬─────┬─────┬───────┬───────────┐
| │ foo ┆ bar ┆ ham ┆ apple ┆ ham_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╪═══════════╡
| │ 1 ┆ 6.0 ┆ a ┆ x ┆ a │
| │ 1 ┆ 6.0 ┆ a ┆ y ┆ b │
| │ 1 ┆ 6.0 ┆ a ┆ z ┆ d │
| │ 2 ┆ 7.0 ┆ b ┆ x ┆ a │
| │ 2 ┆ 7.0 ┆ b ┆ y ┆ b │
| │ 2 ┆ 7.0 ┆ b ┆ z ┆ d │
| │ 3 ┆ 8.0 ┆ c ┆ x ┆ a │
| │ 3 ┆ 8.0 ┆ c ┆ y ┆ b │
| │ 3 ┆ 8.0 ┆ c ┆ z ┆ d │
| └─────┴─────┴─────┴───────┴───────────┘
|
| join_asof(self, other: 'DataFrame', *, left_on: 'str | None | Expr' = None, right_on: 'str | None | Expr' = None, on: 'str | None | Expr' = None, by_left: 'str | Sequence[str] | None' = None, by_right: 'str | Sequence[str] | None' = None, by: 'str | Sequence[str] | None' = None, strategy: 'AsofJoinStrategy' = 'backward', suffix: 'str' = '_right', tolerance: 'str | int | float | timedelta | None' = None, allow_parallel: 'bool' = True, force_parallel: 'bool' = False, coalesce: 'bool' = True, allow_exact_matches: 'bool' = True, check_sortedness: 'bool' = True) -> 'DataFrame'
| Perform an asof join.
|
| This is similar to a left-join except that we match on nearest key rather than
| equal keys.
|
| Both DataFrames must be sorted by the `on` key (within each `by` group, if
| specified).
|
| For each row in the left DataFrame:
|
| - A "backward" search selects the last row in the right DataFrame whose
| 'on' key is less than or equal to the left's key.
|
| - A "forward" search selects the first row in the right DataFrame whose
| 'on' key is greater than or equal to the left's key.
|
| - A "nearest" search selects the last row in the right DataFrame whose value
| is nearest to the left's key. String keys are not currently supported for a
| nearest search.
|
| The default is "backward".
|
| Parameters
| ----------
| other
| Lazy DataFrame to join with.
| left_on
| Join column of the left DataFrame.
| right_on
| Join column of the right DataFrame.
| on
| Join column of both DataFrames. If set, `left_on` and `right_on` should be
| None.
| by_left
| Join on these columns before doing asof join
| by_right
| Join on these columns before doing asof join
| by
| Join on these columns before doing asof join
| strategy : {'backward', 'forward', 'nearest'}
| Join strategy.
| suffix
| Suffix to append to columns with a duplicate name.
| tolerance
| Numeric tolerance. By setting this the join will only be done if the near
| keys are within this distance. If an asof join is done on columns of dtype
| "Date", "Datetime", "Duration" or "Time", use either a datetime.timedelta
| object or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of
| ambiguity, we follow RFC-5545 and preserve the DST fold of the original
| datetime). Similarly for "calendar week", "calendar month",
| "calendar quarter", and "calendar year".
|
| allow_parallel
| Allow the physical plan to optionally evaluate the computation of both
| DataFrames up to the join in parallel.
| force_parallel
| Force the physical plan to evaluate the computation of both DataFrames up to
| the join in parallel.
| coalesce
| Coalescing behavior (merging of `on` / `left_on` / `right_on` columns):
|
| - *True*: Always coalesce join columns.
| - *False*: Never coalesce join columns.
|
| Note that joining on any other expressions than `col`
| will turn off coalescing.
| allow_exact_matches
| Whether exact matches are valid join predicates.
|
| - If True, allow matching with the same ``on`` value
| (i.e. less-than-or-equal-to / greater-than-or-equal-to)
| - If False, don't match the same ``on`` value
| (i.e., strictly less-than / strictly greater-than).
| check_sortedness
| Check the sortedness of the asof keys. If the keys are not sorted Polars
| will error. Currently, the `in-memory` engine cannot check the sortedness
| if 'by' groups are provided. The `streaming` engine will only check the
| sortedness of the rows it processes.
|
| Examples
| --------
| >>> from datetime import date
| >>> gdp = pl.DataFrame(
| ... {
| ... "date": pl.date_range(
| ... date(2016, 1, 1),
| ... date(2020, 1, 1),
| ... "1y",
| ... eager=True,
| ... ),
| ... "gdp": [4164, 4411, 4566, 4696, 4827],
| ... }
| ... )
| >>> gdp
| shape: (5, 2)
| ┌────────────┬──────┐
| │ date ┆ gdp │
| │ --- ┆ --- │
| │ date ┆ i64 │
| ╞════════════╪══════╡
| │ 2016-01-01 ┆ 4164 │
| │ 2017-01-01 ┆ 4411 │
| │ 2018-01-01 ┆ 4566 │
| │ 2019-01-01 ┆ 4696 │
| │ 2020-01-01 ┆ 4827 │
| └────────────┴──────┘
|
| >>> population = pl.DataFrame(
| ... {
| ... "date": [date(2016, 3, 1), date(2018, 8, 1), date(2019, 1, 1)],
| ... "population": [82.19, 82.66, 83.12],
| ... }
| ... ).sort("date")
| >>> population
| shape: (3, 2)
| ┌────────────┬────────────┐
| │ date ┆ population │
| │ --- ┆ --- │
| │ date ┆ f64 │
| ╞════════════╪════════════╡
| │ 2016-03-01 ┆ 82.19 │
| │ 2018-08-01 ┆ 82.66 │
| │ 2019-01-01 ┆ 83.12 │
| └────────────┴────────────┘
|
| Note how the dates don't quite match. If we join them using `join_asof` and
| `strategy='backward'`, then each date from `population` which doesn't have an
| exact match is matched with the closest earlier date from `gdp`:
|
| >>> population.join_asof(gdp, on="date", strategy="backward")
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 4566 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2016-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2018-01-01` from `gdp`.
|
| You can verify this by passing `coalesce=False`:
|
| >>> population.join_asof(gdp, on="date", strategy="backward", coalesce=False)
| shape: (3, 4)
| ┌────────────┬────────────┬────────────┬──────┐
| │ date ┆ population ┆ date_right ┆ gdp │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ date ┆ i64 │
| ╞════════════╪════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 2016-01-01 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 2018-01-01 ┆ 4566 │
| │ 2019-01-01 ┆ 83.12 ┆ 2019-01-01 ┆ 4696 │
| └────────────┴────────────┴────────────┴──────┘
|
| If we instead use `strategy='forward'`, then each date from `population` which
| doesn't have an exact match is matched with the closest later date from `gdp`:
|
| >>> population.join_asof(gdp, on="date", strategy="forward")
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4411 │
| │ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2017-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2019-01-01` from `gdp`.
|
| Finally, `strategy='nearest'` gives us a mix of the two results above, as each
| date from `population` which doesn't have an exact match is matched with the
| closest date from `gdp`, regardless of whether it's earlier or later:
|
| >>> population.join_asof(gdp, on="date", strategy="nearest")
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2016-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2019-01-01` from `gdp`.
|
| The `by` argument allows joining on another column first, before the asof join.
| In this example we join by `country` first, then asof join by date, as above.
|
| >>> gdp_dates = pl.date_range( # fmt: skip
| ... date(2016, 1, 1), date(2020, 1, 1), "1y", eager=True
| ... )
| >>> gdp2 = pl.DataFrame(
| ... {
| ... "country": ["Germany"] * 5 + ["Netherlands"] * 5,
| ... "date": pl.concat([gdp_dates, gdp_dates]),
| ... "gdp": [4164, 4411, 4566, 4696, 4827, 784, 833, 914, 910, 909],
| ... }
| ... ).sort("country", "date")
| >>>
| >>> gdp2
| shape: (10, 3)
| ┌─────────────┬────────────┬──────┐
| │ country ┆ date ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 │
| ╞═════════════╪════════════╪══════╡
| │ Germany ┆ 2016-01-01 ┆ 4164 │
| │ Germany ┆ 2017-01-01 ┆ 4411 │
| │ Germany ┆ 2018-01-01 ┆ 4566 │
| │ Germany ┆ 2019-01-01 ┆ 4696 │
| │ Germany ┆ 2020-01-01 ┆ 4827 │
| │ Netherlands ┆ 2016-01-01 ┆ 784 │
| │ Netherlands ┆ 2017-01-01 ┆ 833 │
| │ Netherlands ┆ 2018-01-01 ┆ 914 │
| │ Netherlands ┆ 2019-01-01 ┆ 910 │
| │ Netherlands ┆ 2020-01-01 ┆ 909 │
| └─────────────┴────────────┴──────┘
| >>> pop2 = pl.DataFrame(
| ... {
| ... "country": ["Germany"] * 3 + ["Netherlands"] * 3,
| ... "date": [
| ... date(2016, 3, 1),
| ... date(2018, 8, 1),
| ... date(2019, 1, 1),
| ... date(2016, 3, 1),
| ... date(2018, 8, 1),
| ... date(2019, 1, 1),
| ... ],
| ... "population": [82.19, 82.66, 83.12, 17.11, 17.32, 17.40],
| ... }
| ... ).sort("country", "date")
| >>>
| >>> pop2
| shape: (6, 3)
| ┌─────────────┬────────────┬────────────┐
| │ country ┆ date ┆ population │
| │ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ f64 │
| ╞═════════════╪════════════╪════════════╡
| │ Germany ┆ 2016-03-01 ┆ 82.19 │
| │ Germany ┆ 2018-08-01 ┆ 82.66 │
| │ Germany ┆ 2019-01-01 ┆ 83.12 │
| │ Netherlands ┆ 2016-03-01 ┆ 17.11 │
| │ Netherlands ┆ 2018-08-01 ┆ 17.32 │
| │ Netherlands ┆ 2019-01-01 ┆ 17.4 │
| └─────────────┴────────────┴────────────┘
| >>> pop2.join_asof(gdp2, by="country", on="date", strategy="nearest")
| shape: (6, 4)
| ┌─────────────┬────────────┬────────────┬──────┐
| │ country ┆ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ f64 ┆ i64 │
| ╞═════════════╪════════════╪════════════╪══════╡
| │ Germany ┆ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ Germany ┆ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ Germany ┆ 2019-01-01 ┆ 83.12 ┆ 4696 │
| │ Netherlands ┆ 2016-03-01 ┆ 17.11 ┆ 784 │
| │ Netherlands ┆ 2018-08-01 ┆ 17.32 ┆ 910 │
| │ Netherlands ┆ 2019-01-01 ┆ 17.4 ┆ 910 │
| └─────────────┴────────────┴────────────┴──────┘
|
| join_where(self, other: 'DataFrame', *predicates: 'Expr | Iterable[Expr]', suffix: 'str' = '_right') -> 'DataFrame'
| Perform a join based on one or multiple (in)equality predicates.
|
| This performs an inner join, so only rows where all predicates are true
| are included in the result, and a row from either DataFrame may be included
| multiple times in the result.
|
| .. note::
| The row order of the input DataFrames is not preserved.
|
| .. warning::
| This functionality is experimental. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| other
| DataFrame to join with.
| *predicates
| (In)Equality condition to join the two tables on.
| When a column name occurs in both tables, the proper suffix must
| be applied in the predicate.
| suffix
| Suffix to append to columns with a duplicate name.
|
| Examples
| --------
| Join two dataframes together based on two predicates which get AND-ed together.
|
| >>> east = pl.DataFrame(
| ... {
| ... "id": [100, 101, 102],
| ... "dur": [120, 140, 160],
| ... "rev": [12, 14, 16],
| ... "cores": [2, 8, 4],
| ... }
| ... )
| >>> west = pl.DataFrame(
| ... {
| ... "t_id": [404, 498, 676, 742],
| ... "time": [90, 130, 150, 170],
| ... "cost": [9, 13, 15, 16],
| ... "cores": [4, 2, 1, 4],
| ... }
| ... )
| >>> east.join_where(
| ... west,
| ... pl.col("dur") < pl.col("time"),
| ... pl.col("rev") < pl.col("cost"),
| ... )
| shape: (5, 8)
| ┌─────┬─────┬─────┬───────┬──────┬──────┬──────┬─────────────┐
| │ id ┆ dur ┆ rev ┆ cores ┆ t_id ┆ time ┆ cost ┆ cores_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╪══════╪══════╪══════╪═════════════╡
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 498 ┆ 130 ┆ 13 ┆ 2 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| └─────┴─────┴─────┴───────┴──────┴──────┴──────┴─────────────┘
|
| To OR them together, use a single expression and the `|` operator.
|
| >>> east.join_where(
| ... west,
| ... (pl.col("dur") < pl.col("time")) | (pl.col("rev") < pl.col("cost")),
| ... )
| shape: (6, 8)
| ┌─────┬─────┬─────┬───────┬──────┬──────┬──────┬─────────────┐
| │ id ┆ dur ┆ rev ┆ cores ┆ t_id ┆ time ┆ cost ┆ cores_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╪══════╪══════╪══════╪═════════════╡
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 498 ┆ 130 ┆ 13 ┆ 2 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 102 ┆ 160 ┆ 16 ┆ 4 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| └─────┴─────┴─────┴───────┴──────┴──────┴──────┴─────────────┘
|
| lazy(self) -> 'LazyFrame'
| Start a lazy query from this point. This returns a `LazyFrame` object.
|
| Operations on a `LazyFrame` are not executed until this is triggered
| by calling one of:
|
| * :meth:`.collect() <polars.LazyFrame.collect>`
| (run on all data)
| * :meth:`.explain() <polars.LazyFrame.explain>`
| (print the query plan)
| * :meth:`.show_graph() <polars.LazyFrame.show_graph>`
| (show the query plan as graphviz graph)
| * :meth:`.collect_schema() <polars.LazyFrame.collect_schema>`
| (return the final frame schema)
|
| Lazy operations are recommended because they allow for query optimization and
| additional parallelism.
|
| Returns
| -------
| LazyFrame
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> df.lazy()
| <LazyFrame at ...>
|
| limit(self, n: 'int' = 5) -> 'DataFrame'
| Get the first `n` rows.
|
| Alias for :func:`DataFrame.head`.
|
| Parameters
| ----------
| n
| Number of rows to return. If a negative value is passed, return all rows
| except the last `abs(n)`.
|
| See Also
| --------
| head
|
| Examples
| --------
| Get the first 3 rows of a DataFrame.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> df.limit(3)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| map_columns(self, column_names: 'str | Sequence[str] | pl.Selector', function: 'Callable[Concatenate[Series, P], Series]', *args: 'P.args', **kwargs: 'P.kwargs') -> 'DataFrame'
| Apply eager functions to columns of a DataFrame.
|
| Users should always prefer :meth:`with_columns` unless they are using
| expressions that are only possible on `Series` and not on `Expr`. This is almost
| never the case, except for a very select few functions that cannot know the
| output datatype without looking at the data.
|
| Parameters
| ----------
| column_names
| The columns to apply the UDF to.
| function
| Callable; will receive a column series as the first parameter,
| followed by any given args/kwargs.
| *args
| Arguments to pass to the UDF.
| **kwargs
| Keyword arguments to pass to the UDF.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4], "b": ["10", "20", "30", "40"]})
| >>> df.map_columns("a", lambda s: s.shrink_dtype())
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i8 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ 10 │
| │ 2 ┆ 20 │
| │ 3 ┆ 30 │
| │ 4 ┆ 40 │
| └─────┴─────┘
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ['{"x":"a"}', None, '{"x":"b"}', None],
| ... "b": ['{"a":1, "b": true}', None, '{"a":2, "b": false}', None],
| ... }
| ... )
| >>> df.map_columns(["a", "b"], lambda s: s.str.json_decode())
| shape: (4, 2)
| ┌───────────┬───────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ struct[1] ┆ struct[2] │
| ╞═══════════╪═══════════╡
| │ {"a"} ┆ {1,true} │
| │ null ┆ null │
| │ {"b"} ┆ {2,false} │
| │ null ┆ null │
| └───────────┴───────────┘
| >>> import polars.selectors as cs
| >>> df.map_columns(cs.all(), lambda s: s.str.json_decode())
| shape: (4, 2)
| ┌───────────┬───────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ struct[1] ┆ struct[2] │
| ╞═══════════╪═══════════╡
| │ {"a"} ┆ {1,true} │
| │ null ┆ null │
| │ {"b"} ┆ {2,false} │
| │ null ┆ null │
| └───────────┴───────────┘
|
| See Also
| --------
| with_columns
|
| map_rows(self, function: 'Callable[[tuple[Any, ...]], Any]', return_dtype: 'PolarsDataType | None' = None, *, inference_size: 'int' = 256) -> 'DataFrame'
| Apply a custom/user-defined function (UDF) over the rows of the DataFrame.
|
| .. warning::
| This method is much slower than the native expressions API.
| Only use it if you cannot implement your logic otherwise.
|
| The UDF will receive each row as a tuple of values: `udf(row)`.
|
| Implementing logic using a Python function is almost always *significantly*
| slower and more memory intensive than implementing the same logic using
| the native expression API because:
|
| - The native expression engine runs in Rust; UDFs run in Python.
| - Use of Python UDFs forces the DataFrame to be materialized in memory.
| - Polars-native expressions can be parallelised (UDFs typically cannot).
| - Polars-native expressions can be logically optimised (UDFs cannot).
|
| Wherever possible you should strongly prefer the native expression API
| to achieve the best performance.
|
| Parameters
| ----------
| function
| Custom function or lambda.
| return_dtype
| Output type of the operation. If none given, Polars tries to infer the type.
| inference_size
| Only used in the case when the custom function returns rows.
| This uses the first `n` rows to determine the output schema.
|
| Notes
| -----
| * The frame-level `map_rows` cannot track column names (as the UDF is a
| black-box that may arbitrarily drop, rearrange, transform, or add new
| columns); if you want to apply a UDF such that column names are preserved,
| you should use the expression-level `map_elements` syntax instead.
|
| * If your function is expensive and you don't want it to be called more than
| once for a given input, consider applying an `@lru_cache` decorator to it.
| If your data is suitable you may achieve *significant* speedups.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [-1, 5, 8]})
|
| Return a DataFrame by mapping each row to a tuple:
|
| >>> df.map_rows(lambda t: (t[0] * 2, t[1] * 3))
| shape: (3, 2)
| ┌──────────┬──────────┐
| │ column_0 ┆ column_1 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════════╪══════════╡
| │ 2 ┆ -3 │
| │ 4 ┆ 15 │
| │ 6 ┆ 24 │
| └──────────┴──────────┘
|
| However, it is much better to implement this with a native expression:
|
| >>> df.select(
| ... pl.col("foo") * 2,
| ... pl.col("bar") * 3,
| ... ) # doctest: +IGNORE_RESULT
|
| Return a DataFrame with a single column by mapping each row to a scalar:
|
| >>> df.map_rows(lambda t: t[0] * 2 + t[1])
| shape: (3, 1)
| ┌─────┐
| │ map │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 9 │
| │ 14 │
| └─────┘
|
| In this case it is better to use the following native expression:
|
| >>> df.select(pl.col("foo") * 2 + pl.col("bar")) # doctest: +IGNORE_RESULT
|
| match_to_schema(self, schema: 'SchemaDict | Schema', *, missing_columns: "Literal['insert', 'raise'] | Mapping[str, Literal['insert', 'raise'] | Expr]" = 'raise', missing_struct_fields: "Literal['insert', 'raise'] | Mapping[str, Literal['insert', 'raise']]" = 'raise', extra_columns: "Literal['ignore', 'raise']" = 'raise', extra_struct_fields: "Literal['ignore', 'raise'] | Mapping[str, Literal['ignore', 'raise']]" = 'raise', integer_cast: "Literal['upcast', 'forbid'] | Mapping[str, Literal['upcast', 'forbid']]" = 'forbid', float_cast: "Literal['upcast', 'forbid'] | Mapping[str, Literal['upcast', 'forbid']]" = 'forbid') -> 'DataFrame'
| Match or evolve the schema of a LazyFrame into a specific schema.
|
| By default, match_to_schema returns an error if the input schema does not
| exactly match the target schema. It also allows columns to be freely reordered,
| with additional coercion rules available through optional parameters.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| schema
| Target schema to match or evolve to.
| missing_columns
| Raise of insert missing columns from the input with respect to the `schema`.
|
| This can also be an expression per column with what to insert if it is
| missing.
| missing_struct_fields
| Raise of insert missing struct fields from the input with respect to the
| `schema`.
| extra_columns
| Raise of ignore extra columns from the input with respect to the `schema`.
| extra_struct_fields
| Raise of ignore extra struct fields from the input with respect to the
| `schema`.
| integer_cast
| Forbid of upcast for integer columns from the input to the respective column
| in `schema`.
| float_cast
| Forbid of upcast for float columns from the input to the respective column
| in `schema`.
|
| Examples
| --------
| Ensuring the schema matches
|
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": ["A", "B", "C"]})
| >>> df.match_to_schema({"a": pl.Int64, "b": pl.String})
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ A │
| │ 2 ┆ B │
| │ 3 ┆ C │
| └─────┴─────┘
| >>> df.match_to_schema({"a": pl.Int64}) # doctest: +SKIP
| polars.exceptions.SchemaError: extra columns in `match_to_schema`: "b"
|
| Adding missing columns
|
| >>> (
| ... pl.DataFrame({"a": [1, 2, 3]}).match_to_schema(
| ... {"a": pl.Int64, "b": pl.String},
| ... missing_columns="insert",
| ... )
| ... )
| shape: (3, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪══════╡
| │ 1 ┆ null │
| │ 2 ┆ null │
| │ 3 ┆ null │
| └─────┴──────┘
| >>> (
| ... pl.DataFrame({"a": [1, 2, 3]}).match_to_schema(
| ... {"a": pl.Int64, "b": pl.String},
| ... missing_columns={"b": pl.col.a.cast(pl.String)},
| ... )
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| └─────┴─────┘
|
| Removing extra columns
|
| >>> (
| ... pl.DataFrame({"a": [1, 2, 3], "b": ["A", "B", "C"]}).match_to_schema(
| ... {"a": pl.Int64},
| ... extra_columns="ignore",
| ... )
| ... )
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Upcasting integers and floats
|
| >>> (
| ... pl.DataFrame(
| ... {"a": [1, 2, 3], "b": [1.0, 2.0, 3.0]},
| ... schema={"a": pl.Int32, "b": pl.Float32},
| ... ).match_to_schema(
| ... {"a": pl.Int64, "b": pl.Float64},
| ... integer_cast="upcast",
| ... float_cast="upcast",
| ... )
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 1.0 │
| │ 2 ┆ 2.0 │
| │ 3 ┆ 3.0 │
| └─────┴─────┘
|
| max(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their maximum value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.max()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| max_horizontal(self) -> 'Series'
| Get the maximum value horizontally across columns.
|
| Returns
| -------
| Series
| A Series named `"max"`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4.0, 5.0, 6.0],
| ... }
| ... )
| >>> df.max_horizontal()
| shape: (3,)
| Series: 'max' [f64]
| [
| 4.0
| 5.0
| 6.0
| ]
|
| mean(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their mean value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... "spam": [True, False, None],
| ... }
| ... )
| >>> df.mean()
| shape: (1, 4)
| ┌─────┬─────┬──────┬──────┐
| │ foo ┆ bar ┆ ham ┆ spam │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str ┆ f64 │
| ╞═════╪═════╪══════╪══════╡
| │ 2.0 ┆ 7.0 ┆ null ┆ 0.5 │
| └─────┴─────┴──────┴──────┘
|
| mean_horizontal(self, *, ignore_nulls: 'bool' = True) -> 'Series'
| Take the mean of all values horizontally across columns.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default).
| If set to `False`, any null value in the input will lead to a null output.
|
| Returns
| -------
| Series
| A Series named `"mean"`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4.0, 5.0, 6.0],
| ... }
| ... )
| >>> df.mean_horizontal()
| shape: (3,)
| Series: 'mean' [f64]
| [
| 2.5
| 3.5
| 4.5
| ]
|
| median(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their median value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.median()
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 2.0 ┆ 7.0 ┆ null │
| └─────┴─────┴──────┘
|
| melt(self, id_vars: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, value_vars: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, variable_name: 'str | None' = None, value_name: 'str | None' = None) -> 'DataFrame'
| Unpivot a DataFrame from wide to long format.
|
| Optionally leaves identifiers set.
|
| This function is useful to massage a DataFrame into a format where one or more
| columns are identifier variables (id_vars) while all other columns, considered
| measured variables (value_vars), are "unpivoted" to the row axis leaving just
| two non-identifier columns, 'variable' and 'value'.
|
| .. deprecated:: 1.0.0
| Use the :meth:`.unpivot` method instead.
|
| Parameters
| ----------
| id_vars
| Column(s) or selector(s) to use as identifier variables.
| value_vars
| Column(s) or selector(s) to use as values variables; if `value_vars`
| is empty all columns that are not in `id_vars` will be used.
| variable_name
| Name to give to the `variable` column. Defaults to "variable"
| value_name
| Name to give to the `value` column. Defaults to "value"
|
| merge_sorted(self, other: 'DataFrame', key: 'str | Sequence[str]', *, maintain_order: 'bool' = False) -> 'DataFrame'
| Take two sorted DataFrames and merge them by the sorted key.
|
| The output of this operation will also be sorted.
| It is the callers responsibility that the frames
| are sorted in ascending order by the key(s), with null
| keys at the start, otherwise the order of the output
| will not make sense.
|
| The schemas of both DataFrames must be equal.
|
| Parameters
| ----------
| other
| Other DataFrame that must be merged
| key
| Key column(s) that the frames are sorted by. A single column name or a
| sequence of column names can be passed. When multiple keys are given the
| frames are merged as if sorted by those keys in order.
| maintain_order
| If ``True``, the output is guaranteed to have left-biased ordering
| for equal keys: rows from the left frame appear before rows from
| the right frame when their keys are equal.
|
| Examples
| --------
| >>> df0 = pl.DataFrame(
| ... {"name": ["steve", "elise", "bob"], "age": [42, 44, 18]}
| ... ).sort("age")
| >>> df0
| shape: (3, 2)
| ┌───────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═════╡
| │ bob ┆ 18 │
| │ steve ┆ 42 │
| │ elise ┆ 44 │
| └───────┴─────┘
| >>> df1 = pl.DataFrame(
| ... {"name": ["anna", "megan", "steve", "thomas"], "age": [21, 33, 42, 20]}
| ... ).sort("age")
| >>> df1
| shape: (4, 2)
| ┌────────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞════════╪═════╡
| │ thomas ┆ 20 │
| │ anna ┆ 21 │
| │ megan ┆ 33 │
| │ steve ┆ 42 │
| └────────┴─────┘
| >>> df0.merge_sorted(df1, key="age")
| shape: (7, 2)
| ┌────────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞════════╪═════╡
| │ bob ┆ 18 │
| │ thomas ┆ 20 │
| │ anna ┆ 21 │
| │ megan ┆ 33 │
| │ steve ┆ 42 │
| │ steve ┆ 42 │
| │ elise ┆ 44 │
| └────────┴─────┘
|
| Multiple keys can be passed to merge frames sorted by a composite key. The
| frames are merged as if sorted by ``key_1``, then ``key_2``.
|
| >>> df0 = pl.DataFrame({"key_1": [1, 1, 3], "key_2": [1, 4, 2]})
| >>> df1 = pl.DataFrame({"key_1": [1, 2, 3], "key_2": [2, 1, 1]})
| >>> df0.merge_sorted(df1, key=["key_1", "key_2"])
| shape: (6, 2)
| ┌───────┬───────┐
| │ key_1 ┆ key_2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪═══════╡
| │ 1 ┆ 1 │
| │ 1 ┆ 2 │
| │ 1 ┆ 4 │
| │ 2 ┆ 1 │
| │ 3 ┆ 1 │
| │ 3 ┆ 2 │
| └───────┴───────┘
|
| Notes
| -----
| Unless ``maintain_order=True``, no guarantee is given over the output
| row order when the key is equal between the both dataframes.
|
| The key(s) must be sorted in ascending order.
|
| min(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their minimum value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.min()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| min_horizontal(self) -> 'Series'
| Get the minimum value horizontally across columns.
|
| Returns
| -------
| Series
| A Series named `"min"`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4.0, 5.0, 6.0],
| ... }
| ... )
| >>> df.min_horizontal()
| shape: (3,)
| Series: 'min' [f64]
| [
| 1.0
| 2.0
| 3.0
| ]
|
| n_chunks(self, strategy: "Literal['first', 'all']" = 'first') -> 'int | list[int]'
| Get number of chunks used by the ChunkedArrays of this DataFrame.
|
| Parameters
| ----------
| strategy : {'first', 'all'}
| Return the number of chunks of the 'first' column,
| or 'all' columns in this DataFrame.
|
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> df.n_chunks()
| 1
| >>> df.n_chunks(strategy="all")
| [1, 1, 1]
|
| n_unique(self, subset: 'str | Expr | Sequence[str | Expr] | None' = None) -> 'int'
| Return the number of unique rows, or the number of unique row-subsets.
|
| Parameters
| ----------
| subset
| One or more columns/expressions that define what to count;
| omit to return the count of unique rows.
|
| Notes
| -----
| This method operates at the `DataFrame` level; to operate on subsets at the
| expression level you can make use of struct-packing instead, for example:
|
| >>> expr_unique_subset = pl.struct("a", "b").n_unique()
|
| If instead you want to count the number of unique values per-column, you can
| also use expression-level syntax to return a new frame containing that result:
|
| >>> df = pl.DataFrame(
| ... [[1, 2, 3], [1, 2, 4]], schema=["a", "b", "c"], orient="row"
| ... )
| >>> df_nunique = df.select(pl.all().n_unique())
|
| In aggregate context there is also an equivalent method for returning the
| unique values per-group:
|
| >>> df_agg_nunique = df.group_by("a").n_unique()
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 1, 2, 3, 4, 5],
| ... "b": [0.5, 0.5, 1.0, 2.0, 3.0, 3.0],
| ... "c": [True, True, True, False, True, True],
| ... }
| ... )
| >>> df.n_unique()
| 5
|
| Simple columns subset.
|
| >>> df.n_unique(subset=["b", "c"])
| 4
|
| Expression subset.
|
| >>> df.n_unique(
| ... subset=[
| ... (pl.col("a") // 2),
| ... (pl.col("c") | (pl.col("b") >= 2)),
| ... ],
| ... )
| 3
|
| null_count(self) -> 'DataFrame'
| Create a new DataFrame that shows the null counts per column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, None, 3],
| ... "bar": [6, 7, None],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.null_count()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 1 ┆ 0 │
| └─────┴─────┴─────┘
|
| partition_by(self, by: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector]', *more_by: 'ColumnNameOrSelector', maintain_order: 'bool' = True, include_key: 'bool' = True, as_dict: 'bool' = False) -> 'list[DataFrame] | dict[tuple[Any, ...], DataFrame]'
| Group by the given columns and return the groups as separate dataframes.
|
| Parameters
| ----------
| by
| Column name(s) or selector(s) to group by.
| *more_by
| Additional names of columns to group by, specified as positional arguments.
| maintain_order
| Ensure that the order of the groups is consistent with the input data.
| This is slower than a default partition by operation.
| include_key
| Include the columns used to partition the DataFrame in the output.
| as_dict
| Return a dictionary instead of a list. The dictionary keys are tuples of
| the distinct group values that identify each group.
|
| Examples
| --------
| Pass a single column name to partition by that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "c"],
| ... "b": [1, 2, 1, 3, 3],
| ... "c": [5, 4, 3, 2, 1],
| ... }
| ... )
| >>> df.partition_by("a") # doctest: +IGNORE_RESULT
| [shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ a ┆ 1 ┆ 3 │
| └─────┴─────┴─────┘,
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| └─────┴─────┴─────┘,
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘]
|
| Partition by multiple columns by either passing a list of column names, or by
| specifying each column name as a positional argument.
|
| >>> df.partition_by("a", "b") # doctest: +IGNORE_RESULT
| [shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ a ┆ 1 ┆ 3 │
| └─────┴─────┴─────┘,
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 2 ┆ 4 │
| └─────┴─────┴─────┘,
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 3 ┆ 2 │
| └─────┴─────┴─────┘,
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘]
|
| Return the partitions as a dictionary by specifying `as_dict=True`.
|
| >>> import polars.selectors as cs
| >>> df.partition_by(cs.string(), as_dict=True) # doctest: +IGNORE_RESULT
| {('a',): shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ a ┆ 1 ┆ 3 │
| └─────┴─────┴─────┘,
| ('b',): shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| └─────┴─────┴─────┘,
| ('c',): shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘}
|
| pipe(self, function: 'Callable[Concatenate[DataFrame, P], T]', *args: 'P.args', **kwargs: 'P.kwargs') -> 'T'
| Offers a structured way to apply a sequence of user-defined functions (UDFs).
|
| Parameters
| ----------
| function
| Callable; will receive the frame as the first parameter,
| followed by any given args/kwargs.
| *args
| Arguments to pass to the UDF.
| **kwargs
| Keyword arguments to pass to the UDF.
|
| Notes
| -----
| It is recommended to use LazyFrame when piping operations, in order
| to fully take advantage of query optimization and parallelization.
| See :meth:`df.lazy() <polars.DataFrame.lazy>`.
|
| Examples
| --------
| >>> def cast_str_to_int(data, col_name):
| ... return data.with_columns(pl.col(col_name).cast(pl.Int64))
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4], "b": ["10", "20", "30", "40"]})
| >>> df.pipe(cast_str_to_int, col_name="b")
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 10 │
| │ 2 ┆ 20 │
| │ 3 ┆ 30 │
| │ 4 ┆ 40 │
| └─────┴─────┘
|
| >>> df = pl.DataFrame({"b": [1, 2], "a": [3, 4]})
| >>> df
| shape: (2, 2)
| ┌─────┬─────┐
| │ b ┆ a │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
| >>> df.pipe(lambda tdf: tdf.select(sorted(tdf.columns)))
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 1 │
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| pivot(self, on: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector]', on_columns: 'Sequence[Any] | pl.Series | pl.DataFrame | None' = None, *, index: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, values: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, aggregate_function: 'PivotAgg | Expr | None' = None, maintain_order: 'bool' = True, sort_columns: 'bool' = False, separator: 'str' = '_', column_naming: "Literal['auto', 'combine']" = 'auto') -> 'DataFrame'
| Create a spreadsheet-style pivot table as a DataFrame.
|
| Only available in eager mode. See "Examples" section below for how to do a
| "lazy pivot" if you know the unique column values in advance.
|
| .. versionchanged:: 1.0.0
| The `columns` parameter was renamed `on`.
|
| Parameters
| ----------
| on
| The column(s) whose values will be used as the new columns of the output
| DataFrame.
| on_columns
| What value combinations will be considered for the output table.
| index
| The column(s) that remain from the input to the output. The output DataFrame will have one row
| for each unique combination of the `index`'s values.
| If None, all remaining columns not specified on `on` and `values` will be used. At least one
| of `index` and `values` must be specified.
| values
| The existing column(s) of values which will be moved under the new columns from index. If an
| aggregation is specified, these are the values on which the aggregation will be computed.
| If None, all remaining columns not specified on `on` and `index` will be used.
| At least one of `index` and `values` must be specified.
| aggregate_function
| Choose from:
|
| - None: no aggregation takes place, will raise error if multiple values are in group.
| - A predefined aggregate function string, one of
| {'min', 'max', 'first', 'last', 'sum', 'mean', 'median', 'len'}
| - An expression to do the aggregation. The expression can only access data from the respective
| 'values' columns as generated by pivot, through `pl.element()`.
| maintain_order
| Ensure the values of `index` are sorted by discovery order.
| sort_columns
| Sort the transposed columns by name. Default is by order of discovery.
| separator
| Used as separator/delimiter in generated column names in case of multiple
| `values` columns.
| column_naming : {'auto', 'combine'}
| How resulting column names will be constructed.
|
| * 'auto': The default; combine with separator if there are multiple
| `values` columns, otherwise just use the `on_columns` names.
| * 'combine': Always combine the `values` columns' names with
| the `on_columns` names.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Notes
| -----
| In some other frameworks, you might know this operation as `pivot_wider`.
|
| Examples
| --------
| You can use `pivot` to reshape a dataframe from "long" to "wide" format.
|
| For example, suppose we have a dataframe of test scores achieved by some
| students, where each row represents a distinct test.
|
| >>> df = pl.DataFrame(
| ... {
| ... "name": ["Cady", "Cady", "Karen", "Karen"],
| ... "subject": ["maths", "physics", "maths", "physics"],
| ... "test_1": [98, 99, 61, 58],
| ... "test_2": [100, 100, 60, 60],
| ... }
| ... )
| >>> df
| shape: (4, 4)
| ┌───────┬─────────┬────────┬────────┐
| │ name ┆ subject ┆ test_1 ┆ test_2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 ┆ i64 │
| ╞═══════╪═════════╪════════╪════════╡
| │ Cady ┆ maths ┆ 98 ┆ 100 │
| │ Cady ┆ physics ┆ 99 ┆ 100 │
| │ Karen ┆ maths ┆ 61 ┆ 60 │
| │ Karen ┆ physics ┆ 58 ┆ 60 │
| └───────┴─────────┴────────┴────────┘
|
| Using `pivot`, we can reshape so we have one row per student, with different
| subjects as columns, and their `test_1` scores as values:
|
| >>> df.pivot("subject", index="name", values="test_1")
| shape: (2, 3)
| ┌───────┬───────┬─────────┐
| │ name ┆ maths ┆ physics │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════╪═══════╪═════════╡
| │ Cady ┆ 98 ┆ 99 │
| │ Karen ┆ 61 ┆ 58 │
| └───────┴───────┴─────────┘
|
| If you want to only pivot over a limited set of `subject` values or already
| know the `subject` values ahead of time, you can provide these using the
| `on_columns` argument.
|
| >>> df.pivot(
| ... "subject",
| ... on_columns=["maths", "physics"],
| ... index="name",
| ... values="test_1",
| ... )
| shape: (2, 3)
| ┌───────┬───────┬─────────┐
| │ name ┆ maths ┆ physics │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════╪═══════╪═════════╡
| │ Cady ┆ 98 ┆ 99 │
| │ Karen ┆ 61 ┆ 58 │
| └───────┴───────┴─────────┘
|
| You can use selectors too - here we include all test scores in the pivoted table:
|
| >>> import polars.selectors as cs
| >>> df.pivot("subject", values=cs.starts_with("test"))
| shape: (2, 5)
| ┌───────┬──────────────┬────────────────┬──────────────┬────────────────┐
| │ name ┆ test_1_maths ┆ test_1_physics ┆ test_2_maths ┆ test_2_physics │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═══════╪══════════════╪════════════════╪══════════════╪════════════════╡
| │ Cady ┆ 98 ┆ 99 ┆ 100 ┆ 100 │
| │ Karen ┆ 61 ┆ 58 ┆ 60 ┆ 60 │
| └───────┴──────────────┴────────────────┴──────────────┴────────────────┘
|
| If you end up with multiple values per cell, you can specify how to aggregate
| them with `aggregate_function`:
|
| >>> df = pl.DataFrame(
| ... {
| ... "ix": [1, 1, 2, 2, 1, 2],
| ... "col": ["a", "a", "a", "a", "b", "b"],
| ... "foo": [0, 1, 2, 2, 7, 1],
| ... "bar": [0, 2, 0, 0, 9, 4],
| ... }
| ... )
| >>> df.pivot("col", index="ix", aggregate_function="sum")
| shape: (2, 5)
| ┌─────┬───────┬───────┬───────┬───────┐
| │ ix ┆ foo_a ┆ foo_b ┆ bar_a ┆ bar_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═══════╪═══════╪═══════╪═══════╡
| │ 1 ┆ 1 ┆ 7 ┆ 2 ┆ 9 │
| │ 2 ┆ 4 ┆ 1 ┆ 0 ┆ 4 │
| └─────┴───────┴───────┴───────┴───────┘
|
| You can also pass a custom aggregation function using
| :meth:`polars.element`:
|
| >>> df = pl.DataFrame(
| ... {
| ... "col1": ["a", "a", "a", "b", "b", "b"],
| ... "col2": ["x", "x", "x", "x", "y", "y"],
| ... "col3": [6, 7, 3, 2, 5, 7],
| ... }
| ... )
| >>> df.pivot(
| ... "col2",
| ... index="col1",
| ... values="col3",
| ... aggregate_function=pl.element().tanh().mean(),
| ... )
| shape: (2, 3)
| ┌──────┬──────────┬──────────┐
| │ col1 ┆ x ┆ y │
| │ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 │
| ╞══════╪══════════╪══════════╡
| │ a ┆ 0.998347 ┆ null │
| │ b ┆ 0.964028 ┆ 0.999954 │
| └──────┴──────────┴──────────┘
|
| See Also
| --------
| LazyFrame.pivot
|
| product(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their product values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": [0.5, 4, 10],
| ... "c": [True, True, False],
| ... }
| ... )
|
| >>> df.product()
| shape: (1, 3)
| ┌─────┬──────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ i64 │
| ╞═════╪══════╪═════╡
| │ 6 ┆ 20.0 ┆ 0 │
| └─────┴──────┴─────┘
|
| quantile(self, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest') -> 'DataFrame'
| Aggregate the columns of this DataFrame to their quantile value.
|
| Parameters
| ----------
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.quantile(0.5, "nearest")
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 2.0 ┆ 7.0 ┆ null │
| └─────┴─────┴──────┘
|
| rechunk(self) -> 'DataFrame'
| Rechunk the data in this DataFrame to a contiguous allocation.
|
| This will make sure all subsequent operations have optimal and predictable
| performance.
|
| remove(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]', **constraints: 'Any') -> 'DataFrame'
| Remove rows, dropping those that match the given predicate expression(s).
|
| The original order of the remaining rows is preserved.
|
| Rows where the filter predicate does not evaluate to True are retained
| (this includes rows where the predicate evaluates as `null`).
|
| Parameters
| ----------
| predicates
| Expression(s) that evaluate to a boolean Series. When multiple predicates
| are provided they are combined using `&` (logical AND), so a row is only
| removed when *every* predicate evaluates to True for that row.
| constraints
| Column filters; use `name = value` to filter columns using the supplied
| value. Each constraint behaves the same as `pl.col(name).eq(value)`,
| and is implicitly joined with the other filter conditions using `&`.
|
| Notes
| -----
| If you are transitioning from Pandas, and performing filter operations based on
| the comparison of two or more columns, please note that in Polars any comparison
| involving `null` values will result in a `null` result, *not* boolean True or
| False. As a result, these rows will not be removed. Ensure that null values
| are handled appropriately to avoid unexpected behaviour (see examples below).
|
| See Also
| --------
| filter
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [2, 3, None, 4, 0],
| ... "bar": [5, 6, None, None, 0],
| ... "ham": ["a", "b", None, "c", "d"],
| ... }
| ... )
|
| Remove rows matching a condition:
|
| >>> df.remove(pl.col("bar") >= 5)
| shape: (3, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| │ 0 ┆ 0 ┆ d │
| └──────┴──────┴──────┘
|
| Discard rows based on multiple conditions, combined with and/or operators:
|
| >>> df.remove(
| ... (pl.col("foo") >= 0) & (pl.col("bar") >= 0),
| ... )
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| >>> df.remove(
| ... (pl.col("foo") >= 0) | (pl.col("bar") >= 0),
| ... )
| shape: (1, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| └──────┴──────┴──────┘
|
| Provide multiple constraints using `*args` syntax:
|
| >>> df.remove(
| ... pl.col("ham").is_not_null(),
| ... pl.col("bar") >= 0,
| ... )
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| Provide constraints(s) using `**kwargs` syntax:
|
| >>> df.remove(foo=0, bar=0)
| shape: (4, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ 2 ┆ 5 ┆ a │
| │ 3 ┆ 6 ┆ b │
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| Remove rows by comparing two columns against each other:
|
| >>> df.remove(
| ... pl.col("foo").ne_missing(pl.col("bar")),
| ... )
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 0 ┆ 0 ┆ d │
| └──────┴──────┴──────┘
|
| rename(self, mapping: 'Mapping[str, str] | Callable[[str], str]', *, strict: 'bool' = True) -> 'DataFrame'
| Rename column names.
|
| Parameters
| ----------
| mapping
| Key value pairs that map from old name to new name, or a function
| that takes the old name as input and returns the new name.
| strict
| Validate that all column names exist in the current schema,
| and throw an exception if any do not. (Note that this parameter
| is a no-op when passing a function to `mapping`).
|
| See Also
| --------
| Expr.name.replace
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"foo": [1, 2, 3], "bar": [6, 7, 8], "ham": ["a", "b", "c"]}
| ... )
| >>> df.rename({"foo": "apple"})
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ apple ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═══════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └───────┴─────┴─────┘
| >>> df.rename(lambda column_name: "c" + column_name[1:])
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ coo ┆ car ┆ cam │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| replace_column(self, index: 'int', column: 'Series') -> 'DataFrame'
| Replace a column at an index location.
|
| This operation is in place.
|
| Parameters
| ----------
| index
| Column index.
| column
| Series that will replace the column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> s = pl.Series("apple", [10, 20, 30])
| >>> df.replace_column(0, s)
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ apple ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═══════╪═════╪═════╡
| │ 10 ┆ 6 ┆ a │
| │ 20 ┆ 7 ┆ b │
| │ 30 ┆ 8 ┆ c │
| └───────┴─────┴─────┘
|
| reverse(self) -> 'DataFrame'
| Reverse the DataFrame.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "key": ["a", "b", "c"],
| ... "val": [1, 2, 3],
| ... }
| ... )
| >>> df.reverse()
| shape: (3, 2)
| ┌─────┬─────┐
| │ key ┆ val │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ c ┆ 3 │
| │ b ┆ 2 │
| │ a ┆ 1 │
| └─────┴─────┘
|
| rolling(self, index_column: 'IntoExpr', *, period: 'str | timedelta', offset: 'str | timedelta | None' = None, closed: 'ClosedInterval' = 'right', group_by: 'IntoExpr | Iterable[IntoExpr] | None' = None) -> 'RollingGroupBy'
| Create rolling groups based on a temporal or integer column.
|
| Different from a `group_by_dynamic` the windows are now determined by the
| individual values and are not of constant intervals. For constant intervals use
| :func:`DataFrame.group_by_dynamic`.
|
| If you have a time series `<t_0, t_1, ..., t_n>`, then by default the
| windows created will be
|
| * (t_0 - period, t_0]
| * (t_1 - period, t_1]
| * ...
| * (t_n - period, t_n]
|
| whereas if you pass a non-default `offset`, then the windows will be
|
| * (t_0 + offset, t_0 + offset + period]
| * (t_1 + offset, t_1 + offset + period]
| * ...
| * (t_n + offset, t_n + offset + period]
|
| The `period` and `offset` arguments are created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order (or, if `group_by` is
| specified, then it must be sorted in ascending order within each group).
|
| In case of a rolling operation on indices, dtype needs to be one of
| {UInt32, UInt64, Int32, Int64}. Note that the first three get temporarily
| cast to Int64, so if performance matters use an Int64 column.
| period
| Length of the window - must be non-negative.
| offset
| Offset of the window. Default is `-period`.
| closed : {'right', 'left', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
| group_by
| Also group by this column/these columns
|
| Returns
| -------
| RollingGroupBy
| Object you can call `.agg` on to aggregate by groups, the result
| of which will be sorted by `index_column` (but note that if `group_by`
| columns are passed, it will only be sorted within each group).
|
| See Also
| --------
| group_by_dynamic
|
| Examples
| --------
| >>> dates = [
| ... "2020-01-01 13:45:48",
| ... "2020-01-01 16:42:13",
| ... "2020-01-01 16:45:09",
| ... "2020-01-02 18:12:48",
| ... "2020-01-03 19:45:32",
| ... "2020-01-08 23:16:43",
| ... ]
| >>> df = pl.DataFrame({"dt": dates, "a": [3, 7, 5, 9, 2, 1]}).with_columns(
| ... pl.col("dt").str.strptime(pl.Datetime).set_sorted()
| ... )
| >>> out = df.rolling(index_column="dt", period="2d").agg(
| ... [
| ... pl.sum("a").alias("sum_a"),
| ... pl.min("a").alias("min_a"),
| ... pl.max("a").alias("max_a"),
| ... ]
| ... )
| >>> assert out["sum_a"].to_list() == [3, 10, 15, 24, 11, 1]
| >>> assert out["max_a"].to_list() == [3, 7, 7, 9, 9, 1]
| >>> assert out["min_a"].to_list() == [3, 3, 3, 3, 2, 1]
| >>> out
| shape: (6, 4)
| ┌─────────────────────┬───────┬───────┬───────┐
| │ dt ┆ sum_a ┆ min_a ┆ max_a │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ i64 ┆ i64 │
| ╞═════════════════════╪═══════╪═══════╪═══════╡
| │ 2020-01-01 13:45:48 ┆ 3 ┆ 3 ┆ 3 │
| │ 2020-01-01 16:42:13 ┆ 10 ┆ 3 ┆ 7 │
| │ 2020-01-01 16:45:09 ┆ 15 ┆ 3 ┆ 7 │
| │ 2020-01-02 18:12:48 ┆ 24 ┆ 3 ┆ 9 │
| │ 2020-01-03 19:45:32 ┆ 11 ┆ 2 ┆ 9 │
| │ 2020-01-08 23:16:43 ┆ 1 ┆ 1 ┆ 1 │
| └─────────────────────┴───────┴───────┴───────┘
|
| If you use an index count in `period` or `offset`, then it's based on the
| values in `index_column`:
|
| >>> df = pl.DataFrame({"int": [0, 4, 5, 6, 8], "value": [1, 4, 2, 4, 1]})
| >>> df.rolling("int", period="3i").agg(pl.col("int").alias("aggregated"))
| shape: (5, 2)
| ┌─────┬────────────┐
| │ int ┆ aggregated │
| │ --- ┆ --- │
| │ i64 ┆ list[i64] │
| ╞═════╪════════════╡
| │ 0 ┆ [0] │
| │ 4 ┆ [4] │
| │ 5 ┆ [4, 5] │
| │ 6 ┆ [4, 5, 6] │
| │ 8 ┆ [6, 8] │
| └─────┴────────────┘
|
| If you want the index count to be based on row number, then you may want to
| combine `rolling` with :meth:`.with_row_index`.
|
| row(self, index: 'int | None' = None, *, by_predicate: 'Expr | None' = None, named: 'bool' = False) -> 'tuple[Any, ...] | dict[str, Any]'
| Get the values of a single row, either by index or by predicate.
|
| Parameters
| ----------
| index
| Row index.
| by_predicate
| Select the row according to a given expression/predicate.
| named
| Return a dictionary instead of a tuple. The dictionary is a mapping of
| column name to row value. This is more expensive than returning a regular
| tuple, but allows for accessing values by column name.
|
| Returns
| -------
| tuple (default) or dictionary of row values
|
| Notes
| -----
| The `index` and `by_predicate` params are mutually exclusive. Additionally,
| to ensure clarity, the `by_predicate` parameter must be supplied by keyword.
|
| When using `by_predicate` it is an error condition if anything other than
| one row is returned; more than one row raises `TooManyRowsReturnedError`, and
| zero rows will raise `NoRowsReturnedError` (both inherit from `RowsError`).
|
| Warnings
| --------
| You should NEVER use this method to iterate over a DataFrame; if you require
| row-iteration you should strongly prefer use of `iter_rows()` instead.
|
| See Also
| --------
| iter_rows : Row iterator over frame data (does not materialise all rows).
| rows : Materialise all frame data as a list of rows (potentially expensive).
| item: Return dataframe element as a scalar.
|
| Examples
| --------
| Specify an index to return the row at the given index as a tuple.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.row(2)
| (3, 8, 'c')
|
| Specify `named=True` to get a dictionary instead with a mapping of column
| names to row values.
|
| >>> df.row(2, named=True)
| {'foo': 3, 'bar': 8, 'ham': 'c'}
|
| Use `by_predicate` to return the row that matches the given predicate.
|
| >>> df.row(by_predicate=(pl.col("ham") == "b"))
| (2, 7, 'b')
|
| rows(self, *, named: 'bool' = False) -> 'list[tuple[Any, ...]] | list[dict[str, Any]]'
| Returns all data in the DataFrame as a list of rows of python-native values.
|
| By default, each row is returned as a tuple of values given in the same order
| as the frame columns. Setting `named=True` will return rows of dictionaries
| instead.
|
| Parameters
| ----------
| named
| Return dictionaries instead of tuples. The dictionaries are a mapping of
| column name to row value. This is more expensive than returning a regular
| tuple, but allows for accessing values by column name.
|
| Notes
| -----
| If you have `ns`-precision temporal values you should be aware that Python
| natively only supports up to `μs`-precision; `ns`-precision values will be
| truncated to microseconds on conversion to Python. If this matters to your
| use-case you should export to a different format (such as Arrow or NumPy).
|
| Warnings
| --------
| Row-iteration is not optimal as the underlying data is stored in columnar form;
| where possible, prefer export via one of the dedicated export/output methods.
| You should also consider using `iter_rows` instead, to avoid materialising all
| the data at once; there is little performance difference between the two, but
| peak memory can be reduced if processing rows in batches.
|
| Returns
| -------
| list of row value tuples (default), or list of dictionaries (if `named=True`).
|
| See Also
| --------
| iter_rows : Row iterator over frame data (does not materialise all rows).
| rows_by_key : Materialises frame data as a key-indexed dictionary.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "x": ["a", "b", "b", "a"],
| ... "y": [1, 2, 3, 4],
| ... "z": [0, 3, 6, 9],
| ... }
| ... )
| >>> df.rows()
| [('a', 1, 0), ('b', 2, 3), ('b', 3, 6), ('a', 4, 9)]
| >>> df.rows(named=True)
| [{'x': 'a', 'y': 1, 'z': 0},
| {'x': 'b', 'y': 2, 'z': 3},
| {'x': 'b', 'y': 3, 'z': 6},
| {'x': 'a', 'y': 4, 'z': 9}]
|
| rows_by_key(self, key: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector]', *, named: 'bool' = False, include_key: 'bool' = False, unique: 'bool' = False) -> 'dict[Any, Any]'
| Returns all data as a dictionary of python-native values keyed by some column.
|
| This method is like `rows`, but instead of returning rows in a flat list, rows
| are grouped by the values in the `key` column(s) and returned as a dictionary.
|
| Note that this method should not be used in place of native operations, due to
| the high cost of materializing all frame data out into a dictionary; it should
| be used only when you need to move the values out into a Python data structure
| or other object that cannot operate directly with Polars/Arrow.
|
| Parameters
| ----------
| key
| The column(s) to use as the key for the returned dictionary. If multiple
| columns are specified, the key will be a tuple of those values, otherwise
| it will be a string.
| named
| Return dictionary rows instead of tuples, mapping column name to row value.
| include_key
| Include key values inline with the associated data (by default the key
| values are omitted as a memory/performance optimisation, as they can be
| reoconstructed from the key).
| unique
| Indicate that the key is unique; this will result in a 1:1 mapping from
| key to a single associated row. Note that if the key is *not* actually
| unique the last row with the given key will be returned.
|
| Notes
| -----
| If you have `ns`-precision temporal values you should be aware that Python
| natively only supports up to `μs`-precision; `ns`-precision values will be
| truncated to microseconds on conversion to Python. If this matters to your
| use-case you should export to a different format (such as Arrow or NumPy).
|
| See Also
| --------
| rows : Materialize all frame data as a list of rows (potentially expensive).
| iter_rows : Row iterator over frame data (does not materialize all rows).
| to_dict : Convert DataFrame to a dictionary mapping column name to values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "w": ["a", "b", "b", "a"],
| ... "x": ["q", "q", "q", "k"],
| ... "y": [1.0, 2.5, 3.0, 4.5],
| ... "z": [9, 8, 7, 6],
| ... }
| ... )
|
| Group rows by the given key column(s):
|
| >>> df.rows_by_key(key=["w"])
| defaultdict(<class 'list'>,
| {'a': [('q', 1.0, 9), ('k', 4.5, 6)],
| 'b': [('q', 2.5, 8), ('q', 3.0, 7)]})
|
| Return the same row groupings as dictionaries:
|
| >>> df.rows_by_key(key=["w"], named=True)
| defaultdict(<class 'list'>,
| {'a': [{'x': 'q', 'y': 1.0, 'z': 9},
| {'x': 'k', 'y': 4.5, 'z': 6}],
| 'b': [{'x': 'q', 'y': 2.5, 'z': 8},
| {'x': 'q', 'y': 3.0, 'z': 7}]})
|
| Return row groupings, assuming keys are unique:
|
| >>> df.rows_by_key(key=["z"], unique=True)
| {9: ('a', 'q', 1.0),
| 8: ('b', 'q', 2.5),
| 7: ('b', 'q', 3.0),
| 6: ('a', 'k', 4.5)}
|
| Return row groupings as dictionaries, assuming keys are unique:
|
| >>> df.rows_by_key(key=["z"], named=True, unique=True)
| {9: {'w': 'a', 'x': 'q', 'y': 1.0},
| 8: {'w': 'b', 'x': 'q', 'y': 2.5},
| 7: {'w': 'b', 'x': 'q', 'y': 3.0},
| 6: {'w': 'a', 'x': 'k', 'y': 4.5}}
|
| Return dictionary rows grouped by a compound key, including key values:
|
| >>> df.rows_by_key(key=["w", "x"], named=True, include_key=True)
| defaultdict(<class 'list'>,
| {('a', 'q'): [{'w': 'a', 'x': 'q', 'y': 1.0, 'z': 9}],
| ('b', 'q'): [{'w': 'b', 'x': 'q', 'y': 2.5, 'z': 8},
| {'w': 'b', 'x': 'q', 'y': 3.0, 'z': 7}],
| ('a', 'k'): [{'w': 'a', 'x': 'k', 'y': 4.5, 'z': 6}]})
|
| sample(self, n: 'int | Series | None' = None, *, fraction: 'float | Series | None' = None, with_replacement: 'bool' = False, shuffle: 'bool | None' = None, seed: 'int | None' = None) -> 'DataFrame'
| Sample from this DataFrame.
|
| Parameters
| ----------
| n
| Number of items to return. Cannot be used with `fraction`. Defaults to 1 if
| `fraction` is None.
| fraction
| Fraction of items to return. Cannot be used with `n`.
| with_replacement
| Allow values to be sampled more than once.
| shuffle
| Determines the order of the sampled rows.
| If True, sampled rows are explicitly shuffled.
| If False, the relative order of the sampled rows is preserved.
| (i.e. they appear in the same order as the original DataFrame).
| If None (default), no ordering guarantee; uses the most performant
| algorithm.
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated for each time the sample is called.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.sample(n=2, shuffle=False, seed=0) # doctest: +IGNORE_RESULT
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8 ┆ c │
| │ 2 ┆ 7 ┆ b │
| └─────┴─────┴─────┘
|
| select(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'DataFrame'
| Select columns from this DataFrame.
|
| Parameters
| ----------
| *exprs
| Column(s) to select, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names,
| other non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to select, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Examples
| --------
| Pass the name of a column to select that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.select("foo")
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Multiple columns can be selected by passing a list of column names.
|
| >>> df.select(["foo", "bar"])
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 6 │
| │ 2 ┆ 7 │
| │ 3 ┆ 8 │
| └─────┴─────┘
|
| Multiple columns can also be selected using positional arguments instead of a
| list. Expressions are also accepted.
|
| >>> df.select(pl.col("foo"), pl.col("bar") + 1)
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| └─────┴─────┘
|
| Use keyword arguments to easily name your expression inputs.
|
| >>> df.select(threshold=pl.when(pl.col("foo") > 2).then(10).otherwise(0))
| shape: (3, 1)
| ┌───────────┐
| │ threshold │
| │ --- │
| │ i32 │
| ╞═══════════╡
| │ 0 │
| │ 0 │
| │ 10 │
| └───────────┘
|
| select_seq(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'DataFrame'
| Select columns from this DataFrame.
|
| This will run all expression sequentially instead of in parallel.
| Use this when the work per expression is cheap.
|
| Parameters
| ----------
| *exprs
| Column(s) to select, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names,
| other non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to select, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| See Also
| --------
| select
|
| serialize(self, file: 'IOBase | str | Path | None' = None, *, format: 'SerializationFormat' = 'binary') -> 'bytes | str | None'
| Serialize this DataFrame to a file or string in JSON format.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| If set to `None` (default), the output is returned as a string instead.
| format
| The format in which to serialize. Options:
|
| - `"binary"`: Serialize to binary format (bytes). This is the default.
| - `"json"`: Serialize to JSON format (string).
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| Serialize the DataFrame into a binary representation.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... }
| ... )
| >>> bytes = df.serialize()
| >>> type(bytes)
| <class 'bytes'>
|
| The bytes can later be deserialized back into a DataFrame.
|
| >>> import io
| >>> pl.DataFrame.deserialize(io.BytesIO(bytes))
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 6 │
| │ 2 ┆ 7 │
| │ 3 ┆ 8 │
| └─────┴─────┘
|
| set_sorted(self, column: 'str', *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'DataFrame'
| Flag a column as sorted.
|
| This can speed up future operations.
|
| Parameters
| ----------
| column
| Column that is sorted
| descending
| Whether the column is sorted in descending order.
| nulls_last
| Whether the nulls are at the end.
|
| Warnings
| --------
| This can lead to incorrect results if the data is NOT sorted!!
| Use with care!
|
| shift(self, n: 'int' = 1, *, fill_value: 'IntoExpr | None' = None) -> 'DataFrame'
| Shift values by the given number of indices.
|
| Parameters
| ----------
| n
| Number of indices to shift forward. If a negative value is passed, values
| are shifted in the opposite direction instead.
| fill_value
| Fill the resulting null values with this value. Accepts scalar expression
| input. Non-expression inputs are parsed as literals.
|
| Notes
| -----
| This method is similar to the `LAG` operation in SQL when the value for `n`
| is positive. With a negative value for `n`, it is similar to `LEAD`.
|
| Examples
| --------
| By default, values are shifted forward by one index.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [5, 6, 7, 8],
| ... }
| ... )
| >>> df.shift()
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ null ┆ null │
| │ 1 ┆ 5 │
| │ 2 ┆ 6 │
| │ 3 ┆ 7 │
| └──────┴──────┘
|
| Pass a negative value to shift in the opposite direction instead.
|
| >>> df.shift(-2)
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ 3 ┆ 7 │
| │ 4 ┆ 8 │
| │ null ┆ null │
| │ null ┆ null │
| └──────┴──────┘
|
| Specify `fill_value` to fill the resulting null values.
|
| >>> df.shift(-2, fill_value=100)
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 7 │
| │ 4 ┆ 8 │
| │ 100 ┆ 100 │
| │ 100 ┆ 100 │
| └─────┴─────┘
|
| show(self, limit: 'int | None' = 5, *, ascii_tables: 'bool | None' = None, decimal_separator: 'str | None' = None, thousands_separator: 'str | bool | None' = None, float_precision: 'int | None' = None, fmt_float: 'FloatFmt | None' = None, fmt_str_lengths: 'int | None' = None, fmt_table_cell_list_len: 'int | None' = None, tbl_cell_alignment: 'Alignment | None' = None, tbl_cell_numeric_alignment: 'Alignment | None' = None, tbl_cols: 'int | None' = None, tbl_column_data_type_inline: 'bool | None' = None, tbl_dataframe_shape_below: 'bool | None' = None, tbl_formatting: 'TableFormatNames | None' = None, tbl_hide_column_data_types: 'bool | None' = None, tbl_hide_column_names: 'bool | None' = None, tbl_hide_dtype_separator: 'bool | None' = None, tbl_hide_dataframe_shape: 'bool | None' = None, tbl_width_chars: 'int | None' = None, trim_decimal_zeros: 'bool | None' = True) -> 'None'
| Show the first `n` rows.
|
| Parameters
| ----------
| limit : int
| Numbers of rows to show. If a negative value is passed, return all rows
| except the last `abs(n)`. If None is passed, return all rows.
| ascii_tables : bool
| Use ASCII characters to display table outlines. Set False to revert to the
| default UTF8_FULL_CONDENSED formatting style. See
| :func:`Config.set_ascii_tables` for more information.
| decimal_separator : str
| Set the decimal separator character. See
| :func:`Config.set_decimal_separator` for more information.
| thousands_separator : str, bool
| Set the thousands grouping separator character. See
| :func:`Config.set_thousands_separator` for more information.
| float_precision : int
| Number of decimal places to display for floating point values. See
| :func:`Config.set_float_precision` for more information.
| fmt_float : {"mixed", "full"}
| Control how floating point values are displayed. See
| :func:`Config.set_fmt_float` for more information. Supported options are:
|
| * "mixed": Limit the number of decimal places and use scientific notation
| for large/small values.
| * "full": Print the full precision of the floating point number.
|
| fmt_str_lengths : int
| Number of characters to display for string values. See
| :func:`Config.set_fmt_str_lengths` for more information.
| fmt_table_cell_list_len : int
| Number of elements to display for List values. See
| :func:`Config.set_fmt_table_cell_list_len` for more information.
| tbl_cell_alignment : str
| Set table cell alignment. See :func:`Config.set_tbl_cell_alignment` for more
| information. Supported options are:
|
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| tbl_cell_numeric_alignment : str
| Set table cell alignment for numeric columns. See
| :func:`Config.set_tbl_cell_numeric_alignment` for more information.
| Supported options are:
|
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| tbl_cols : int
| Number of columns to display. See :func:`Config.set_tbl_cols` for more
| information.
| tbl_column_data_type_inline : bool
| Moves the data type inline with the column name (to the right, in
| parentheses). See :func:`Config.set_tbl_column_data_type_inline` for more
| information.
| tbl_dataframe_shape_below : bool
| Print the DataFrame shape information below the data when displaying tables.
| See :func:`Config.set_tbl_dataframe_shape_below` for more information.
| tbl_formatting : str
| Set table formatting style. See :func:`Config.set_tbl_formatting` for more
| information. Supported options are:
|
| * "ASCII_FULL": ASCII, with all borders and lines, including row dividers.
| * "ASCII_FULL_CONDENSED": Same as ASCII_FULL, but with dense row spacing.
| * "ASCII_NO_BORDERS": ASCII, no borders.
| * "ASCII_BORDERS_ONLY": ASCII, borders only.
| * "ASCII_BORDERS_ONLY_CONDENSED": ASCII, borders only, dense row spacing.
| * "ASCII_HORIZONTAL_ONLY": ASCII, horizontal lines only.
| * "ASCII_MARKDOWN": Markdown format (ascii ellipses for truncated values).
| * "MARKDOWN": Markdown format (utf8 ellipses for truncated values).
| * "UTF8_FULL": UTF8, with all borders and lines, including row dividers.
| * "UTF8_FULL_CONDENSED": Same as UTF8_FULL, but with dense row spacing.
| * "UTF8_NO_BORDERS": UTF8, no borders.
| * "UTF8_BORDERS_ONLY": UTF8, borders only.
| * "UTF8_HORIZONTAL_ONLY": UTF8, horizontal lines only.
| * "NOTHING": No borders or other lines.
|
| tbl_hide_column_data_types : bool
| Hide table column data types (i64, f64, str etc.). See
| :func:`Config.set_tbl_hide_column_data_types` for more information.
| tbl_hide_column_names : bool
| Hide table column names. See :func:`Config.set_tbl_hide_column_names` for
| more information.
| tbl_hide_dtype_separator : bool
| Hide the '---' separator between the column names and column types. See
| :func:`Config.set_tbl_hide_dtype_separator` for more information.
| tbl_hide_dataframe_shape : bool
| Hide the DataFrame shape information when displaying tables. See
| :func:`Config.set_tbl_hide_dataframe_shape` for more information.
| tbl_width_chars : int
| Set the maximum width of a table in characters. See
| :func:`Config.set_tbl_width_chars` for more information.
| trim_decimal_zeros : bool
| Strip trailing zeros from Decimal data type values. See
| :func:`Config.set_trim_decimal_zeros` for more information.
|
| See Also
| --------
| head
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> df.show(3)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Pass a negative value to get all rows `except` the last `abs(n)`.
|
| >>> df.show(-3)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| └─────┴─────┴─────┘
|
| shrink_to_fit(self, *, in_place: 'bool' = False) -> 'DataFrame'
| Shrink DataFrame memory usage.
|
| Shrinks to fit the exact capacity needed to hold the data.
|
| slice(self, offset: 'int', length: 'int | None' = None) -> 'DataFrame'
| Get a slice of this DataFrame.
|
| Parameters
| ----------
| offset
| Start index. Negative indexing is supported.
| length
| Length of the slice. If set to `None`, all rows starting at the offset
| will be selected.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.slice(1, 2)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ 7.0 ┆ b │
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| sort(self, by: 'IntoExpr | Iterable[IntoExpr]', *more_by: 'IntoExpr', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False, multithreaded: 'bool' = True, maintain_order: 'bool' = False) -> 'DataFrame'
| Sort the dataframe by the given columns.
|
| Parameters
| ----------
| by
| Column(s) to sort by. Accepts expression input, including selectors. Strings
| are parsed as column names.
| *more_by
| Additional columns to sort by, specified as positional arguments.
| descending
| Sort in descending order. When sorting by multiple columns, can be specified
| per column by passing a sequence of booleans.
| nulls_last
| Place null values last; can specify a single boolean applying to all columns
| or a sequence of booleans for per-column control.
| multithreaded
| Sort using multiple threads.
| maintain_order
| Whether the order should be maintained if elements are equal.
|
| Examples
| --------
| Pass a single column name to sort by that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": [6.0, 5.0, 4.0],
| ... "c": ["a", "c", "b"],
| ... }
| ... )
| >>> df.sort("a")
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ null ┆ 4.0 ┆ b │
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 5.0 ┆ c │
| └──────┴─────┴─────┘
|
| Sorting by expressions is also supported.
|
| >>> df.sort(pl.col("a") + pl.col("b") * 2, nulls_last=True)
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 2 ┆ 5.0 ┆ c │
| │ 1 ┆ 6.0 ┆ a │
| │ null ┆ 4.0 ┆ b │
| └──────┴─────┴─────┘
|
| Sort by multiple columns by passing a list of columns.
|
| >>> df.sort(["c", "a"], descending=True)
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 2 ┆ 5.0 ┆ c │
| │ null ┆ 4.0 ┆ b │
| │ 1 ┆ 6.0 ┆ a │
| └──────┴─────┴─────┘
|
| Or use positional arguments to sort by multiple columns in the same way.
|
| >>> df.sort("c", "a", descending=[False, True])
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ null ┆ 4.0 ┆ b │
| │ 2 ┆ 5.0 ┆ c │
| └──────┴─────┴─────┘
|
| sql(self, query: 'str', *, table_name: 'str' = 'self') -> 'DataFrame'
| Execute a SQL query against the DataFrame.
|
| .. versionadded:: 0.20.24
|
| .. warning::
| This functionality is considered **unstable**, although it is close to
| being considered stable. It may be changed at any point without it being
| considered a breaking change.
|
| Parameters
| ----------
| query
| SQL query to execute.
| table_name
| Optionally provide an explicit name for the table that represents the
| calling frame (defaults to "self").
|
| Notes
| -----
| * The calling DataFrame is automatically registered as a table in the SQLContext
| under the name "self". If you want access to the DataFrames and LazyFrames
| found in the current globals, use the top-level :meth:`pl.sql <polars.sql>`.
| * More control over registration and execution behaviour is available by
| using the :class:`SQLContext` object.
| * The SQL query executes in lazy mode before being collected and returned
| as a DataFrame.
|
| See Also
| --------
| SQLContext
|
| Examples
| --------
| >>> from datetime import date
| >>> df1 = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["zz", "yy", "xx"],
| ... "c": [date(1999, 12, 31), date(2010, 10, 10), date(2077, 8, 8)],
| ... }
| ... )
|
| Query the DataFrame using SQL:
|
| >>> df1.sql("SELECT c, b FROM self WHERE a > 1")
| shape: (2, 2)
| ┌────────────┬─────┐
| │ c ┆ b │
| │ --- ┆ --- │
| │ date ┆ str │
| ╞════════════╪═════╡
| │ 2010-10-10 ┆ yy │
| │ 2077-08-08 ┆ xx │
| └────────────┴─────┘
|
| Apply transformations to a DataFrame using SQL, aliasing "self" to "frame".
|
| >>> df1.sql(
| ... query='''
| ... SELECT
| ... a,
| ... (a % 2 == 0) AS a_is_even,
| ... CONCAT_WS(':', b, b) AS b_b,
| ... EXTRACT(year FROM c) AS year,
| ... 0::float4 AS "zero",
| ... FROM frame
| ... ''',
| ... table_name="frame",
| ... )
| shape: (3, 5)
| ┌─────┬───────────┬───────┬──────┬──────┐
| │ a ┆ a_is_even ┆ b_b ┆ year ┆ zero │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ bool ┆ str ┆ i32 ┆ f32 │
| ╞═════╪═══════════╪═══════╪══════╪══════╡
| │ 1 ┆ false ┆ zz:zz ┆ 1999 ┆ 0.0 │
| │ 2 ┆ true ┆ yy:yy ┆ 2010 ┆ 0.0 │
| │ 3 ┆ false ┆ xx:xx ┆ 2077 ┆ 0.0 │
| └─────┴───────────┴───────┴──────┴──────┘
|
| std(self, ddof: 'int' = 1) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their standard deviation value.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.std()
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 1.0 ┆ 1.0 ┆ null │
| └─────┴─────┴──────┘
| >>> df.std(ddof=0)
| shape: (1, 3)
| ┌──────────┬──────────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════════╪══════════╪══════╡
| │ 0.816497 ┆ 0.816497 ┆ null │
| └──────────┴──────────┴──────┘
|
| sum(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their sum value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.sum()
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 6 ┆ 21 ┆ null │
| └─────┴─────┴──────┘
|
| sum_horizontal(self, *, ignore_nulls: 'bool' = True) -> 'Series'
| Sum all values horizontally across columns.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default).
| If set to `False`, any null value in the input will lead to a null output.
|
| Returns
| -------
| Series
| A Series named `"sum"`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4.0, 5.0, 6.0],
| ... }
| ... )
| >>> df.sum_horizontal()
| shape: (3,)
| Series: 'sum' [f64]
| [
| 5.0
| 7.0
| 9.0
| ]
|
| tail(self, n: 'int' = 5) -> 'DataFrame'
| Get the last `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return. If a negative value is passed, return all rows
| except the first `abs(n)`.
|
| See Also
| --------
| head, slice
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> df.tail(3)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ 9 ┆ d │
| │ 5 ┆ 10 ┆ e │
| └─────┴─────┴─────┘
|
| Pass a negative value to get all rows `except` the first `abs(n)`.
|
| >>> df.tail(-3)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 4 ┆ 9 ┆ d │
| │ 5 ┆ 10 ┆ e │
| └─────┴─────┴─────┘
|
| to_arrow(self, *, compat_level: 'CompatLevel | None' = None) -> 'pa.Table'
| Collect the underlying arrow arrays in an Arrow Table.
|
| This operation is mostly zero copy.
|
| Data types that do copy:
| - CategoricalType
|
| .. versionchanged:: 1.1
| The `future` parameter was renamed `compat_level`.
|
| Parameters
| ----------
| compat_level
| Compatibility level to use when exporting Polars data structures.
| The default compatibility level is recommended for most users.
| Use ``pl.CompatLevel.oldest()`` for the most compatible level.
| ``pl.CompatLevel.newest()`` uses the highest supported compatibility
| level, but is considered unstable and may change without it being
| considered a breaking change.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"foo": [1, 2, 3, 4, 5, 6], "bar": ["a", "b", "c", "d", "e", "f"]}
| ... )
| >>> df.to_arrow()
| pyarrow.Table
| foo: int64
| bar: large_string
| ----
| foo: [[1,2,3,4,5,6]]
| bar: [["a","b","c","d","e","f"]]
|
| to_dict(self, *, as_series: 'bool' = True) -> 'dict[str, Series] | dict[str, list[Any]]'
| Convert DataFrame to a dictionary mapping column name to values.
|
| Parameters
| ----------
| as_series
| True -> Values are Series
| False -> Values are List[Any]
|
| See Also
| --------
| rows_by_key
| to_dicts
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1, 2, 3, 4, 5],
| ... "fruits": ["banana", "banana", "apple", "apple", "banana"],
| ... "B": [5, 4, 3, 2, 1],
| ... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"],
| ... "optional": [28, 300, None, 2, -30],
| ... }
| ... )
| >>> df
| shape: (5, 5)
| ┌─────┬────────┬─────┬────────┬──────────┐
| │ A ┆ fruits ┆ B ┆ cars ┆ optional │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ i64 ┆ str ┆ i64 │
| ╞═════╪════════╪═════╪════════╪══════════╡
| │ 1 ┆ banana ┆ 5 ┆ beetle ┆ 28 │
| │ 2 ┆ banana ┆ 4 ┆ audi ┆ 300 │
| │ 3 ┆ apple ┆ 3 ┆ beetle ┆ null │
| │ 4 ┆ apple ┆ 2 ┆ beetle ┆ 2 │
| │ 5 ┆ banana ┆ 1 ┆ beetle ┆ -30 │
| └─────┴────────┴─────┴────────┴──────────┘
| >>> df.to_dict(as_series=False)
| {'A': [1, 2, 3, 4, 5],
| 'fruits': ['banana', 'banana', 'apple', 'apple', 'banana'],
| 'B': [5, 4, 3, 2, 1],
| 'cars': ['beetle', 'audi', 'beetle', 'beetle', 'beetle'],
| 'optional': [28, 300, None, 2, -30]}
| >>> df.to_dict(as_series=True)
| {'A': shape: (5,)
| Series: 'A' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| ], 'fruits': shape: (5,)
| Series: 'fruits' [str]
| [
| "banana"
| "banana"
| "apple"
| "apple"
| "banana"
| ], 'B': shape: (5,)
| Series: 'B' [i64]
| [
| 5
| 4
| 3
| 2
| 1
| ], 'cars': shape: (5,)
| Series: 'cars' [str]
| [
| "beetle"
| "audi"
| "beetle"
| "beetle"
| "beetle"
| ], 'optional': shape: (5,)
| Series: 'optional' [i64]
| [
| 28
| 300
| null
| 2
| -30
| ]}
|
| to_dicts(self) -> 'list[dict[str, Any]]'
| Convert every row to a dictionary of Python-native values.
|
| Notes
| -----
| If you have `ns`-precision temporal values you should be aware that Python
| natively only supports up to `μs`-precision; `ns`-precision values will be
| truncated to microseconds on conversion to Python. If this matters to your
| use-case you should export to a different format (such as Arrow or NumPy).
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df.to_dicts()
| [{'foo': 1, 'bar': 4}, {'foo': 2, 'bar': 5}, {'foo': 3, 'bar': 6}]
|
| to_dummies(self, columns: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, *, separator: 'str' = '_', drop_first: 'bool' = False, drop_nulls: 'bool' = False) -> 'DataFrame'
| Convert categorical variables into dummy/indicator variables.
|
| Parameters
| ----------
| columns
| Column name(s) or selector(s) that should be converted to dummy
| variables. If set to `None` (default), convert all columns.
| separator
| Separator/delimiter used when generating column names.
| drop_first
| Remove the first category from the variables being encoded.
| drop_nulls
| If there are `None` values in the series, a `null` column is not generated
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2],
| ... "bar": [3, 4],
| ... "ham": ["a", "b"],
| ... }
| ... )
| >>> df.to_dummies()
| shape: (2, 6)
| ┌───────┬───────┬───────┬───────┬───────┬───────┐
| │ foo_1 ┆ foo_2 ┆ bar_3 ┆ bar_4 ┆ ham_a ┆ ham_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ u8 ┆ u8 ┆ u8 ┆ u8 │
| ╞═══════╪═══════╪═══════╪═══════╪═══════╪═══════╡
| │ 1 ┆ 0 ┆ 1 ┆ 0 ┆ 1 ┆ 0 │
| │ 0 ┆ 1 ┆ 0 ┆ 1 ┆ 0 ┆ 1 │
| └───────┴───────┴───────┴───────┴───────┴───────┘
|
| >>> df.to_dummies(drop_first=True)
| shape: (2, 3)
| ┌───────┬───────┬───────┐
| │ foo_2 ┆ bar_4 ┆ ham_b │
| │ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ u8 │
| ╞═══════╪═══════╪═══════╡
| │ 0 ┆ 0 ┆ 0 │
| │ 1 ┆ 1 ┆ 1 │
| └───────┴───────┴───────┘
|
| >>> import polars.selectors as cs
| >>> df.to_dummies(cs.integer(), separator=":")
| shape: (2, 5)
| ┌───────┬───────┬───────┬───────┬─────┐
| │ foo:1 ┆ foo:2 ┆ bar:3 ┆ bar:4 ┆ ham │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ u8 ┆ u8 ┆ str │
| ╞═══════╪═══════╪═══════╪═══════╪═════╡
| │ 1 ┆ 0 ┆ 1 ┆ 0 ┆ a │
| │ 0 ┆ 1 ┆ 0 ┆ 1 ┆ b │
| └───────┴───────┴───────┴───────┴─────┘
|
| >>> df.to_dummies(cs.integer(), drop_first=True, separator=":")
| shape: (2, 3)
| ┌───────┬───────┬─────┐
| │ foo:2 ┆ bar:4 ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ str │
| ╞═══════╪═══════╪═════╡
| │ 0 ┆ 0 ┆ a │
| │ 1 ┆ 1 ┆ b │
| └───────┴───────┴─────┘
|
| to_init_repr(self, n: 'int' = 1000) -> 'str'
| Convert DataFrame to instantiable string representation.
|
| Parameters
| ----------
| n
| Only use first n rows.
|
| See Also
| --------
| polars.Series.to_init_repr
| polars.from_repr
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... [
| ... pl.Series("foo", [1, 2, 3], dtype=pl.UInt8),
| ... pl.Series("bar", [6.0, 7.0, 8.0], dtype=pl.Float32),
| ... pl.Series("ham", ["a", "b", "c"], dtype=pl.String),
| ... ]
| ... )
| >>> print(df.to_init_repr())
| pl.DataFrame(
| [
| pl.Series('foo', [1, 2, 3], dtype=pl.UInt8),
| pl.Series('bar', [6.0, 7.0, 8.0], dtype=pl.Float32),
| pl.Series('ham', ['a', 'b', 'c'], dtype=pl.String),
| ]
| )
|
| >>> df_from_str_repr = eval(df.to_init_repr())
| >>> df_from_str_repr
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u8 ┆ f32 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 7.0 ┆ b │
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| to_jax(self, return_type: 'JaxExportType' = 'array', *, device: 'jax.Device | str | None' = None, label: 'str | Expr | Sequence[str | Expr] | None' = None, features: 'str | Expr | Sequence[str | Expr] | None' = None, dtype: 'PolarsDataType | None' = None, order: 'IndexOrder' = 'fortran') -> 'jax.Array | dict[str, jax.Array]'
| Convert DataFrame to a Jax Array, or dict of Jax Arrays.
|
| .. versionadded:: 0.20.27
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| return_type : {"array", "dict"}
| Set return type; a Jax Array, or dict of Jax Arrays.
| device
| Specify the jax `Device` on which the array will be created; can provide
| a string (such as "cpu", "gpu", or "tpu") in which case the device is
| retrieved as `jax.devices(string)[0]`. For more specific control you
| can supply the instantiated `Device` directly. If None, arrays are
| created on the default device.
| label
| One or more column names, expressions, or selectors that label the feature
| data; results in a `{"label": ..., "features": ...}` dict being returned
| when `return_type` is "dict" instead of a `{"col": array, }` dict.
| features
| One or more column names, expressions, or selectors that contain the feature
| data; if omitted, all columns that are not designated as part of the label
| are used. Only applies when `return_type` is "dict".
| dtype
| Unify the dtype of all returned arrays; this casts any column that is
| not already of the required dtype before converting to Array. Note that
| export will be single-precision (32bit) unless the Jax config/environment
| directs otherwise (eg: "jax_enable_x64" was set True in the config object
| at startup, or "JAX_ENABLE_X64" is set to "1" in the environment).
| order : {"c", "fortran"}
| The index order of the returned Jax array, either C-like (row-major) or
| Fortran-like (column-major).
|
| See Also
| --------
| to_dummies
| to_numpy
| to_torch
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "lbl": [0, 1, 2, 3],
| ... "feat1": [1, 0, 0, 1],
| ... "feat2": [1.5, -0.5, 0.0, -2.25],
| ... }
| ... )
|
| Standard return type (2D Array), on the standard device:
|
| >>> df.to_jax()
| Array([[ 0. , 1. , 1.5 ],
| [ 1. , 0. , -0.5 ],
| [ 2. , 0. , 0. ],
| [ 3. , 1. , -2.25]], dtype=float32)
|
| Create the Array on the default GPU device:
|
| >>> a = df.to_jax(device="gpu") # doctest: +SKIP
| >>> a.device() # doctest: +SKIP
| GpuDevice(id=0, process_index=0)
|
| Create the Array on a specific GPU device:
|
| >>> gpu_device = jax.devices("gpu")[1] # doctest: +SKIP
| >>> a = df.to_jax(device=gpu_device) # doctest: +SKIP
| >>> a.device() # doctest: +SKIP
| GpuDevice(id=1, process_index=0)
|
| As a dictionary of individual Arrays:
|
| >>> df.to_jax("dict")
| {'lbl': Array([0, 1, 2, 3], dtype=int32),
| 'feat1': Array([1, 0, 0, 1], dtype=int32),
| 'feat2': Array([ 1.5 , -0.5 , 0. , -2.25], dtype=float32)}
|
| As a "label" and "features" dictionary; note that as "features" is not
| declared, it defaults to all the columns that are not in "label":
|
| >>> df.to_jax("dict", label="lbl")
| {'label': Array([[0],
| [1],
| [2],
| [3]], dtype=int32),
| 'features': Array([[ 1. , 1.5 ],
| [ 0. , -0.5 ],
| [ 0. , 0. ],
| [ 1. , -2.25]], dtype=float32)}
|
| As a "label" and "features" dictionary where each is designated using
| a col or selector expression (which can also be used to cast the data
| if the label and features are better-represented with different dtypes):
|
| >>> import polars.selectors as cs
| >>> df.to_jax(
| ... return_type="dict",
| ... features=cs.float(),
| ... label=pl.col("lbl").cast(pl.UInt8),
| ... )
| {'label': Array([[0],
| [1],
| [2],
| [3]], dtype=uint8),
| 'features': Array([[ 1.5 ],
| [-0.5 ],
| [ 0. ],
| [-2.25]], dtype=float32)}
|
| to_numpy(self, *, order: 'IndexOrder' = 'fortran', writable: 'bool' = False, allow_copy: 'bool' = True, structured: 'bool' = False, use_pyarrow: 'bool | None' = None) -> 'np.ndarray[Any, Any]'
| Convert this DataFrame to a NumPy ndarray.
|
| This operation copies data only when necessary. The conversion is zero copy when
| all of the following hold:
|
| - The DataFrame is fully contiguous in memory, with all Series back-to-back and
| all Series consisting of a single chunk.
| - The data type is an integer or float.
| - The DataFrame contains no null values.
| - The `order` parameter is set to `fortran` (default).
| - The `writable` parameter is set to `False` (default).
|
| Parameters
| ----------
| order
| The index order of the returned NumPy array, either C-like or
| Fortran-like. In general, using the Fortran-like index order is faster.
| However, the C-like order might be more appropriate to use for downstream
| applications to prevent cloning data, e.g. when reshaping into a
| one-dimensional array.
| writable
| Ensure the resulting array is writable. This will force a copy of the data
| if the array was created without copy, as the underlying Arrow data is
| immutable.
| allow_copy
| Allow memory to be copied to perform the conversion. If set to `False`,
| causes conversions that are not zero-copy to fail.
| structured
| Return a `structured array`_ with a data type that corresponds to the
| DataFrame schema. If set to `False` (default), a 2D ndarray is
| returned instead.
|
| .. _structured array: https://numpy.org/doc/stable/user/basics.rec.html
|
| use_pyarrow
| Use `pyarrow.Array.to_numpy
| <https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy>`_
|
| function for the conversion to NumPy if necessary.
|
| .. deprecated:: 0.20.28
| Polars now uses its native engine by default for conversion to NumPy.
|
| Examples
| --------
| Numeric data without nulls can be converted without copying data in some cases.
| The resulting array will not be writable.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> arr = df.to_numpy()
| >>> arr
| array([[1],
| [2],
| [3]])
| >>> arr.flags.writeable
| False
|
| Set `writable=True` to force data copy to make the array writable.
|
| >>> df.to_numpy(writable=True).flags.writeable
| True
|
| If the DataFrame contains different numeric data types, the resulting data type
| will be the supertype. This requires data to be copied. Integer types with
| nulls are cast to a float type with `nan` representing a null value.
|
| >>> df = pl.DataFrame({"a": [1, 2, None], "b": [4.0, 5.0, 6.0]})
| >>> df.to_numpy()
| array([[ 1., 4.],
| [ 2., 5.],
| [nan, 6.]])
|
| Set `allow_copy=False` to raise an error if data would be copied.
|
| >>> s.to_numpy(allow_copy=False) # doctest: +SKIP
| Traceback (most recent call last):
| ...
| RuntimeError: copy not allowed: cannot convert to a NumPy array without copying data
|
| Polars defaults to F-contiguous order. Use `order="c"` to force the resulting
| array to be C-contiguous.
|
| >>> df.to_numpy(order="c").flags.c_contiguous
| True
|
| DataFrames with mixed types will result in an array with an object dtype.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.5, 7.0, 8.5],
| ... "ham": ["a", "b", "c"],
| ... },
| ... schema_overrides={"foo": pl.UInt8, "bar": pl.Float32},
| ... )
| >>> df.to_numpy()
| array([[1, 6.5, 'a'],
| [2, 7.0, 'b'],
| [3, 8.5, 'c']], dtype=object)
|
| Set `structured=True` to convert to a structured array, which can better
| preserve individual column data such as name and data type.
|
| >>> df.to_numpy(structured=True)
| array([(1, 6.5, 'a'), (2, 7. , 'b'), (3, 8.5, 'c')],
| dtype=[('foo', 'u1'), ('bar', '<f4'), ('ham', '<U1')])
|
| to_pandas(self, *, use_pyarrow_extension_array: 'bool' = False, **kwargs: 'Any') -> 'pd.DataFrame'
| Convert this DataFrame to a pandas DataFrame.
|
| This operation copies data if `use_pyarrow_extension_array` is not enabled.
|
| Parameters
| ----------
| use_pyarrow_extension_array
| Use PyArrow-backed extension arrays instead of NumPy arrays for the columns
| of the pandas DataFrame. This allows zero copy operations and preservation
| of null values. Subsequent operations on the resulting pandas DataFrame may
| trigger conversion to NumPy if those operations are not supported by PyArrow
| compute functions.
| **kwargs
| Additional keyword arguments to be passed to
| :meth:`pyarrow.Table.to_pandas`.
|
| Returns
| -------
| :class:`pandas.DataFrame`
|
| Notes
| -----
| This operation requires that both :mod:`pandas` and :mod:`pyarrow` are
| installed.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.to_pandas()
| foo bar ham
| 0 1 6.0 a
| 1 2 7.0 b
| 2 3 8.0 c
|
| Null values in numeric columns are converted to `NaN`.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, None],
| ... "bar": [6.0, None, 8.0],
| ... "ham": [None, "b", "c"],
| ... }
| ... )
| >>> df.to_pandas()
| foo bar ham
| 0 1.0 6.0 NaN
| 1 2.0 NaN b
| 2 NaN 8.0 c
|
| Pass `use_pyarrow_extension_array=True` to get a pandas DataFrame with columns
| backed by PyArrow extension arrays. This will preserve null values.
|
| >>> df.to_pandas(use_pyarrow_extension_array=True)
| foo bar ham
| 0 1 6.0 <NA>
| 1 2 <NA> b
| 2 <NA> 8.0 c
| >>> _.dtypes
| foo int64[pyarrow]
| bar double[pyarrow]
| ham large_string[pyarrow]
| dtype: object
|
| to_series(self, index: 'int' = 0) -> 'Series'
| Select column as Series at index location.
|
| Parameters
| ----------
| index
| Location of selection.
|
| See Also
| --------
| get_column
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.to_series(1)
| shape: (3,)
| Series: 'bar' [i64]
| [
| 6
| 7
| 8
| ]
|
| to_struct(self, name: 'str' = '') -> 'Series'
| Convert a `DataFrame` to a `Series` of type `Struct`.
|
| Parameters
| ----------
| name
| Name for the struct Series
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5],
| ... "b": ["one", "two", "three", "four", "five"],
| ... }
| ... )
| >>> df.to_struct("nums")
| shape: (5,)
| Series: 'nums' [struct[2]]
| [
| {1,"one"}
| {2,"two"}
| {3,"three"}
| {4,"four"}
| {5,"five"}
| ]
|
| to_torch(self, return_type: 'TorchExportType' = 'tensor', *, label: 'str | Expr | Sequence[str | Expr] | None' = None, features: 'str | Expr | Sequence[str | Expr] | None' = None, dtype: 'PolarsDataType | None' = None) -> 'torch.Tensor | dict[str, torch.Tensor] | PolarsDataset'
| Convert DataFrame to a PyTorch Tensor, Dataset, or dict of Tensors.
|
| .. versionadded:: 0.20.23
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| return_type : {"tensor", "dataset", "dict"}
| Set return type; a PyTorch Tensor, PolarsDataset (a frame-specialized
| TensorDataset), or dict of Tensors.
| label
| One or more column names, expressions, or selectors that label the feature
| data; when `return_type` is "dataset", the PolarsDataset will return
| `(features, label)` tensor tuples for each row. Otherwise, it returns
| `(features,)` tensor tuples where the feature contains all the row data.
| features
| One or more column names, expressions, or selectors that contain the feature
| data; if omitted, all columns that are not designated as part of the label
| are used.
| dtype
| Unify the dtype of all returned tensors; this casts any column that is
| not of the required dtype before converting to Tensor. This includes
| the label column *unless* the label is an expression (such as
| `pl.col("label_column").cast(pl.Int16)`).
|
| See Also
| --------
| to_dummies
| to_jax
| to_numpy
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "lbl": [0, 1, 2, 3],
| ... "feat1": [1, 0, 0, 1],
| ... "feat2": [1.5, -0.5, 0.0, -2.25],
| ... }
| ... )
|
| Standard return type (Tensor), with f32 supertype:
|
| >>> df.to_torch(dtype=pl.Float32)
| tensor([[ 0.0000, 1.0000, 1.5000],
| [ 1.0000, 0.0000, -0.5000],
| [ 2.0000, 0.0000, 0.0000],
| [ 3.0000, 1.0000, -2.2500]])
|
| As a dictionary of individual Tensors:
|
| >>> df.to_torch("dict")
| {'lbl': tensor([0, 1, 2, 3]),
| 'feat1': tensor([1, 0, 0, 1]),
| 'feat2': tensor([ 1.5000, -0.5000, 0.0000, -2.2500], dtype=torch.float64)}
|
| As a "label" and "features" dictionary; note that as "features" is not
| declared, it defaults to all the columns that are not in "label":
|
| >>> df.to_torch("dict", label="lbl", dtype=pl.Float32)
| {'label': tensor([[0.],
| [1.],
| [2.],
| [3.]]),
| 'features': tensor([[ 1.0000, 1.5000],
| [ 0.0000, -0.5000],
| [ 0.0000, 0.0000],
| [ 1.0000, -2.2500]])}
|
| As a PolarsDataset, with f64 supertype:
|
| >>> ds = df.to_torch("dataset", dtype=pl.Float64)
| >>> ds[3]
| (tensor([ 3.0000, 1.0000, -2.2500], dtype=torch.float64),)
| >>> ds[:2]
| (tensor([[ 0.0000, 1.0000, 1.5000],
| [ 1.0000, 0.0000, -0.5000]], dtype=torch.float64),)
| >>> ds[[0, 3]]
| (tensor([[ 0.0000, 1.0000, 1.5000],
| [ 3.0000, 1.0000, -2.2500]], dtype=torch.float64),)
|
| As a convenience the PolarsDataset can opt in to half-precision data
| for experimentation (usually this would be set on the model/pipeline):
|
| >>> list(ds.half())
| [(tensor([0.0000, 1.0000, 1.5000], dtype=torch.float16),),
| (tensor([ 1.0000, 0.0000, -0.5000], dtype=torch.float16),),
| (tensor([2., 0., 0.], dtype=torch.float16),),
| (tensor([ 3.0000, 1.0000, -2.2500], dtype=torch.float16),)]
|
| Pass PolarsDataset to a DataLoader, designating the label:
|
| >>> from torch.utils.data import DataLoader
| >>> ds = df.to_torch("dataset", label="lbl")
| >>> dl = DataLoader(ds, batch_size=2)
| >>> batches = list(dl)
| >>> batches[0]
| [tensor([[ 1.0000, 1.5000],
| [ 0.0000, -0.5000]], dtype=torch.float64), tensor([0, 1])]
|
| Note that labels can be given as expressions, allowing them to have
| a dtype independent of the feature columns (multi-column labels are
| supported).
|
| >>> ds = df.to_torch(
| ... return_type="dataset",
| ... dtype=pl.Float32,
| ... label=pl.col("lbl").cast(pl.Int16),
| ... )
| >>> ds[:2]
| (tensor([[ 1.0000, 1.5000],
| [ 0.0000, -0.5000]]), tensor([0, 1], dtype=torch.int16))
|
| Easily integrate with (for example) scikit-learn and other datasets:
|
| >>> from sklearn.datasets import fetch_california_housing # doctest: +SKIP
| >>> housing = fetch_california_housing() # doctest: +SKIP
| >>> df = pl.DataFrame(
| ... data=housing.data,
| ... schema=housing.feature_names,
| ... ).with_columns(
| ... Target=housing.target,
| ... ) # doctest: +SKIP
| >>> train = df.to_torch("dataset", label="Target") # doctest: +SKIP
| >>> loader = DataLoader(
| ... train,
| ... shuffle=True,
| ... batch_size=64,
| ... ) # doctest: +SKIP
|
| top_k(self, k: 'int', *, by: 'IntoExpr | Iterable[IntoExpr]', reverse: 'bool | Sequence[bool]' = False) -> 'DataFrame'
| Return the `k` largest rows.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| k
| Number of rows to return.
| by
| Column(s) used to determine the top rows.
| Accepts expression input. Strings are parsed as column names.
| reverse
| Consider the `k` smallest elements of the `by` column(s) (instead of the `k`
| largest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| bottom_k
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [2, 1, 1, 3, 2, 1],
| ... }
| ... )
|
| Get the rows which contain the 4 largest values in column b.
|
| >>> df.top_k(4, by="b")
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 3 │
| │ a ┆ 2 │
| │ b ┆ 2 │
| │ b ┆ 1 │
| └─────┴─────┘
|
| Get the rows which contain the 4 largest values when sorting on column b and a.
|
| >>> df.top_k(4, by=["b", "a"])
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 3 │
| │ b ┆ 2 │
| │ a ┆ 2 │
| │ c ┆ 1 │
| └─────┴─────┘
|
| transpose(self, *, include_header: 'bool' = False, header_name: 'str' = 'column', column_names: 'str | Iterable[str] | None' = None) -> 'DataFrame'
| Transpose a DataFrame over the diagonal.
|
| Parameters
| ----------
| include_header
| If set, the column names will be added as first column.
| header_name
| If `include_header` is set, this determines the name of the column that will
| be inserted.
| column_names
| Optional iterable yielding strings or a string naming an existing column.
| These will name the value (non-header) columns in the transposed data.
|
| Notes
| -----
| This is a very expensive operation. Perhaps you can do it differently.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
| >>> df.transpose(include_header=True)
| shape: (2, 4)
| ┌────────┬──────────┬──────────┬──────────┐
| │ column ┆ column_0 ┆ column_1 ┆ column_2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞════════╪══════════╪══════════╪══════════╡
| │ a ┆ 1 ┆ 2 ┆ 3 │
| │ b ┆ 4 ┆ 5 ┆ 6 │
| └────────┴──────────┴──────────┴──────────┘
|
| Replace the auto-generated column names with a list
|
| >>> df.transpose(include_header=False, column_names=["x", "y", "z"])
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ x ┆ y ┆ z │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| Include the header as a separate column
|
| >>> df.transpose(
| ... include_header=True, header_name="foo", column_names=["x", "y", "z"]
| ... )
| shape: (2, 4)
| ┌─────┬─────┬─────┬─────┐
| │ foo ┆ x ┆ y ┆ z │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╡
| │ a ┆ 1 ┆ 2 ┆ 3 │
| │ b ┆ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┴─────┘
|
| Replace the auto-generated column with column names from a generator function
|
| >>> def name_generator():
| ... base_name = "my_column_"
| ... count = 0
| ... while True:
| ... yield f"{base_name}{count}"
| ... count += 1
| >>> df.transpose(include_header=False, column_names=name_generator())
| shape: (2, 3)
| ┌─────────────┬─────────────┬─────────────┐
| │ my_column_0 ┆ my_column_1 ┆ my_column_2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════════════╪═════════════╪═════════════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────────────┴─────────────┴─────────────┘
|
| Use an existing column as the new column names
|
| >>> df = pl.DataFrame(dict(id=["i", "j", "k"], a=[1, 2, 3], b=[4, 5, 6]))
| >>> df.transpose(column_names="id")
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ i ┆ j ┆ k │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
| >>> df.transpose(include_header=True, header_name="new_id", column_names="id")
| shape: (2, 4)
| ┌────────┬─────┬─────┬─────┐
| │ new_id ┆ i ┆ j ┆ k │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╪═════╡
| │ a ┆ 1 ┆ 2 ┆ 3 │
| │ b ┆ 4 ┆ 5 ┆ 6 │
| └────────┴─────┴─────┴─────┘
|
| unique(self, subset: 'IntoExpr | Collection[IntoExpr] | None' = None, *, keep: 'UniqueKeepStrategy' = 'any', maintain_order: 'bool' = False) -> 'DataFrame'
| Drop duplicate rows from this DataFrame.
|
| Parameters
| ----------
| subset
| Column name(s), selector(s), or expressions to consider when identifying
| duplicate rows. If set to `None` (default), all columns are considered.
| keep : {'first', 'last', 'any', 'none'}
| Which of the duplicate rows to keep.
|
| * 'any': Does not give any guarantee of which row is kept.
| This allows more optimizations.
| * 'none': Don't keep duplicate rows.
| * 'first': Keep the first unique row.
| * 'last': Keep the last unique row.
| maintain_order
| Keep the same order as the original DataFrame. This is more expensive
| to compute. Settings this to `True` blocks the possibility to run on
| the streaming engine.
|
| Returns
| -------
| DataFrame
| DataFrame with unique rows.
|
| Notes
| -----
| If you're coming from Pandas, this is similar to
| `pandas.DataFrame.drop_duplicates`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 1, 1],
| ... "bar": ["a", "a", "a", "x", "x"],
| ... "ham": ["b", "b", "b", "y", "y"],
| ... }
| ... )
|
| By default, all columns are considered when determining which rows are unique:
|
| >>> df.unique(maintain_order=True)
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
|
| We can also consider only a subset of columns when determining uniqueness,
| controlling which row we keep when duplicates are found:
|
| >>> df.unique(subset="foo", keep="first", maintain_order=True)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| └─────┴─────┴─────┘
| >>> df.unique(subset="foo", keep="last", maintain_order=True)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
| >>> df.unique(subset="foo", keep="none", maintain_order=True)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| └─────┴─────┴─────┘
|
| Selectors can be used to define the "subset" parameter:
|
| >>> import polars.selectors as cs
| >>> df.unique(subset=cs.string(), maintain_order=True)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
|
| We can also use an arbitrary expression in the "subset" parameter; in this
| example we use the part of the label in front of ":" to determine uniqueness:
|
| >>> df = pl.DataFrame(
| ... {
| ... "label": ["xx:1", "xx:2", "yy:3", "yy:4"],
| ... "value": [100, 200, 300, 400],
| ... }
| ... )
| >>> df.unique(
| ... subset=pl.col("label").str.extract(r"^(\w+):"),
| ... maintain_order=True,
| ... keep="first",
| ... )
| shape: (2, 2)
| ┌───────┬───────┐
| │ label ┆ value │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═══════╡
| │ xx:1 ┆ 100 │
| │ yy:3 ┆ 300 │
| └───────┴───────┘
|
| unnest(self, columns: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None, *more_columns: 'ColumnNameOrSelector', separator: 'str | None' = None) -> 'DataFrame'
| Decompose struct columns into separate columns for each of their fields.
|
| The new columns will be inserted into the dataframe at the location of the
| struct column.
|
| If no columns are provided, all struct columns are unnested.
|
| Parameters
| ----------
| columns
| Name of the struct column(s) that should be unnested.
| *more_columns
| Additional columns to unnest, specified as positional arguments.
| separator
| Rename output column names as combination of the struct column name,
| name separator and field name.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "before": ["foo", "bar"],
| ... "t_a": [1, 2],
| ... "t_b": ["a", "b"],
| ... "t_c": [True, None],
| ... "t_d": [[1, 2], [3]],
| ... "after": ["baz", "womp"],
| ... }
| ... ).select("before", pl.struct(pl.col("^t_.$")).alias("t_struct"), "after")
| >>> df
| shape: (2, 3)
| ┌────────┬─────────────────────┬───────┐
| │ before ┆ t_struct ┆ after │
| │ --- ┆ --- ┆ --- │
| │ str ┆ struct[4] ┆ str │
| ╞════════╪═════════════════════╪═══════╡
| │ foo ┆ {1,"a",true,[1, 2]} ┆ baz │
| │ bar ┆ {2,"b",null,[3]} ┆ womp │
| └────────┴─────────────────────┴───────┘
| >>> df.unnest("t_struct")
| shape: (2, 6)
| ┌────────┬─────┬─────┬──────┬───────────┬───────┐
| │ before ┆ t_a ┆ t_b ┆ t_c ┆ t_d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪═════╪═════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴─────┴─────┴──────┴───────────┴───────┘
|
| Unnest all struct columns by calling without arguments:
|
| >>> df.unnest()
| shape: (2, 6)
| ┌────────┬─────┬─────┬──────┬───────────┬───────┐
| │ before ┆ t_a ┆ t_b ┆ t_c ┆ t_d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪═════╪═════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴─────┴─────┴──────┴───────────┴───────┘
|
| >>> df = pl.DataFrame(
| ... {
| ... "before": ["foo", "bar"],
| ... "t_a": [1, 2],
| ... "t_b": ["a", "b"],
| ... "t_c": [True, None],
| ... "t_d": [[1, 2], [3]],
| ... "after": ["baz", "womp"],
| ... }
| ... ).select(
| ... "before",
| ... pl.struct(pl.col("^t_.$").name.map(lambda t: t[2:])).alias("t"),
| ... "after",
| ... )
| >>> df.unnest("t", separator="::")
| shape: (2, 6)
| ┌────────┬──────┬──────┬──────┬───────────┬───────┐
| │ before ┆ t::a ┆ t::b ┆ t::c ┆ t::d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪══════╪══════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴──────┴──────┴──────┴───────────┴───────┘
|
| unpivot(self, on: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, *, index: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, variable_name: 'str | None' = None, value_name: 'str | None' = None) -> 'DataFrame'
| Unpivot a DataFrame from wide to long format.
|
| Optionally leaves identifiers set.
|
| This function is useful to massage a DataFrame into a format where one or more
| columns are identifier variables (index) while all other columns, considered
| measured variables (on), are "unpivoted" to the row axis leaving just
| two non-identifier columns, 'variable' and 'value'.
|
| Parameters
| ----------
| on
| Column(s) or selector(s) to use as values variables; if `on`
| is empty no columns will be used. If set to `None` (default)
| all columns that are not in `index` will be used.
| index
| Column(s) or selector(s) to use as identifier variables.
| variable_name
| Name to give to the `variable` column. Defaults to "variable"
| value_name
| Name to give to the `value` column. Defaults to "value"
|
| Notes
| -----
| If you're coming from pandas, this is similar to `pandas.DataFrame.melt`,
| but with `index` replacing `id_vars` and `on` replacing `value_vars`.
| In other frameworks, you might know this operation as `pivot_longer`.
|
| The resulting row order is unspecified.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["x", "y", "z"],
| ... "b": [1, 3, 5],
| ... "c": [2, 4, 6],
| ... }
| ... )
| >>> import polars.selectors as cs
| >>> df.unpivot(cs.numeric(), index="a")
| shape: (6, 3)
| ┌─────┬──────────┬───────┐
| │ a ┆ variable ┆ value │
| │ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 │
| ╞═════╪══════════╪═══════╡
| │ x ┆ b ┆ 1 │
| │ y ┆ b ┆ 3 │
| │ z ┆ b ┆ 5 │
| │ x ┆ c ┆ 2 │
| │ y ┆ c ┆ 4 │
| │ z ┆ c ┆ 6 │
| └─────┴──────────┴───────┘
|
| unstack(self, *, step: 'int', how: 'UnstackDirection' = 'vertical', columns: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, fill_values: 'list[Any] | None' = None) -> 'DataFrame'
| Unstack a long table to a wide form without doing an aggregation.
|
| This can be much faster than a pivot, because it can skip the grouping phase.
|
| Parameters
| ----------
| step
| Number of rows in the unstacked frame.
| how : { 'vertical', 'horizontal' }
| Direction of the unstack.
| columns
| Column name(s) or selector(s) to include in the operation.
| If set to `None` (default), use all columns.
| fill_values
| Fill values that don't fit the new size with this value.
|
| Examples
| --------
| >>> from string import ascii_uppercase
| >>> df = pl.DataFrame(
| ... {
| ... "x": list(ascii_uppercase[0:8]),
| ... "y": pl.int_range(1, 9, eager=True),
| ... }
| ... ).with_columns(
| ... z=pl.int_ranges(pl.col("y"), pl.col("y") + 2, dtype=pl.UInt8),
| ... )
| >>> df
| shape: (8, 3)
| ┌─────┬─────┬──────────┐
| │ x ┆ y ┆ z │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ list[u8] │
| ╞═════╪═════╪══════════╡
| │ A ┆ 1 ┆ [1, 2] │
| │ B ┆ 2 ┆ [2, 3] │
| │ C ┆ 3 ┆ [3, 4] │
| │ D ┆ 4 ┆ [4, 5] │
| │ E ┆ 5 ┆ [5, 6] │
| │ F ┆ 6 ┆ [6, 7] │
| │ G ┆ 7 ┆ [7, 8] │
| │ H ┆ 8 ┆ [8, 9] │
| └─────┴─────┴──────────┘
| >>> df.unstack(step=4, how="vertical")
| shape: (4, 6)
| ┌─────┬─────┬─────┬─────┬──────────┬──────────┐
| │ x_0 ┆ x_1 ┆ y_0 ┆ y_1 ┆ z_0 ┆ z_1 │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 ┆ i64 ┆ list[u8] ┆ list[u8] │
| ╞═════╪═════╪═════╪═════╪══════════╪══════════╡
| │ A ┆ E ┆ 1 ┆ 5 ┆ [1, 2] ┆ [5, 6] │
| │ B ┆ F ┆ 2 ┆ 6 ┆ [2, 3] ┆ [6, 7] │
| │ C ┆ G ┆ 3 ┆ 7 ┆ [3, 4] ┆ [7, 8] │
| │ D ┆ H ┆ 4 ┆ 8 ┆ [4, 5] ┆ [8, 9] │
| └─────┴─────┴─────┴─────┴──────────┴──────────┘
| >>> df.unstack(step=2, how="horizontal")
| shape: (4, 6)
| ┌─────┬─────┬─────┬─────┬──────────┬──────────┐
| │ x_0 ┆ x_1 ┆ y_0 ┆ y_1 ┆ z_0 ┆ z_1 │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 ┆ i64 ┆ list[u8] ┆ list[u8] │
| ╞═════╪═════╪═════╪═════╪══════════╪══════════╡
| │ A ┆ B ┆ 1 ┆ 2 ┆ [1, 2] ┆ [2, 3] │
| │ C ┆ D ┆ 3 ┆ 4 ┆ [3, 4] ┆ [4, 5] │
| │ E ┆ F ┆ 5 ┆ 6 ┆ [5, 6] ┆ [6, 7] │
| │ G ┆ H ┆ 7 ┆ 8 ┆ [7, 8] ┆ [8, 9] │
| └─────┴─────┴─────┴─────┴──────────┴──────────┘
| >>> import polars.selectors as cs
| >>> df.unstack(step=5, columns=cs.numeric(), fill_values=0)
| shape: (5, 2)
| ┌─────┬─────┐
| │ y_0 ┆ y_1 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 6 │
| │ 2 ┆ 7 │
| │ 3 ┆ 8 │
| │ 4 ┆ 0 │
| │ 5 ┆ 0 │
| └─────┴─────┘
|
| update(self, other: 'DataFrame', on: 'str | Sequence[str] | None' = None, how: "Literal['left', 'inner', 'full']" = 'left', *, left_on: 'str | Sequence[str] | None' = None, right_on: 'str | Sequence[str] | None' = None, include_nulls: 'bool' = False, maintain_order: 'MaintainOrderJoin | None' = 'left') -> 'DataFrame'
| Update the values in this `DataFrame` with the values in `other`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| other
| DataFrame that will be used to update the values
| on
| Column names that will be joined on. If set to `None` (default),
| the implicit row index of each frame is used as a join key.
| how : {'left', 'inner', 'full'}
| * 'left' will keep all rows from the left table; rows may be duplicated
| if multiple rows in the right frame match the left row's key.
| * 'inner' keeps only those rows where the key exists in both frames.
| * 'full' will update existing rows where the key matches while also
| adding any new rows contained in the given frame.
| left_on
| Join column(s) of the left DataFrame.
| right_on
| Join column(s) of the right DataFrame.
| include_nulls
| Overwrite values in the left frame with null values from the right frame.
| If set to `False` (default), null values in the right frame are ignored.
| maintain_order : {'none', 'left', 'right', 'left_right', 'right_left'}
| Which order of rows from the inputs to preserve. See :func:`~DataFrame.join`
| for details. Unlike `join` this function preserves the left order by
| default.
|
| Notes
| -----
| This is syntactic sugar for a left/inner join that preserves the order
| of the left `DataFrame` by default, with an optional coalesce when
| `include_nulls = False`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1, 2, 3, 4],
| ... "B": [400, 500, 600, 700],
| ... }
| ... )
| >>> df
| shape: (4, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 400 │
| │ 2 ┆ 500 │
| │ 3 ┆ 600 │
| │ 4 ┆ 700 │
| └─────┴─────┘
| >>> new_df = pl.DataFrame(
| ... {
| ... "B": [-66, None, -99],
| ... "C": [5, 3, 1],
| ... }
| ... )
|
| Update `df` values with the non-null values in `new_df`, by row index:
|
| >>> df.update(new_df)
| shape: (4, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -66 │
| │ 2 ┆ 500 │
| │ 3 ┆ -99 │
| │ 4 ┆ 700 │
| └─────┴─────┘
|
| Update `df` values with the non-null values in `new_df`, by row index,
| but only keeping those rows that are common to both frames:
|
| >>> df.update(new_df, how="inner")
| shape: (3, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -66 │
| │ 2 ┆ 500 │
| │ 3 ┆ -99 │
| └─────┴─────┘
|
| Update `df` values with the non-null values in `new_df`, using a full
| outer join strategy that defines explicit join columns in each frame:
|
| >>> df.update(new_df, left_on=["A"], right_on=["C"], how="full")
| shape: (5, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -99 │
| │ 2 ┆ 500 │
| │ 3 ┆ 600 │
| │ 4 ┆ 700 │
| │ 5 ┆ -66 │
| └─────┴─────┘
|
| Update `df` values including null values in `new_df`, using a full outer
| join strategy that defines explicit join columns in each frame:
|
| >>> df.update(new_df, left_on="A", right_on="C", how="full", include_nulls=True)
| shape: (5, 2)
| ┌─────┬──────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════╡
| │ 1 ┆ -99 │
| │ 2 ┆ 500 │
| │ 3 ┆ null │
| │ 4 ┆ 700 │
| │ 5 ┆ -66 │
| └─────┴──────┘
|
| upsample(self, time_column: 'str', *, every: 'str | timedelta', group_by: 'str | Sequence[str] | None' = None, maintain_order: 'bool' = False) -> 'DataFrame'
| Upsample a DataFrame at a regular frequency.
|
| The `every` argument is created with the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them:
|
| - "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| time_column
| Time column will be used to determine a date_range.
| Note that this column has to be sorted for the output to make sense.
| every
| Interval will start 'every' duration.
| group_by
| First group by these columns and then upsample for every group.
| maintain_order
| Keep the ordering predictable. This is slower.
|
| Returns
| -------
| DataFrame
| Result will be sorted by `time_column` (but note that if `group_by` columns
| are passed, it will only be sorted within each group).
|
| Examples
| --------
| Upsample a DataFrame by a certain interval.
|
| >>> from datetime import datetime
| >>> df = pl.DataFrame(
| ... {
| ... "time": [
| ... datetime(2021, 2, 1),
| ... datetime(2021, 4, 1),
| ... datetime(2021, 5, 1),
| ... datetime(2021, 6, 1),
| ... ],
| ... "groups": ["A", "B", "A", "B"],
| ... "values": [0, 1, 2, 3],
| ... }
| ... ).set_sorted("time")
| >>> df.upsample(
| ... time_column="time", every="1mo", group_by="groups", maintain_order=True
| ... ).select(pl.all().fill_null(strategy="forward"))
| shape: (7, 3)
| ┌─────────────────────┬────────┬────────┐
| │ time ┆ groups ┆ values │
| │ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ str ┆ i64 │
| ╞═════════════════════╪════════╪════════╡
| │ 2021-02-01 00:00:00 ┆ A ┆ 0 │
| │ 2021-03-01 00:00:00 ┆ A ┆ 0 │
| │ 2021-04-01 00:00:00 ┆ A ┆ 0 │
| │ 2021-05-01 00:00:00 ┆ A ┆ 2 │
| │ 2021-04-01 00:00:00 ┆ B ┆ 1 │
| │ 2021-05-01 00:00:00 ┆ B ┆ 1 │
| │ 2021-06-01 00:00:00 ┆ B ┆ 3 │
| └─────────────────────┴────────┴────────┘
|
| var(self, ddof: 'int' = 1) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their variance value.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.var()
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 1.0 ┆ 1.0 ┆ null │
| └─────┴─────┴──────┘
| >>> df.var(ddof=0)
| shape: (1, 3)
| ┌──────────┬──────────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════════╪══════════╪══════╡
| │ 0.666667 ┆ 0.666667 ┆ null │
| └──────────┴──────────┴──────┘
|
| vstack(self, other: 'DataFrame', *, in_place: 'bool' = False) -> 'DataFrame'
| Grow this DataFrame vertically by stacking a DataFrame to it.
|
| Parameters
| ----------
| other
| DataFrame to stack.
| in_place
| Modify in place.
|
| See Also
| --------
| extend
|
| Examples
| --------
| >>> df1 = pl.DataFrame(
| ... {
| ... "foo": [1, 2],
| ... "bar": [6, 7],
| ... "ham": ["a", "b"],
| ... }
| ... )
| >>> df2 = pl.DataFrame(
| ... {
| ... "foo": [3, 4],
| ... "bar": [8, 9],
| ... "ham": ["c", "d"],
| ... }
| ... )
| >>> df1.vstack(df2)
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ 9 ┆ d │
| └─────┴─────┴─────┘
|
| with_columns(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'DataFrame'
| Add columns to this DataFrame.
|
| Added columns will replace existing columns with the same name.
|
| Parameters
| ----------
| *exprs
| Column(s) to add, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names, other
| non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to add, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| DataFrame
| A new DataFrame with the columns added.
|
| Notes
| -----
| Creating a new DataFrame using this method does not create a new copy of
| existing data.
|
| Examples
| --------
| Pass an expression to add it as a new column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> df.with_columns((pl.col("a") ** 2).alias("a^2"))
| shape: (4, 4)
| ┌─────┬──────┬───────┬─────┐
| │ a ┆ b ┆ c ┆ a^2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 │
| ╞═════╪══════╪═══════╪═════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 │
| │ 2 ┆ 4.0 ┆ true ┆ 4 │
| │ 3 ┆ 10.0 ┆ false ┆ 9 │
| │ 4 ┆ 13.0 ┆ true ┆ 16 │
| └─────┴──────┴───────┴─────┘
|
| Added columns will replace existing columns with the same name.
|
| >>> df.with_columns(pl.col("a").cast(pl.Float64))
| shape: (4, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╡
| │ 1.0 ┆ 0.5 ┆ true │
| │ 2.0 ┆ 4.0 ┆ true │
| │ 3.0 ┆ 10.0 ┆ false │
| │ 4.0 ┆ 13.0 ┆ true │
| └─────┴──────┴───────┘
|
| Multiple columns can be added using positional arguments.
|
| >>> df.with_columns(
| ... (pl.col("a") ** 2).alias("a^2"),
| ... (pl.col("b") / 2).alias("b/2"),
| ... (pl.col("c").not_()).alias("not c"),
| ... )
| shape: (4, 6)
| ┌─────┬──────┬───────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ a^2 ┆ b/2 ┆ not c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 ┆ 0.25 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 4 ┆ 2.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 9 ┆ 5.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 16 ┆ 6.5 ┆ false │
| └─────┴──────┴───────┴─────┴──────┴───────┘
|
| Multiple columns can also be added by passing a list of expressions.
|
| >>> df.with_columns(
| ... [
| ... (pl.col("a") ** 2).alias("a^2"),
| ... (pl.col("b") / 2).alias("b/2"),
| ... (pl.col("c").not_()).alias("not c"),
| ... ]
| ... )
| shape: (4, 6)
| ┌─────┬──────┬───────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ a^2 ┆ b/2 ┆ not c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 ┆ 0.25 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 4 ┆ 2.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 9 ┆ 5.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 16 ┆ 6.5 ┆ false │
| └─────┴──────┴───────┴─────┴──────┴───────┘
|
| Use keyword arguments to easily name your expression inputs.
|
| >>> df.with_columns(
| ... ab=pl.col("a") * pl.col("b"),
| ... not_c=pl.col("c").not_(),
| ... )
| shape: (4, 5)
| ┌─────┬──────┬───────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ ab ┆ not_c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 0.5 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 8.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 30.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 52.0 ┆ false │
| └─────┴──────┴───────┴──────┴───────┘
|
| with_columns_seq(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'DataFrame'
| Add columns to this DataFrame.
|
| Added columns will replace existing columns with the same name.
|
| This will run all expression sequentially instead of in parallel.
| Use this when the work per expression is cheap.
|
| Parameters
| ----------
| *exprs
| Column(s) to add, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names, other
| non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to add, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| DataFrame
| A new DataFrame with the columns added.
|
| See Also
| --------
| with_columns
|
| with_row_count(self, name: 'str' = 'row_nr', offset: 'int' = 0) -> 'DataFrame'
| Add a column at index 0 that counts the rows.
|
| .. deprecated:: 0.20.4
| Use the :meth:`with_row_index` method instead.
| Note that the default column name has changed from 'row_nr' to 'index'.
|
| Parameters
| ----------
| name
| Name of the column to add.
| offset
| Start the row count at this offset. Default = 0
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> df.with_row_count() # doctest: +SKIP
| shape: (3, 3)
| ┌────────┬─────┬─────┐
| │ row_nr ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └────────┴─────┴─────┘
|
| with_row_index(self, name: 'str' = 'index', offset: 'int' = 0) -> 'DataFrame'
| Add a row index as the first column in the DataFrame.
|
| Parameters
| ----------
| name
| Name of the index column.
| offset
| Start the index at this offset. Cannot be negative.
|
| Notes
| -----
| The resulting column does not have any special properties. It is a regular
| column of type `UInt32` (or `UInt64` in `polars[rt64]`).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> df.with_row_index()
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ index ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞═══════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └───────┴─────┴─────┘
| >>> df.with_row_index("id", offset=1000)
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ id ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞══════╪═════╪═════╡
| │ 1000 ┆ 1 ┆ 2 │
| │ 1001 ┆ 3 ┆ 4 │
| │ 1002 ┆ 5 ┆ 6 │
| └──────┴─────┴─────┘
|
| An index column can also be created using the expressions :func:`int_range`
| and :func:`len`.
|
| >>> df.select(
| ... pl.int_range(pl.len(), dtype=pl.UInt32).alias("index"),
| ... pl.all(),
| ... )
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ index ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞═══════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └───────┴─────┴─────┘
|
| write_avro(self, file: 'str | Path | IO[bytes]', compression: 'AvroCompression' = 'uncompressed', name: 'str' = '') -> 'None'
| Write to Apache Avro file.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the data will be written.
| compression : {'uncompressed', 'snappy', 'deflate'}
| Compression method. Defaults to "uncompressed".
| name
| Schema name. Defaults to empty string.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.avro"
| >>> df.write_avro(path)
|
| write_clipboard(self, *, separator: 'str' = '\t', **kwargs: 'Any') -> 'None'
| Copy `DataFrame` in csv format to the system clipboard with `write_csv`.
|
| Useful for pasting into Excel or other similar spreadsheet software.
|
| Parameters
| ----------
| separator
| Separate CSV fields with this symbol.
| kwargs
| Additional arguments to pass to `write_csv`.
|
| See Also
| --------
| polars.read_clipboard: Read a DataFrame from the clipboard.
| write_csv: Write to comma-separated values (CSV) file.
|
| write_csv(self, file: 'str | Path | IO[str] | IO[bytes] | None' = None, *, include_bom: 'bool' = False, compression: "Literal['uncompressed', 'gzip', 'zstd']" = 'uncompressed', compression_level: 'int | None' = None, check_extension: 'bool' = True, include_header: 'bool' = True, separator: 'str' = ',', line_terminator: 'str' = '\n', quote_char: 'str' = '"', batch_size: 'int' = 1024, datetime_format: 'str | None' = None, date_format: 'str | None' = None, time_format: 'str | None' = None, float_scientific: 'bool | None' = None, float_precision: 'int | None' = None, decimal_comma: 'bool' = False, null_value: 'str | None' = None, quote_style: 'CsvQuoteStyle | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None) -> 'str | None'
| Write to comma-separated values (CSV) file.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| If set to `None` (default), the output is returned as a string instead.
| include_bom
| Whether to include UTF-8 BOM in the CSV output.
| compression
| What compression format to use.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| compression_level
| The compression level to use, typically 0-9 or `None` to let the
| engine choose.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| check_extension
| Whether to check if the filename matches the compression settings.
| Will raise an error if compression is set to 'uncompressed' and the
| filename ends in one of (".gz", ".zst", ".zstd") or if
| compression != 'uncompressed' and the filename does not end in the
| appropriate extension. Only applies if file is a path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| include_header
| Whether to include header in the CSV output.
| separator
| Separate CSV fields with this symbol.
| line_terminator
| String used to end each row.
| quote_char
| Byte to use as quoting character.
| batch_size
| Number of rows that will be processed per thread.
| datetime_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate. If no format specified, the default fractional-second
| precision is inferred from the maximum timeunit found in the frame's
| Datetime cols (if any).
| date_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate.
| time_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate.
| float_scientific
| Whether to use scientific form always (true), never (false), or
| automatically (None) for floating-point datatypes.
| float_precision
| Number of decimal places to write, applied to both floating-point
| data types.
| decimal_comma
| Use a comma as the decimal separator instead of a point in standard
| notation. Floats will be encapsulated in quotes if necessary; set the
| field separator to override.
| null_value
| A string representing null values (defaulting to the empty string).
| quote_style : {'necessary', 'always', 'non_numeric', 'never'}
| Determines the quoting strategy used.
|
| - necessary (default): This puts quotes around fields only when necessary.
| They are necessary when fields contain a quote,
| separator or record terminator.
| Quotes are also necessary when writing an empty record
| (which is indistinguishable from a record with one empty field).
| This is the default.
| - always: This puts quotes around every field. Always.
| - never: This never puts quotes around fields, even if that results in
| invalid CSV data (e.g.: by not quoting strings containing the separator).
| - non_numeric: This puts quotes around all fields that are non-numeric.
| Namely, when writing a field that does not parse as a valid float
| or integer, then quotes will be used even if they aren`t strictly
| necessary.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.csv"
| >>> df.write_csv(path, separator=",")
|
| write_database(self, table_name: 'str', connection: 'ConnectionOrCursor | str', *, if_table_exists: 'DbWriteMode' = 'fail', engine: 'DbWriteEngine | None' = None, engine_options: 'dict[str, Any] | None' = None) -> 'int'
| Write the data in a Polars DataFrame to a database.
|
| .. versionadded:: 0.20.26
| Support for instantiated connection objects in addition to URI strings, and
| a new `engine_options` parameter.
|
| Parameters
| ----------
| table_name
| Schema-qualified name of the table to create or append to in the target
| SQL database. If your table name contains special characters, it should
| be quoted.
| connection
| An existing SQLAlchemy or ADBC connection against the target database, or
| a URI string that will be used to instantiate such a connection, such as:
|
| * "postgresql://user:pass@server:port/database"
| * "sqlite:////path/to/database.db"
| if_table_exists : {'append', 'replace', 'fail'}
| The insert mode:
|
| * 'replace' will create a new database table, overwriting an existing one.
| * 'append' will append to an existing table.
| * 'fail' will fail if table already exists.
| engine : {'sqlalchemy', 'adbc'}
| Select the engine to use for writing frame data; only necessary when
| supplying a URI string (defaults to 'sqlalchemy' if unset)
| engine_options
| Additional options to pass to the insert method associated with the engine
| specified by the option `engine`.
|
| * Setting `engine` to "sqlalchemy" currently inserts using Pandas' `to_sql`
| method (though this will eventually be phased out in favor of a native
| solution).
| * Setting `engine` to "adbc" inserts using the ADBC cursor's `adbc_ingest`
| method. Note that when passing an instantiated connection object, PyArrow
| is required for SQLite and Snowflake drivers.
|
| Examples
| --------
| Insert into a temporary table using a PostgreSQL URI and the ADBC engine:
|
| >>> df.write_database(
| ... table_name="target_table",
| ... connection="postgresql://user:pass@server:port/database",
| ... engine="adbc",
| ... engine_options={"temporary": True},
| ... ) # doctest: +SKIP
|
| Insert into a table using a `pyodbc` SQLAlchemy connection to SQL Server
| that was instantiated with "fast_executemany=True" to improve performance:
|
| >>> pyodbc_uri = (
| ... "mssql+pyodbc://user:pass@server:1433/test?"
| ... "driver=ODBC+Driver+18+for+SQL+Server"
| ... )
| >>> engine = create_engine(pyodbc_uri, fast_executemany=True) # doctest: +SKIP
| >>> df.write_database(
| ... table_name="target_table",
| ... connection=engine,
| ... ) # doctest: +SKIP
|
| Returns
| -------
| int
| The number of rows affected, if the driver provides this information.
| Otherwise, returns -1.
|
| write_delta(self, target: 'str | Path | deltalake.DeltaTable', *, mode: "Literal['error', 'append', 'overwrite', 'ignore', 'merge']" = 'error', overwrite_schema: 'bool | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', delta_write_options: 'dict[str, Any] | None' = None, delta_merge_options: 'dict[str, Any] | None' = None) -> 'deltalake.table.TableMerger | None'
| Write DataFrame as delta table.
|
| Parameters
| ----------
| target
| URI of a table or a DeltaTable object.
| mode : {'error', 'append', 'overwrite', 'ignore', 'merge'}
| How to handle existing data.
|
| - If 'error', throw an error if the table already exists (default).
| - If 'append', will add new data.
| - If 'overwrite', will replace table with new data.
| - If 'ignore', will not write anything if table already exists.
| - If 'merge', return a `TableMerger` object to merge data from the DataFrame
| with the existing data.
| overwrite_schema
| If True, allows updating the schema of the table.
|
| .. deprecated:: 0.20.14
| Use the parameter `delta_write_options` instead and pass
| `{"schema_mode": "overwrite"}`.
| storage_options
| Extra options for the storage backends supported by `deltalake`.
| For cloud storages, this may include configurations for authentication etc.
|
| - See a list of supported storage options for S3 `here <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html#variants>`__.
| - See a list of supported storage options for GCS `here <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html#variants>`__.
| - See a list of supported storage options for Azure `here <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants>`__.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| delta_write_options
| Additional keyword arguments while writing a Delta lake Table.
| See a list of supported write options `here <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake>`__.
| delta_merge_options
| Keyword arguments which are required to `MERGE` a Delta lake Table.
| See a list of supported merge options `here <https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.merge>`__.
|
| Raises
| ------
| TypeError
| If the DataFrame contains unsupported data types.
| ArrowInvalidError
| If the DataFrame contains data types that could not be cast to their
| primitive type.
| TableNotFoundError
| If the delta table doesn't exist and MERGE action is triggered
|
| Notes
| -----
| The Polars data types :class:`Null` and :class:`Time` are not supported
| by the delta protocol specification and will raise a TypeError. Columns
| using The :class:`Categorical` data type will be converted to
| normal (non-categorical) strings when written.
|
| Polars columns are always nullable. To write data to a delta table with
| non-nullable columns, a custom pyarrow schema has to be passed to the
| `delta_write_options`. See the last example below.
|
| Examples
| --------
| Write a dataframe to the local filesystem as a Delta Lake table.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> table_path = "/path/to/delta-table/"
| >>> df.write_delta(table_path) # doctest: +SKIP
|
| Append data to an existing Delta Lake table on the local filesystem.
| Note that this will fail if the schema of the new data does not match the
| schema of the existing table.
|
| >>> df.write_delta(table_path, mode="append") # doctest: +SKIP
|
| Overwrite a Delta Lake table as a new version.
| If the schemas of the new and old data are the same, specifying the
| `schema_mode` is not required.
|
| >>> existing_table_path = "/path/to/delta-table/"
| >>> df.write_delta(
| ... existing_table_path,
| ... mode="overwrite",
| ... delta_write_options={"schema_mode": "overwrite"},
| ... ) # doctest: +SKIP
|
| Write a DataFrame as a Delta Lake table to a cloud object store like S3.
|
| >>> table_path = "s3://bucket/prefix/to/delta-table/"
| >>> df.write_delta(
| ... table_path,
| ... storage_options={
| ... "AWS_REGION": "THE_AWS_REGION",
| ... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
| ... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
| ... },
| ... ) # doctest: +SKIP
|
| Write DataFrame as a Delta Lake table with non-nullable columns.
|
| >>> import pyarrow as pa
| >>> existing_table_path = "/path/to/delta-table/"
| >>> df.write_delta(
| ... existing_table_path,
| ... delta_write_options={
| ... "schema": pa.schema([pa.field("foo", pa.int64(), nullable=False)])
| ... },
| ... ) # doctest: +SKIP
|
| Write DataFrame as a Delta Lake table with zstd compression.
| For all `delta_write_options` keyword arguments, check the deltalake docs
| `here
| <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake>`__,
| and for Writer Properties in particular `here
| <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.WriterProperties>`__.
|
| >>> import deltalake
| >>> df.write_delta(
| ... table_path,
| ... delta_write_options={
| ... "writer_properties": deltalake.WriterProperties(compression="zstd"),
| ... },
| ... ) # doctest: +SKIP
|
| Merge the DataFrame with an existing Delta Lake table.
| For all `TableMerger` methods, check the deltalake docs
| `here <https://delta-io.github.io/delta-rs/api/delta_table/delta_table_merger/>`__.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> table_path = "/path/to/delta-table/"
| >>> (
| ... df.write_delta(
| ... "table_path",
| ... mode="merge",
| ... delta_merge_options={
| ... "predicate": "s.foo = t.foo",
| ... "source_alias": "s",
| ... "target_alias": "t",
| ... },
| ... )
| ... .when_matched_update_all()
| ... .when_not_matched_insert_all()
| ... .execute()
| ... ) # doctest: +SKIP
|
| write_excel(self, workbook: 'str | Workbook | IO[bytes] | Path | None' = None, worksheet: 'str | Worksheet | None' = None, *, position: 'tuple[int, int] | str' = 'A1', table_style: 'str | dict[str, Any] | None' = None, table_name: 'str | None' = None, column_formats: 'ColumnFormatDict | None' = None, dtype_formats: 'dict[OneOrMoreDataTypes, str] | None' = None, conditional_formats: 'ConditionalFormatDict | None' = None, header_format: 'dict[str, Any] | None' = None, column_totals: 'ColumnTotalsDefinition | None' = None, column_widths: 'ColumnWidthsDefinition | None' = None, row_totals: 'RowTotalsDefinition | None' = None, row_heights: 'dict[int | tuple[int, ...], int] | int | None' = None, sparklines: 'dict[str, Sequence[str] | dict[str, Any]] | None' = None, formulas: 'dict[str, str | dict[str, str]] | None' = None, float_precision: 'int' = 3, include_header: 'bool' = True, autofilter: 'bool' = True, autofit: 'bool' = False, hidden_columns: 'Sequence[str] | SelectorType | None' = None, hide_gridlines: 'bool' = False, sheet_zoom: 'int | None' = None, freeze_panes: 'str | tuple[int, int] | tuple[str, int, int] | tuple[int, int, int, int] | None' = None, use_zip64: 'bool' = False) -> 'Workbook'
| Write frame data to a table in an Excel workbook/worksheet.
|
| Parameters
| ----------
| workbook : {str, Workbook}
| String name or path of the workbook to create, BytesIO object, file opened
| in binary-mode, or an `xlsxwriter.Workbook` object that has not been closed.
| If None, writes to `dataframe.xlsx` in the working directory.
| worksheet : {str, Worksheet}
| Name of target worksheet or an `xlsxwriter.Worksheet` object (in which
| case `workbook` must be the parent `xlsxwriter.Workbook` object); if None,
| writes to "Sheet1" when creating a new workbook (note that writing to an
| existing workbook requires a valid existing -or new- worksheet name).
| position : {str, tuple}
| Table position in Excel notation (eg: "A1"), or a (row,col) integer tuple.
| table_style : {str, dict}
| A named Excel table style, such as "Table Style Medium 4", or a dictionary
| of `{"key":value,}` options containing one or more of the following keys:
| "style", "first_column", "last_column", "banded_columns, "banded_rows".
| table_name : str
| Name of the output table object in the worksheet; can then be referred to
| in the sheet by formulae/charts, or by subsequent `xlsxwriter` operations.
| column_formats : dict
| A `{colname(s):str,}` or `{selector:str,}` dictionary for applying an
| Excel format string to the given columns. Formats defined here (such as
| "dd/mm/yyyy", "0.00%", etc) will override any defined in `dtype_formats`.
| dtype_formats : dict
| A `{dtype:str,}` dictionary that sets the default Excel format for the
| given dtype. (This can be overridden on a per-column basis by the
| `column_formats` param).
| conditional_formats : dict
| A dictionary of colname (or selector) keys to a format str, dict, or list
| that defines conditional formatting options for the specified columns.
|
| * If supplying a string typename, should be one of the valid `xlsxwriter`
| types such as "3_color_scale", "data_bar", etc.
| * If supplying a dictionary you can make use of any/all `xlsxwriter`
| supported options, including icon sets, formulae, etc.
| * Supplying multiple columns as a tuple/key will apply a single format
| across all columns - this is effective in creating a heatmap, as the
| min/max values will be determined across the entire range, not per-column.
| * Finally, you can also supply a list made up from the above options
| in order to apply *more* than one conditional format to the same range.
| header_format : dict
| A `{key:value,}` dictionary of `xlsxwriter` format options to apply
| to the table header row, such as `{"bold":True, "font_color":"#702963"}`.
| column_totals : {bool, list, dict}
| Add a column-total row to the exported table.
|
| * If True, all numeric columns will have an associated total using "sum".
| * If passing a string, it must be one of the valid total function names
| and all numeric columns will have an associated total using that function.
| * If passing a list of colnames, only those given will have a total.
| * For more control, pass a `{colname:funcname,}` dict.
|
| Valid column-total function names are "average", "count_nums", "count",
| "max", "min", "std_dev", "sum", and "var".
| column_widths : {dict, int}
| A `{colname:int,}` or `{selector:int,}` dict or a single integer that
| sets (or overrides if autofitting) table column widths, in integer pixel
| units. If given as an integer the same value is used for all table columns.
| row_totals : {dict, list, bool}
| Add a row-total column to the right-hand side of the exported table.
|
| * If True, a column called "total" will be added at the end of the table
| that applies a "sum" function row-wise across all numeric columns.
| * If passing a list/sequence of column names, only the matching columns
| will participate in the sum.
| * Can also pass a `{colname:columns,}` dictionary to create one or
| more total columns with distinct names, referencing different columns.
| row_heights : {dict, int}
| An int or `{row_index:int,}` dictionary that sets the height of the given
| rows (if providing a dictionary) or all rows (if providing an integer) that
| intersect with the table body (including any header and total row) in
| integer pixel units. Note that `row_index` starts at zero and will be
| the header row (unless `include_header` is False).
| sparklines : dict
| A `{colname:list,}` or `{colname:dict,}` dictionary defining one or more
| sparklines to be written into a new column in the table.
|
| * If passing a list of colnames (used as the source of the sparkline data)
| the default sparkline settings are used (eg: line chart with no markers).
| * For more control an `xlsxwriter`-compliant options dict can be supplied,
| in which case three additional polars-specific keys are available:
| "columns", "insert_before", and "insert_after". These allow you to define
| the source columns and position the sparkline(s) with respect to other
| table columns. If no position directive is given, sparklines are added to
| the end of the table (eg: to the far right) in the order they are given.
| formulas : dict
| A `{colname:formula,}` or `{colname:dict,}` dictionary defining one or
| more formulas to be written into a new column in the table. Note that you
| are strongly advised to use structured references in your formulae wherever
| possible to make it simple to reference columns by name.
|
| * If providing a string formula (such as "=[@colx]*[@coly]") the column will
| be added to the end of the table (eg: to the far right), after any default
| sparklines and before any row_totals.
| * For the most control supply an options dictionary with the following keys:
| "formula" (mandatory), one of "insert_before" or "insert_after", and
| optionally "return_dtype". The latter is used to appropriately format the
| output of the formula and allow it to participate in row/column totals.
| float_precision : int
| Default number of decimals displayed for floating point columns (note that
| this is purely a formatting directive; the actual values are not rounded).
| include_header : bool
| Indicate if the table should be created with a header row.
| autofilter : bool
| If the table has headers, provide autofilter capability.
| autofit : bool
| Calculate individual column widths from the data.
| hidden_columns : str | list
| A column name, list of column names, or a selector representing table
| columns to mark as hidden in the output worksheet.
| hide_gridlines : bool
| Do not display any gridlines on the output worksheet.
| sheet_zoom : int
| Set the default zoom level of the output worksheet.
| freeze_panes : str | (str, int, int) | (int, int) | (int, int, int, int)
| Freeze workbook panes.
|
| * If (row, col) is supplied, panes are split at the top-left corner of the
| specified cell, which are 0-indexed. Thus, to freeze only the top row,
| supply (1, 0).
| * Alternatively, cell notation can be used to supply the cell. For example,
| "A2" indicates the split occurs at the top-left of cell A2, which is the
| equivalent of (1, 0).
| * If (row, col, top_row, top_col) are supplied, the panes are split based on
| the `row` and `col`, and the scrolling region is initialized to begin at
| the `top_row` and `top_col`. Thus, to freeze only the top row and have the
| scrolling region begin at row 10, column D (5th col), supply (1, 0, 9, 4).
| Using cell notation for (row, col), supplying ("A2", 9, 4) is equivalent.
| use_zip64 : bool
| Whether to use ZIP64 extensions when writing the Workbook. This allows for
| writing exceptionally large workbook files (>=4GB when uncompressed), but
| is less broadly compatible.
|
| Notes
| -----
| * A list of compatible `xlsxwriter` format property names can be found here:
| https://xlsxwriter.readthedocs.io/format.html#format-methods-and-format-properties
|
| * Conditional formatting dictionaries should provide xlsxwriter-compatible
| definitions; polars will take care of how they are applied on the worksheet
| with respect to the relative sheet/column position. For supported options,
| see: https://xlsxwriter.readthedocs.io/working_with_conditional_formats.html
|
| * Similarly, sparkline option dictionaries should contain xlsxwriter-compatible
| key/values, as well as a mandatory polars "columns" key that defines the
| sparkline source data; these source columns should all be adjacent. Two other
| polars-specific keys are available to help define where the sparkline appears
| in the table: "insert_after", and "insert_before". The value associated with
| these keys should be the name of a column in the exported table.
| https://xlsxwriter.readthedocs.io/working_with_sparklines.html
|
| * Formula dictionaries *must* contain a key called "formula", and then optional
| "insert_after", "insert_before", and/or "return_dtype" keys. These additional
| keys allow the column to be injected into the table at a specific location,
| and/or to define the return type of the formula (eg: "Int64", "Float64", etc).
| Formulas that refer to table columns should use Excel's structured references
| syntax to ensure the formula is applied correctly and is table-relative.
| https://support.microsoft.com/en-us/office/using-structured-references-with-excel-tables-f5ed2452-2337-4f71-bed3-c8ae6d2b276e
|
| * If you want unformatted output, you can use a selector to apply the "General"
| format to all columns (or all *non-temporal* columns to preserve formatting
| of date/datetime columns), eg: `column_formats={~cs.temporal(): "General"}`.
|
| Examples
| --------
| Instantiate a basic DataFrame:
|
| >>> from random import uniform
| >>> from datetime import date
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "dtm": [date(2023, 1, 1), date(2023, 1, 2), date(2023, 1, 3)],
| ... "num": [uniform(-500, 500), uniform(-500, 500), uniform(-500, 500)],
| ... "val": [10_000, 20_000, 30_000],
| ... }
| ... )
|
| Export to "dataframe.xlsx" (the default workbook name, if not specified) in the
| working directory, add column totals on all numeric columns ("sum" by default),
| then autofit:
|
| >>> df.write_excel(column_totals=True, autofit=True) # doctest: +SKIP
|
| Write frame to a specific location on the sheet, set a named table style,
| apply US-style date formatting, increase floating point formatting precision,
| apply a non-default column total function to a specific column, autofit:
|
| >>> df.write_excel( # doctest: +SKIP
| ... position="B4",
| ... table_style="Table Style Light 16",
| ... dtype_formats={pl.Date: "mm/dd/yyyy"},
| ... column_totals={"num": "average"},
| ... float_precision=6,
| ... autofit=True,
| ... )
|
| Write the same frame to a named worksheet twice, applying different styles
| and conditional formatting to each table, adding custom-formatted table
| titles using explicit `xlsxwriter` integration:
|
| >>> from xlsxwriter import Workbook
| >>> with Workbook("multi_frame.xlsx") as wb: # doctest: +SKIP
| ... # basic/default conditional formatting
| ... df.write_excel(
| ... workbook=wb,
| ... worksheet="data",
| ... position=(3, 1), # specify position as (row,col) coordinates
| ... conditional_formats={"num": "3_color_scale", "val": "data_bar"},
| ... table_style="Table Style Medium 4",
| ... )
| ...
| ... # advanced conditional formatting, custom styles
| ... df.write_excel(
| ... workbook=wb,
| ... worksheet="data",
| ... position=(df.height + 7, 1),
| ... table_style={
| ... "style": "Table Style Light 4",
| ... "first_column": True,
| ... },
| ... conditional_formats={
| ... "num": {
| ... "type": "3_color_scale",
| ... "min_color": "#76933c",
| ... "mid_color": "#c4d79b",
| ... "max_color": "#ebf1de",
| ... },
| ... "val": {
| ... "type": "data_bar",
| ... "data_bar_2010": True,
| ... "bar_color": "#9bbb59",
| ... "bar_negative_color_same": True,
| ... "bar_negative_border_color_same": True,
| ... },
| ... },
| ... column_formats={"num": "#,##0.000;[White]-#,##0.000"},
| ... column_widths={"val": 125},
| ... autofit=True,
| ... )
| ...
| ... # add some table titles (with a custom format)
| ... ws = wb.get_worksheet_by_name("data")
| ... fmt_title = wb.add_format(
| ... {
| ... "font_color": "#4f6228",
| ... "font_size": 12,
| ... "italic": True,
| ... "bold": True,
| ... }
| ... )
| ... ws.write(2, 1, "Basic/default conditional formatting", fmt_title)
| ... ws.write(df.height + 6, 1, "Custom conditional formatting", fmt_title)
|
| Export a table containing two different types of sparklines. Use default
| options for the "trend" sparkline and customized options (and positioning)
| for the "+/-" `win_loss` sparkline, with non-default integer formatting,
| column totals, a subtle two-tone heatmap and hidden worksheet gridlines:
|
| >>> df = pl.DataFrame(
| ... {
| ... "id": ["aaa", "bbb", "ccc", "ddd", "eee"],
| ... "q1": [100, 55, -20, 0, 35],
| ... "q2": [30, -10, 15, 60, 20],
| ... "q3": [-50, 0, 40, 80, 80],
| ... "q4": [75, 55, 25, -10, -55],
| ... }
| ... )
| >>> df.write_excel( # doctest: +SKIP
| ... table_style="Table Style Light 2",
| ... # apply accounting format to all flavours of integer
| ... dtype_formats={dt: "#,##0_);(#,##0)" for dt in [pl.Int32, pl.Int64]},
| ... sparklines={
| ... # default options; just provide source cols
| ... "trend": ["q1", "q2", "q3", "q4"],
| ... # customized sparkline type, with positioning directive
| ... "+/-": {
| ... "columns": ["q1", "q2", "q3", "q4"],
| ... "insert_after": "id",
| ... "type": "win_loss",
| ... },
| ... },
| ... conditional_formats={
| ... # create a unified multi-column heatmap
| ... ("q1", "q2", "q3", "q4"): {
| ... "type": "2_color_scale",
| ... "min_color": "#95b3d7",
| ... "max_color": "#ffffff",
| ... },
| ... },
| ... column_totals=["q1", "q2", "q3", "q4"],
| ... row_totals=True,
| ... hide_gridlines=True,
| ... )
|
| Export a table containing an Excel formula-based column that calculates a
| standardised Z-score, showing use of structured references in conjunction
| with positioning directives, column totals, and custom formatting.
|
| >>> df = pl.DataFrame(
| ... {
| ... "id": ["a123", "b345", "c567", "d789", "e101"],
| ... "points": [99, 45, 50, 85, 35],
| ... }
| ... )
| >>> df.write_excel( # doctest: +SKIP
| ... table_style={
| ... "style": "Table Style Medium 15",
| ... "first_column": True,
| ... },
| ... column_formats={
| ... "id": {"font": "Consolas"},
| ... "points": {"align": "center"},
| ... "z-score": {"align": "center"},
| ... },
| ... column_totals="average",
| ... formulas={
| ... "z-score": {
| ... # use structured references to refer to the table columns and 'totals' row
| ... "formula": "=STANDARDIZE([@points], [[#Totals],[points]], STDEV([points]))",
| ... "insert_after": "points",
| ... "return_dtype": pl.Float64,
| ... }
| ... },
| ... hide_gridlines=True,
| ... sheet_zoom=125,
| ... )
|
| Create and reference a Worksheet object directly, adding a basic chart.
| Taking advantage of structured references to set chart series values and
| categories is *strongly* recommended so you do not have to calculate
| cell positions with respect to the frame data and worksheet:
|
| >>> with Workbook("basic_chart.xlsx") as wb: # doctest: +SKIP
| ... # create worksheet object and write frame data to it
| ... ws = wb.add_worksheet("demo")
| ... df.write_excel(
| ... workbook=wb,
| ... worksheet=ws,
| ... table_name="DataTable",
| ... table_style="Table Style Medium 26",
| ... hide_gridlines=True,
| ... )
| ... # create chart object, point to the written table
| ... # data using structured references, and style it
| ... chart = wb.add_chart({"type": "column"})
| ... chart.set_title({"name": "Example Chart"})
| ... chart.set_legend({"none": True})
| ... chart.set_style(38)
| ... chart.add_series(
| ... { # note the use of structured references
| ... "values": "=DataTable[points]",
| ... "categories": "=DataTable[id]",
| ... "data_labels": {"value": True},
| ... }
| ... )
| ... # add chart to the worksheet
| ... ws.insert_chart("D1", chart)
|
| Export almost entirely unformatted data (no numeric styling or standardised
| floating point precision), omit autofilter, but keep date/datetime formatting:
|
| >>> import polars.selectors as cs
| >>> df = pl.DataFrame(
| ... {
| ... "n1": [-100, None, 200, 555],
| ... "n2": [987.4321, -200, 44.444, 555.5],
| ... }
| ... )
| >>> df.write_excel( # doctest: +SKIP
| ... column_formats={~cs.temporal(): "General"},
| ... autofilter=False,
| ... )
|
| write_iceberg(self, target: 'str | pyiceberg.table.Table', mode: "Literal['append', 'overwrite']") -> 'None'
| Write DataFrame to an Iceberg table.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| target
| Name of the table or the Table object representing an Iceberg table.
| mode : {'append', 'overwrite'}
| How to handle existing data.
|
| - If 'append', will add new data.
| - If 'overwrite', will replace table with new data.
|
| write_ipc(self, file: 'str | Path | IO[bytes] | None', *, compression: 'IpcCompression' = 'uncompressed', compat_level: 'CompatLevel | None' = None, record_batch_size: 'int | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None) -> 'BytesIO | None'
| Write to Arrow IPC binary stream or Feather file.
|
| See "File or Random Access format" in https://arrow.apache.org/docs/python/ipc.html.
|
| .. versionchanged:: 1.1
| The `future` parameter was renamed `compat_level`.
|
| Parameters
| ----------
| file
| Path or writable file-like object to which the IPC data will be
| written. If set to `None`, the output is returned as a BytesIO object.
| compression : {'uncompressed', 'lz4', 'zstd'}
| Compression method. Defaults to "uncompressed".
| compat_level
| Use a specific compatibility level
| when exporting Polars' internal data structures.
| record_batch_size
| Size of the record batches in number of rows.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.arrow"
| >>> df.write_ipc(path)
|
| Write to a ``BytesIO`` object by passing ``file=None``. The returned
| buffer's position is at the end of the written data, so call ``seek(0)``
| before reading it back.
|
| >>> buf = df.write_ipc(file=None)
| >>> buf.seek(0)
| 0
| >>> pl.read_ipc(buf).equals(df)
| True
|
| write_ipc_stream(self, file: 'str | Path | IO[bytes] | None', *, compression: 'IpcCompression' = 'uncompressed', compat_level: 'CompatLevel | None' = None) -> 'BytesIO | None'
| Write to Arrow IPC record batch stream.
|
| See "Streaming format" in https://arrow.apache.org/docs/python/ipc.html.
|
| .. versionchanged:: 1.1
| The `future` parameter was renamed `compat_level`.
|
| Parameters
| ----------
| file
| Path or writable file-like object to which the IPC record batch data will
| be written. If set to `None`, the output is returned as a BytesIO object.
| compression : {'uncompressed', 'lz4', 'zstd'}
| Compression method. Defaults to "uncompressed".
| compat_level
| Use a specific compatibility level
| when exporting Polars' internal data structures.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.arrow"
| >>> df.write_ipc_stream(path)
|
| write_json(self, file: 'IOBase | str | Path | None' = None) -> 'str | None'
| Serialize to JSON representation.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| If set to `None` (default), the output is returned as a string instead.
|
| See Also
| --------
| DataFrame.write_ndjson
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... }
| ... )
| >>> df.write_json()
| '[{"foo":1,"bar":6},{"foo":2,"bar":7},{"foo":3,"bar":8}]'
|
| write_ndjson(self, file: 'str | Path | IO[bytes] | IO[str] | None' = None, *, compression: "Literal['uncompressed', 'gzip', 'zstd']" = 'uncompressed', compression_level: 'int | None' = None, check_extension: 'bool' = True) -> 'str | None'
| Serialize to newline delimited JSON representation.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| If set to `None` (default), the output is returned as a string instead.
| compression
| What compression format to use.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| compression_level
| The compression level to use, typically 0-9 or `None` to let the
| engine choose.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| check_extension
| Whether to check if the filename matches the compression settings.
| Will raise an error if compression is set to 'uncompressed' and the
| filename ends in one of (".gz", ".zst", ".zstd") or if
| compression != 'uncompressed' and the file uses an mismatched
| extension. Only applies if file is a path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... }
| ... )
| >>> df.write_ndjson()
| '{"foo":1,"bar":6}\n{"foo":2,"bar":7}\n{"foo":3,"bar":8}\n'
|
| write_parquet(self, file: 'str | Path | IO[bytes]', *, compression: 'ParquetCompression' = 'zstd', compression_level: 'int | None' = None, statistics: 'bool | str | dict[str, bool]' = True, row_group_size: 'int | None' = None, data_page_size: 'int | None' = None, use_pyarrow: 'bool' = False, pyarrow_options: 'dict[str, Any] | None' = None, partition_by: 'str | Sequence[str] | None' = None, partition_chunk_size_bytes: 'int' = 4294967296, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, metadata: 'ParquetMetadata | None' = None, arrow_schema: 'ArrowSchemaExportable | None' = None, mkdir: 'bool' = False) -> 'None'
| Write to Apache Parquet file.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| This should be a path to a directory if writing a partitioned dataset.
| compression : {'lz4', 'uncompressed', 'snappy', 'gzip', 'brotli', 'zstd'}
| Choose "zstd" for good compression performance.
| Choose "lz4" for fast compression/decompression.
| Choose "snappy" for more backwards compatibility guarantees
| when you deal with older parquet readers.
| compression_level
| The level of compression to use. Higher compression means smaller files on
| disk.
|
| - "gzip" : min-level: 0, max-level: 9, default: 6.
| - "brotli" : min-level: 0, max-level: 11, default: 1.
| - "zstd" : min-level: 1, max-level: 22, default: 3.
|
| statistics
| Write statistics to the parquet headers. This is the default behavior.
|
| Possible values:
|
| - `True`: enable default set of statistics (default). Some
| statistics may be disabled.
| - `False`: disable all statistics
| - "full": calculate and write all available statistics. Cannot be
| combined with `use_pyarrow`.
| - `{ "statistic-key": True / False, ... }`. Cannot be combined with
| `use_pyarrow`. Available keys:
|
| - "min": column minimum value (default: `True`)
| - "max": column maximum value (default: `True`)
| - "distinct_count": number of unique column values (default: `False`)
| - "null_count": number of null values in column (default: `True`)
| row_group_size
| Size of the row groups in number of rows. Defaults to 512^2 rows.
| data_page_size
| Size of the data page in bytes. Defaults to 1024^2 bytes.
| use_pyarrow
| Use PyArrow's C++ parquet implementation instead of Polars' native
| Rust implementation. This may be useful when specific PyArrow features
| are needed via ``pyarrow_options``. Some options are not supported when
| enabled (e.g. ``statistics="full"``, ``metadata``, ``mkdir``).
| pyarrow_options
| Arguments passed to `pyarrow.parquet.write_table`.
|
| If you pass `partition_cols` here, the dataset will be written
| using `pyarrow.parquet.write_to_dataset`.
| The `partition_cols` parameter leads to write the dataset to a directory.
| Similar to Spark's partitioned datasets. For native partitioned
| writes, consider using ``partition_by`` instead.
| partition_by
| Column(s) to partition by. A partitioned dataset will be written if this is
| specified. This parameter is considered unstable and is subject to change.
| partition_chunk_size_bytes
| Approximate size to split DataFrames within a single partition when
| writing. Note this is calculated using the size of the DataFrame in
| memory - the size of the output file may differ depending on the
| file format / compression.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| metadata
| A dictionary or callback to add key-values to the file-level Parquet
| metadata.
|
| .. warning::
| This functionality is considered **experimental**. It may be removed or
| changed at any point without it being considered a breaking change.
|
| arrow_schema
| Provide a custom arrow schema to write to the file. This allows
| setting custom schema and field-level metadata. Names and dtypes
| must match.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.parquet"
| >>> df.write_parquet(path)
|
| We can write partitioned datasets. The following example will write
| the first row to ../watermark=1/*.parquet and the other rows to
| ../watermark=2/*.parquet.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3], "watermark": [1, 2, 2]})
| >>> path: pathlib.Path = dirpath / "partitioned_object"
| >>> df.write_parquet(
| ... path,
| ... partition_by=["watermark"],
| ... )
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| deserialize(source: 'str | bytes | Path | IOBase', *, format: 'SerializationFormat' = 'binary') -> 'DataFrame'
| Read a serialized DataFrame from a file.
|
| Parameters
| ----------
| source
| Path to a file or a file-like object (by file-like object, we refer to
| objects that have a `read()` method, such as a file handler (e.g.
| via builtin `open` function) or `BytesIO`).
| format
| The format with which the DataFrame was serialized. Options:
|
| - `"binary"`: Deserialize from binary format (bytes). This is the default.
| - `"json"`: Deserialize from JSON format (string).
|
| See Also
| --------
| DataFrame.serialize
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| >>> import io
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4.0, 5.0, 6.0]})
| >>> bytes = df.serialize()
| >>> pl.DataFrame.deserialize(io.BytesIO(bytes))
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 4.0 │
| │ 2 ┆ 5.0 │
| │ 3 ┆ 6.0 │
| └─────┴─────┘
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| dtypes
| Get the column data types.
|
| The data types can also be found in column headers when printing the DataFrame.
|
| Returns
| -------
| list of DataType
| A list containing the data type of each column in order.
|
| See Also
| --------
| schema
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.dtypes
| [Int64, Float64, String]
| >>> df
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 7.0 ┆ b │
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| flags
| Get flags that are set on the columns of this DataFrame.
|
| Returns
| -------
| dict
| Mapping from column names to column flags.
|
| height
| Get the number of rows.
|
| Returns
| -------
| int
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5]})
| >>> df.height
| 5
|
| plot
| Create a plot namespace.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.6.0
| In prior versions of Polars, HvPlot was the plotting backend. If you would
| like to restore the previous plotting functionality, all you need to do
| is add `import hvplot.polars` at the top of your script and replace
| `df.plot` with `df.hvplot`.
|
| Polars does not implement plotting logic itself, but instead defers to
| `Altair <https://altair-viz.github.io/>`_:
|
| - `df.plot.line(**kwargs)`
| is shorthand for
| `alt.Chart(df).mark_line(tooltip=True).encode(**kwargs).interactive()`
| - `df.plot.point(**kwargs)`
| is shorthand for
| `alt.Chart(df).mark_point(tooltip=True).encode(**kwargs).interactive()` (and
| `plot.scatter` is provided as an alias)
| - `df.plot.bar(**kwargs)`
| is shorthand for
| `alt.Chart(df).mark_bar(tooltip=True).encode(**kwargs).interactive()`
| - for any other attribute `attr`, `df.plot.attr(**kwargs)`
| is shorthand for
| `alt.Chart(df).mark_attr(tooltip=True).encode(**kwargs).interactive()`
|
| For configuration, we suggest reading
| `Chart Configuration <https://altair-viz.github.io/altair-tutorial/notebooks/08-Configuration.html>`_.
| For example, you can:
|
| - Change the width/height/title with
| ``.properties(width=500, height=350, title="My amazing plot")``.
| - Change the x-axis label rotation with ``.configure_axisX(labelAngle=30)``.
| - Change the opacity of the points in your scatter plot with
| ``.configure_point(opacity=.5)``.
|
| Examples
| --------
| Scatter plot:
|
| >>> df = pl.DataFrame(
| ... {
| ... "length": [1, 4, 6],
| ... "width": [4, 5, 6],
| ... "species": ["setosa", "setosa", "versicolor"],
| ... }
| ... )
| >>> df.plot.point(x="length", y="width", color="species") # doctest: +SKIP
|
| Set the x-axis title by using ``altair.X``:
|
| >>> import altair as alt
| >>> df.plot.point(
| ... x=alt.X("length", title="Length"), y="width", color="species"
| ... ) # doctest: +SKIP
|
| Line plot:
|
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "date": [date(2020, 1, 2), date(2020, 1, 3), date(2020, 1, 4)] * 2,
| ... "price": [1, 4, 6, 1, 5, 2],
| ... "stock": ["a", "a", "a", "b", "b", "b"],
| ... }
| ... )
| >>> df.plot.line(x="date", y="price", color="stock") # doctest: +SKIP
|
| Bar plot:
|
| >>> df = pl.DataFrame(
| ... {
| ... "day": ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"] * 2,
| ... "group": ["a"] * 7 + ["b"] * 7,
| ... "value": [1, 3, 2, 4, 5, 6, 1, 1, 3, 2, 4, 5, 1, 2],
| ... }
| ... )
| >>> df.plot.bar(
| ... x="day", y="value", color="day", column="group"
| ... ) # doctest: +SKIP
|
| Or, to make a stacked version of the plot above:
|
| >>> df.plot.bar(x="day", y="value", color="group") # doctest: +SKIP
|
| schema
| Get an ordered mapping of column names to their data type.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.schema
| Schema({'foo': Int64, 'bar': Float64, 'ham': String})
|
| shape
| Get the shape of the DataFrame.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5]})
| >>> df.shape
| (5, 1)
|
| style
| Create a Great Table for styling.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Polars does not implement styling logic itself, but instead defers to
| the Great Tables package. Please see the `Great Tables reference <https://posit-dev.github.io/great-tables/reference/>`_
| for more information and documentation.
|
| Examples
| --------
| Import some styling helpers, and create example data:
|
| >>> import polars.selectors as cs
| >>> from great_tables import loc, style
| >>> df = pl.DataFrame(
| ... {
| ... "site_id": [0, 1, 2],
| ... "measure_a": [5, 4, 6],
| ... "measure_b": [7, 3, 3],
| ... }
| ... )
|
| Emphasize the site_id as row names:
|
| >>> df.style.tab_stub(rowname_col="site_id") # doctest: +SKIP
|
| Fill the background for the highest measure_a value row:
|
| >>> df.style.tab_style(
| ... style.fill("yellow"),
| ... loc.body(rows=pl.col("measure_a") == pl.col("measure_a").max()),
| ... ) # doctest: +SKIP
|
| Put a spanner (high-level label) over measure columns:
|
| >>> df.style.tab_spanner(
| ... "Measures", cs.starts_with("measure")
| ... ) # doctest: +SKIP
|
| Format measure_b values to two decimal places:
|
| >>> df.style.fmt_number("measure_b", decimals=2) # doctest: +SKIP
|
| width
| Get the number of columns.
|
| Returns
| -------
| int
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4, 5, 6],
| ... }
| ... )
| >>> df.width
| 2
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| columns
| Get or set column names.
|
| Returns
| -------
| list of str
| A list containing the name of each column in order.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.columns
| ['foo', 'bar', 'ham']
|
| Set column names:
|
| >>> df.columns = ["apple", "banana", "orange"]
| >>> df
| shape: (3, 3)
| ┌───────┬────────┬────────┐
| │ apple ┆ banana ┆ orange │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═══════╪════════╪════════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └───────┴────────┴────────┘
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set[str]]', '_df': 'PyDataF...
|
| __hash__ = None
polars.DataType
Python Library Documentation: class DataType in module polars.datatypes.classes
class DataType(builtins.object)
| Base class for all Polars data types.
|
| Methods defined here:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
polars.DataTypeExpr
Python Library Documentation: class DataTypeExpr in module polars.datatype_expr.datatype_expr
class DataTypeExpr(builtins.object)
| A lazily instantiated :class:`DataType` that can be used in an :class:`Expr`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| This expression is made to represent a :class:`DataType` that can be used to
| reference a datatype in a lazy context.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"a": [1, 2, 3]})
| >>> lf.with_columns(
| ... pl.col.a.map_batches(lambda x: x * 2, return_dtype=pl.dtype_of("a"))
| ... ).collect()
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 4 │
| │ 6 │
| └─────┘
|
| Methods defined here:
|
| __eq__(self, value: 'PolarsDataType | DataTypeExpr') -> 'pl.Expr'
| Return self==value.
|
| __ne__(self, value: 'PolarsDataType | DataTypeExpr') -> 'pl.Expr'
| Return self!=value.
|
| collect_dtype(self, context: 'SchemaDict | pl.Schema | pl.DataFrame | pl.LazyFrame') -> 'DataType'
| Materialize the :class:`DataTypeExpr` in a specific context.
|
| This is a useful function when debugging datatype expressions.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3],
| ... }
| ... )
| >>> pl.dtype_of("a").collect_dtype(lf)
| Int64
| >>> pl.dtype_of("a").collect_dtype({"a": pl.String})
| String
|
| default_value(self, n: 'int' = 1, *, numeric_to_one: 'bool' = False, num_list_values: 'int' = 0) -> 'pl.Expr'
| Get a default value of a specific type.
|
| - Integers and floats are their zero value as default, unless otherwise
| specified
| - Temporals are a physical zero as default
| - `pl.Decimal` is zero as default
| - `pl.String` and `pl.Binary` are an empty string
| - `pl.List` is an empty list, unless otherwise specified
| - `pl.Array` is the inner default value repeated over the shape
| - `pl.Struct` is the inner default value for all fields
| - `pl.Enum` is the first category if it exists
| - `pl.Null`, `pl.Object` and `pl.Categorical` are `null`.
|
| Parameters
| ----------
| n
| Number of types you want the value
| numeric_to_one
| Use `1` instead of `0` as the default value for numeric types
| num_list_values
| The amount of values a list contains
|
| Examples
| --------
| >>> uint32 = pl.UInt32.to_dtype_expr()
| >>> pl.select(default=uint32.default_value())
| shape: (1, 1)
| ┌─────────┐
| │ default │
| │ --- │
| │ u32 │
| ╞═════════╡
| │ 0 │
| └─────────┘
|
| display(self) -> 'pl.Expr'
| Get a formatted version of the output DataType.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["X", "Y", "Z"],
| ... "c": [1.3, 3.7, 4.2],
| ... }
| ... )
| >>> df.select(
| ... a=pl.dtype_of("a").display(),
| ... b=pl.dtype_of("b").display(),
| ... c=pl.dtype_of("c").display(),
| ... ).transpose(include_header=True, column_names=["dtype"])
| shape: (3, 2)
| ┌────────┬───────┐
| │ column ┆ dtype │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞════════╪═══════╡
| │ a ┆ i64 │
| │ b ┆ str │
| │ c ┆ f64 │
| └────────┴───────┘
|
| inner_dtype(self) -> 'DataTypeExpr'
| Get the inner DataType of a List or Array.
|
| matches(self, selector: 'pl.Selector') -> 'pl.Expr'
| Get whether the output DataType is matches a certain selector.
|
| Examples
| --------
| >>> import polars.selectors as cs
| >>> pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... }
| ... ).select(
| ... a_is_string=pl.dtype_of("a").matches(cs.string()),
| ... a_is_integer=pl.dtype_of("a").matches(cs.integer()),
| ... )
| shape: (1, 2)
| ┌─────────────┬──────────────┐
| │ a_is_string ┆ a_is_integer │
| │ --- ┆ --- │
| │ bool ┆ bool │
| ╞═════════════╪══════════════╡
| │ false ┆ true │
| └─────────────┴──────────────┘
|
| to_signed_integer(self) -> 'pl.DataTypeExpr'
| Get the signed integer version of the same bitsize.
|
| Examples
| --------
| >>> uint32 = pl.UInt32.to_dtype_expr()
| >>> uint32.to_signed_integer().collect_dtype({})
| Int32
|
| to_unsigned_integer(self) -> 'pl.DataTypeExpr'
| Get the unsigned integer version of the same bitsize.
|
| Examples
| --------
| >>> int32 = pl.Int32.to_dtype_expr()
| >>> int32.to_unsigned_integer().collect_dtype({})
| UInt32
|
| wrap_in_array(self, *, width: 'int') -> 'DataTypeExpr'
| Get the DataType wrapped in an array.
|
| Examples
| --------
| >>> pl.Int32.to_dtype_expr().wrap_in_array(width=5).collect_dtype({})
| Array(Int32, shape=(5,))
|
| wrap_in_list(self) -> 'DataTypeExpr'
| Get the DataType wrapped in a list.
|
| Examples
| --------
| >>> pl.Int32.to_dtype_expr().wrap_in_list().collect_dtype({})
| List(Int32)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| arr
| Create an object namespace of all array related methods.
|
| list
| Create an object namespace of all list related methods.
|
| struct
| Create an object namespace of all struct related methods.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set[str]]', '_pydatatype_ex...
|
| __hash__ = None
polars.Date
Python Library Documentation: class Date in module polars.datatypes.classes
class Date(TemporalType)
| Data type representing a calendar date.
|
| Notes
| -----
| The underlying representation of this type is a 32-bit signed integer.
| The integer indicates the number of days since the Unix epoch (1970-01-01).
| The number can be negative to indicate dates before the epoch.
|
| Method resolution order:
| Date
| TemporalType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Datetime
Python Library Documentation: class Datetime in module polars.datatypes.classes
class Datetime(TemporalType)
| Datetime(time_unit: 'TimeUnit' = 'us', time_zone: 'str | tzinfo | None' = None) -> 'None'
|
| Data type representing a calendar date and time of day.
|
| Parameters
| ----------
| time_unit : {'us', 'ns', 'ms'}
| Unit of time. Defaults to `'us'` (microseconds).
| time_zone
| Time zone string, as defined in zoneinfo (to see valid strings run
| `import zoneinfo; zoneinfo.available_timezones()` for a full list).
| When used to match dtypes, can set this to "*" to check for Datetime
| columns that have any (non-null) timezone.
|
| Notes
| -----
| The underlying representation of this type is a 64-bit signed integer.
| The integer indicates the number of time units since the Unix epoch
| (1970-01-01 00:00:00). The number can be negative to indicate datetimes before the
| epoch.
|
| Method resolution order:
| Datetime
| TemporalType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, time_unit: 'TimeUnit' = 'us', time_zone: 'str | tzinfo | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'time_unit': 'TimeUnit', 'time_zone': 'str | None'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Decimal
Python Library Documentation: class Decimal in module polars.datatypes.classes
class Decimal(NumericType)
| Decimal(precision: 'int | None' = None, scale: 'int' = 0) -> 'None'
|
| Decimal 128-bit type with an optional precision and non-negative scale.
|
| Parameters
| ----------
| precision
| Maximum number of digits in each number.
| If set to `None` (default), the precision is set to 38 (the maximum
| supported by Polars).
| scale
| Number of digits to the right of the decimal point in each number.
|
| Method resolution order:
| Decimal
| NumericType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, precision: 'int | None' = None, scale: 'int' = 0) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'precision': 'int', 'scale': 'int'}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Duration
Python Library Documentation: class Duration in module polars.datatypes.classes
class Duration(TemporalType)
| Duration(time_unit: 'TimeUnit' = 'us') -> 'None'
|
| Data type representing a time duration.
|
| Parameters
| ----------
| time_unit : {'us', 'ns', 'ms'}
| Unit of time. Defaults to `'us'` (microseconds).
|
| Notes
| -----
| The underlying representation of this type is a 64-bit signed integer.
| The integer indicates an amount of time units and can be negative to indicate
| negative time offsets.
|
| Method resolution order:
| Duration
| TemporalType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, time_unit: 'TimeUnit' = 'us') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'time_unit': 'TimeUnit'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Enum
Python Library Documentation: class Enum in module polars.datatypes.classes
class Enum(DataType)
| Enum(categories: 'Series | Iterable[str] | type[enum.Enum]') -> 'None'
|
| A fixed categorical encoding of a unique set of strings.
|
| Parameters
| ----------
| categories
| The categories in the dataset; must be a unique set of strings, or an
| existing Python string-valued enum.
|
| Examples
| --------
| Explicitly define enumeration categories:
|
| >>> pl.Enum(["north", "south", "east", "west"])
| Enum(categories=['north', 'south', 'east', 'west'])
|
| Initialise from an existing Python enumeration:
|
| >>> from http import HTTPMethod
| >>> pl.Enum(HTTPMethod)
| Enum(categories=['CONNECT', 'DELETE', 'GET', 'HEAD', 'OPTIONS', 'PATCH', 'POST', 'PUT', 'TRACE'])
|
| Method resolution order:
| Enum
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, categories: 'Series | Iterable[str] | type[enum.Enum]') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __or__ = union(self, other: 'Enum') -> 'Enum'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| union(self, other: 'Enum') -> 'Enum'
| Union of two Enums.
|
| .. deprecated:: 1.38
| `Enum.union()` is deprecated and will be removed in version 2.0.
| Enums are ordered sets and union cannot preserve both orderings.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'categories': 'Series'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Expr
Python Library Documentation: class Expr in module polars.expr.expr
class Expr(builtins.object)
| Expressions that can be used in various contexts.
|
| Methods defined here:
|
| __abs__(self) -> 'Expr'
|
| __add__(self, other: 'IntoExpr') -> 'Expr'
| # operators
|
| __and__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
|
| __array_ufunc__(self, ufunc: 'Callable[..., Any]', method: 'str_', *inputs: 'Any', **kwargs: 'Any') -> 'Expr'
| Numpy universal functions.
|
| __bool__(self) -> 'NoReturn'
|
| __eq__(self, other: 'IntoExpr') -> 'Expr'
| Return self==value.
|
| __floordiv__(self, other: 'IntoExpr') -> 'Expr'
|
| __ge__(self, other: 'IntoExpr') -> 'Expr'
| Return self>=value.
|
| __getstate__(self) -> 'bytes'
| Helper for pickle.
|
| __gt__(self, other: 'IntoExpr') -> 'Expr'
| Return self>value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __invert__(self) -> 'Expr'
|
| __le__(self, other: 'IntoExpr') -> 'Expr'
| Return self<=value.
|
| __lt__(self, other: 'IntoExpr') -> 'Expr'
| Return self<value.
|
| __mod__(self, other: 'IntoExpr') -> 'Expr'
|
| __mul__(self, other: 'IntoExpr') -> 'Expr'
|
| __ne__(self, other: 'IntoExpr') -> 'Expr'
| Return self!=value.
|
| __neg__(self) -> 'Expr'
|
| __or__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
| Return self|value.
|
| __pos__(self) -> 'Expr'
|
| __pow__(self, exponent: 'IntoExprColumn | int | float') -> 'Expr'
|
| __radd__(self, other: 'IntoExpr') -> 'Expr'
|
| __rand__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
|
| __repr__(self) -> 'str_'
| Return repr(self).
|
| __rfloordiv__(self, other: 'IntoExpr') -> 'Expr'
|
| __rmod__(self, other: 'IntoExpr') -> 'Expr'
|
| __rmul__(self, other: 'IntoExpr') -> 'Expr'
|
| __ror__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
| Return value|self.
|
| __rpow__(self, base: 'IntoExprColumn | int | float') -> 'Expr'
|
| __rsub__(self, other: 'IntoExpr') -> 'Expr'
|
| __rtruediv__(self, other: 'IntoExpr') -> 'Expr'
|
| __rxor__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
|
| __setstate__(self, state: 'bytes') -> 'None'
|
| __str__(self) -> 'str_'
| Return str(self).
|
| __sub__(self, other: 'IntoExpr') -> 'Expr'
|
| __truediv__(self, other: 'IntoExpr') -> 'Expr'
|
| __xor__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
|
| abs(self) -> 'Expr'
| Compute absolute values.
|
| Same as `abs(expr)`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [-1.0, 0.0, 1.0, 2.0],
| ... }
| ... )
| >>> df.select(pl.col("A").abs())
| shape: (4, 1)
| ┌─────┐
| │ A │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| │ 0.0 │
| │ 1.0 │
| │ 2.0 │
| └─────┘
|
| add(self, other: 'Any') -> 'Expr'
| Method equivalent of addition operator `expr + other`.
|
| Parameters
| ----------
| other
| numeric or string value; accepts expression input.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 3, 4, 5]})
| >>> df.with_columns(
| ... pl.col("x").add(2).alias("x+int"),
| ... pl.col("x").add(pl.col("x").cum_prod()).alias("x+expr"),
| ... )
| shape: (5, 3)
| ┌─────┬───────┬────────┐
| │ x ┆ x+int ┆ x+expr │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═══════╪════════╡
| │ 1 ┆ 3 ┆ 2 │
| │ 2 ┆ 4 ┆ 4 │
| │ 3 ┆ 5 ┆ 9 │
| │ 4 ┆ 6 ┆ 28 │
| │ 5 ┆ 7 ┆ 125 │
| └─────┴───────┴────────┘
|
| >>> df = pl.DataFrame(
| ... {"x": ["a", "d", "g"], "y": ["b", "e", "h"], "z": ["c", "f", "i"]}
| ... )
| >>> df.with_columns(pl.col("x").add(pl.col("y")).add(pl.col("z")).alias("xyz"))
| shape: (3, 4)
| ┌─────┬─────┬─────┬─────┐
| │ x ┆ y ┆ z ┆ xyz │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ str ┆ str │
| ╞═════╪═════╪═════╪═════╡
| │ a ┆ b ┆ c ┆ abc │
| │ d ┆ e ┆ f ┆ def │
| │ g ┆ h ┆ i ┆ ghi │
| └─────┴─────┴─────┴─────┘
|
| agg_groups(self) -> 'Expr'
| Get the group indexes of the group by operation.
|
| .. deprecated:: 1.35
| use `df.with_row_index().group_by(...).agg(pl.col('index'))` instead.
| This method will be removed in Polars 2.0.
|
| Should be used in aggregation context only.
|
| Examples
| --------
| >>> import warnings
| >>> warnings.filterwarnings("ignore", category=DeprecationWarning)
| >>> df = pl.DataFrame(
| ... {
| ... "group": [
| ... "one",
| ... "one",
| ... "one",
| ... "two",
| ... "two",
| ... "two",
| ... ],
| ... "value": [94, 95, 96, 97, 97, 99],
| ... }
| ... )
| >>> df.group_by("group", maintain_order=True).agg(pl.col("value").agg_groups())
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ list[u32] │
| ╞═══════╪═══════════╡
| │ one ┆ [0, 1, 2] │
| │ two ┆ [3, 4, 5] │
| └───────┴───────────┘
|
| New recommended approach:
| >>> (
| ... df.with_row_index()
| ... .group_by("group", maintain_order=True)
| ... .agg(pl.col("index"))
| ... )
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ index │
| │ --- ┆ --- │
| │ str ┆ list[u32] │
| ╞═══════╪═══════════╡
| │ one ┆ [0, 1, 2] │
| │ two ┆ [3, 4, 5] │
| └───────┴───────────┘
|
| alias(self, name: 'str_') -> 'Expr'
| Rename the expression.
|
| Parameters
| ----------
| name
| The new name.
|
| See Also
| --------
| name.map
| name.prefix
| name.suffix
|
| Examples
| --------
| Rename an expression to avoid overwriting an existing column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["x", "y", "z"],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("a") + 10,
| ... pl.col("b").str.to_uppercase().alias("c"),
| ... )
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 11 ┆ x ┆ X │
| │ 12 ┆ y ┆ Y │
| │ 13 ┆ z ┆ Z │
| └─────┴─────┴─────┘
|
| Overwrite the default name of literal columns to prevent errors due to duplicate
| column names.
|
| >>> df.with_columns(
| ... pl.lit(True).alias("c"),
| ... pl.lit(4.0).alias("d"),
| ... )
| shape: (3, 4)
| ┌─────┬─────┬──────┬─────┐
| │ a ┆ b ┆ c ┆ d │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ bool ┆ f64 │
| ╞═════╪═════╪══════╪═════╡
| │ 1 ┆ x ┆ true ┆ 4.0 │
| │ 2 ┆ y ┆ true ┆ 4.0 │
| │ 3 ┆ z ┆ true ┆ 4.0 │
| └─────┴─────┴──────┴─────┘
|
| all(self, *, ignore_nulls: 'bool' = True) -> 'Expr'
| Return whether all values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| .. note::
| This method is not to be confused with the function :func:`polars.all`,
| which can be used to select all columns.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `True`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `False` values,
| the output is null.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [True, True],
| ... "b": [False, True],
| ... "c": [None, True],
| ... }
| ... )
| >>> df.select(pl.col("*").all())
| shape: (1, 3)
| ┌──────┬───────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪══════╡
| │ true ┆ false ┆ true │
| └──────┴───────┴──────┘
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> df.select(pl.col("*").all(ignore_nulls=False))
| shape: (1, 3)
| ┌──────┬───────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪══════╡
| │ true ┆ false ┆ null │
| └──────┴───────┴──────┘
|
| and_(self, *others: 'Any') -> 'Expr'
| Method equivalent of bitwise "and" operator `expr & other & ...`.
|
| This has the effect of combining logical boolean expressions,
| but operates bitwise on integers.
|
| Parameters
| ----------
| *others
| One or more integer or boolean expressions to evaluate/combine.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5, 6, 7, 4, 8],
| ... "y": [1.5, 2.5, 1.0, 4.0, -5.75],
| ... "z": [-9, 2, -1, 4, 8],
| ... }
| ... )
|
| Combine logical "and" conditions:
|
| >>> df.select(
| ... (pl.col("x") >= pl.col("z"))
| ... .and_(
| ... pl.col("y") >= pl.col("z"),
| ... pl.col("y") == pl.col("y"),
| ... pl.col("z") <= pl.col("x"),
| ... pl.col("y") != pl.col("x"),
| ... )
| ... .alias("all")
| ... )
| shape: (5, 1)
| ┌───────┐
| │ all │
| │ --- │
| │ bool │
| ╞═══════╡
| │ true │
| │ true │
| │ true │
| │ false │
| │ false │
| └───────┘
|
| Bitwise "and" operation on integer columns:
|
| >>> df.select("x", "z", x_and_z=pl.col("x").and_(pl.col("z")))
| shape: (5, 3)
| ┌─────┬─────┬─────────┐
| │ x ┆ z ┆ x_and_z │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════════╡
| │ 5 ┆ -9 ┆ 5 │
| │ 6 ┆ 2 ┆ 2 │
| │ 7 ┆ -1 ┆ 7 │
| │ 4 ┆ 4 ┆ 4 │
| │ 8 ┆ 8 ┆ 8 │
| └─────┴─────┴─────────┘
|
| any(self, *, ignore_nulls: 'bool' = True) -> 'Expr'
| Return whether any of the values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `False`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `True` values,
| the output is null.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [True, False],
| ... "b": [False, False],
| ... "c": [None, False],
| ... }
| ... )
| >>> df.select(pl.col("*").any())
| shape: (1, 3)
| ┌──────┬───────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪═══════╡
| │ true ┆ false ┆ false │
| └──────┴───────┴───────┘
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> df.select(pl.col("*").any(ignore_nulls=False))
| shape: (1, 3)
| ┌──────┬───────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪══════╡
| │ true ┆ false ┆ null │
| └──────┴───────┴──────┘
|
| append(self, other: 'IntoExpr', *, upcast: 'bool' = True) -> 'Expr'
| Append expressions.
|
| This is done by adding the chunks of `other` to this `Series`.
|
| Parameters
| ----------
| other
| Expression to append.
| upcast
| Cast both `Series` to the same supertype.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [8, 9, 10],
| ... "b": [None, 4, 4],
| ... }
| ... )
| >>> df.select(pl.all().head(1).append(pl.all().tail(1)))
| shape: (2, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════╡
| │ 8 ┆ null │
| │ 10 ┆ 4 │
| └─────┴──────┘
|
| approx_n_unique(self) -> 'Expr'
| Approximate count of unique values.
|
| This is done using the HyperLogLog++ algorithm for cardinality estimation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [1, 1, 2]})
| >>> df.select(pl.col("n").approx_n_unique())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 2 │
| └─────┘
| >>> df = pl.DataFrame({"n": range(1000)})
| >>> df.select(
| ... exact=pl.col("n").n_unique(),
| ... approx=pl.col("n").approx_n_unique(),
| ... ) # doctest: +SKIP
| shape: (1, 2)
| ┌───────┬────────┐
| │ exact ┆ approx │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═══════╪════════╡
| │ 1000 ┆ 1005 │
| └───────┴────────┘
|
| arccos(self) -> 'Expr'
| Compute the element-wise value for the inverse cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`, measuring an angle in radians.
|
| Notes
| -----
| To convert the result from radians to degrees,
| call :meth:`.degrees() <polars.Expr.degrees>`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, 0.5, 0]})
| >>> df.select(pl.col("a").arccos())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.0 │
| │ 1.047198 │
| │ 1.570796 │
| └──────────┘
| >>> df.select(pl.col("a").arccos().degrees())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 0.0 │
| │ 60.0 │
| │ 90.0 │
| └──────┘
|
| arccosh(self) -> 'Expr'
| Compute the element-wise value for the inverse hyperbolic cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").arccosh())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| └─────┘
|
| arcsin(self) -> 'Expr'
| Compute the element-wise value for the inverse sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`, measuring an angle in radians.
|
| Notes
| -----
| To convert the result from radians to degrees,
| call :meth:`.degrees() <polars.Expr.degrees>`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, 0.5, 0]})
| >>> df.select(pl.col("a").arcsin())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.570796 │
| │ 0.523599 │
| │ 0.0 │
| └──────────┘
| >>> df.select(pl.col("a").arcsin().degrees())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 90.0 │
| │ 30.0 │
| │ 0.0 │
| └──────┘
|
| arcsinh(self) -> 'Expr'
| Compute the element-wise value for the inverse hyperbolic sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").arcsinh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.881374 │
| └──────────┘
|
| arctan(self) -> 'Expr'
| Compute the element-wise value for the inverse tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`, measuring an angle in radians.
|
| Notes
| -----
| To convert the result from radians to degrees,
| call :meth:`.degrees() <polars.Expr.degrees>`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [float("Inf"), 1, 0]})
| >>> df.select(pl.col("a").arctan())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.570796 │
| │ 0.785398 │
| │ 0.0 │
| └──────────┘
| >>> df.select(pl.col("a").arctan().degrees())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 90.0 │
| │ 45.0 │
| │ 0.0 │
| └──────┘
|
| arctanh(self) -> 'Expr'
| Compute the element-wise value for the inverse hyperbolic tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").arctanh())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ inf │
| └─────┘
|
| arg_max(self) -> 'Expr'
| Get the index of the maximal value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [20, 10, 30],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_max())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 2 │
| └─────┘
|
| arg_min(self) -> 'Expr'
| Get the index of the minimal value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [20, 10, 30],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_min())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| └─────┘
|
| arg_sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Expr'
| Get the index values that would sort this column.
|
| Parameters
| ----------
| descending
| Sort in descending (descending) order.
| nulls_last
| Place null values last instead of first.
|
| Returns
| -------
| Expr
| Expression of data type :class:`UInt32`.
|
| See Also
| --------
| Expr.gather: Take values by index.
| Expr.rank : Get the rank of each row.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [20, 10, 30],
| ... "b": [1, 2, 3],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_sort())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| │ 0 │
| │ 2 │
| └─────┘
|
| Use gather to apply the arg sort to other columns.
|
| >>> df.select(pl.col("b").gather(pl.col("a").arg_sort()))
| shape: (3, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 1 │
| │ 3 │
| └─────┘
|
| arg_true(self) -> 'Expr'
| Return indices where expression evaluates `True`.
|
| .. warning::
| Modifies number of rows returned, so will fail in combination with other
| expressions. Use as only expression in `select` / `with_columns`.
|
| See Also
| --------
| Series.arg_true : Return indices where Series is True
| polars.arg_where
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 1]})
| >>> df.select((pl.col("a") == 1).arg_true())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 0 │
| │ 1 │
| │ 3 │
| └─────┘
|
| arg_unique(self) -> 'Expr'
| Get index of first unique value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [8, 9, 10],
| ... "b": [None, 4, 4],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_unique())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 0 │
| │ 1 │
| │ 2 │
| └─────┘
| >>> df.select(pl.col("b").arg_unique())
| shape: (2, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 0 │
| │ 1 │
| └─────┘
|
| backward_fill(self, limit: 'int | None' = None) -> 'Expr'
| Fill missing values with the next non-null value.
|
| This is an alias of `.fill_null(strategy="backward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to backward fill.
|
| See Also
| --------
| fill_null
| forward_fill
| shift
|
| bitwise_and(self) -> 'Expr'
| Perform an aggregation of bitwise ANDs.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-1, 0, 1]})
| >>> df.select(pl.col("n").bitwise_and())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 0 │
| └─────┘
| >>> df = pl.DataFrame(
| ... {"grouper": ["a", "a", "a", "b", "b"], "n": [-1, 0, 1, -1, 1]}
| ... )
| >>> df.group_by("grouper", maintain_order=True).agg(pl.col("n").bitwise_and())
| shape: (2, 2)
| ┌─────────┬─────┐
| │ grouper ┆ n │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════╡
| │ a ┆ 0 │
| │ b ┆ 1 │
| └─────────┴─────┘
|
| bitwise_count_ones(self) -> 'Expr'
| Evaluate the number of set bits.
|
| bitwise_count_zeros(self) -> 'Expr'
| Evaluate the number of unset bits.
|
| bitwise_leading_ones(self) -> 'Expr'
| Evaluate the number most-significant set bits before seeing an unset bit.
|
| bitwise_leading_zeros(self) -> 'Expr'
| Evaluate the number most-significant unset bits before seeing a set bit.
|
| bitwise_or(self) -> 'Expr'
| Perform an aggregation of bitwise ORs.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-1, 0, 1]})
| >>> df.select(pl.col("n").bitwise_or())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ i64 │
| ╞═════╡
| │ -1 │
| └─────┘
| >>> df = pl.DataFrame(
| ... {"grouper": ["a", "a", "a", "b", "b"], "n": [-1, 0, 1, -1, 1]}
| ... )
| >>> df.group_by("grouper", maintain_order=True).agg(pl.col("n").bitwise_or())
| shape: (2, 2)
| ┌─────────┬─────┐
| │ grouper ┆ n │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════╡
| │ a ┆ -1 │
| │ b ┆ -1 │
| └─────────┴─────┘
|
| bitwise_trailing_ones(self) -> 'Expr'
| Evaluate the number least-significant set bits before seeing an unset bit.
|
| bitwise_trailing_zeros(self) -> 'Expr'
| Evaluate the number least-significant unset bits before seeing a set bit.
|
| bitwise_xor(self) -> 'Expr'
| Perform an aggregation of bitwise XORs.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-1, 0, 1]})
| >>> df.select(pl.col("n").bitwise_xor())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ i64 │
| ╞═════╡
| │ -2 │
| └─────┘
| >>> df = pl.DataFrame(
| ... {"grouper": ["a", "a", "a", "b", "b"], "n": [-1, 0, 1, -1, 1]}
| ... )
| >>> df.group_by("grouper", maintain_order=True).agg(pl.col("n").bitwise_xor())
| shape: (2, 2)
| ┌─────────┬─────┐
| │ grouper ┆ n │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════╡
| │ a ┆ -2 │
| │ b ┆ -2 │
| └─────────┴─────┘
|
| bottom_k(self, k: 'int | IntoExprColumn' = 5) -> 'Expr'
| Return the `k` smallest elements.
|
| Non-null elements are always preferred over null elements. The output is
| not guaranteed to be in any particular order, call :func:`sort` after
| this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k_by
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.select(
| ... pl.col("value").top_k().alias("top_k"),
| ... pl.col("value").bottom_k().alias("bottom_k"),
| ... )
| shape: (5, 2)
| ┌───────┬──────────┐
| │ top_k ┆ bottom_k │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪══════════╡
| │ 4 ┆ 1 │
| │ 98 ┆ 98 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| │ 99 ┆ 4 │
| └───────┴──────────┘
|
| bottom_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int | IntoExprColumn' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Expr'
| Return the elements corresponding to the `k` smallest elements of the `by` column(s).
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| by
| Column(s) used to determine the smallest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` largest elements of the `by` column(s) (instead of the `k`
| smallest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [6, 5, 4, 3, 2, 1],
| ... "c": ["Apple", "Orange", "Apple", "Apple", "Banana", "Banana"],
| ... }
| ... )
| >>> df
| shape: (6, 3)
| ┌─────┬─────┬────────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪════════╡
| │ 1 ┆ 6 ┆ Apple │
| │ 2 ┆ 5 ┆ Orange │
| │ 3 ┆ 4 ┆ Apple │
| │ 4 ┆ 3 ┆ Apple │
| │ 5 ┆ 2 ┆ Banana │
| │ 6 ┆ 1 ┆ Banana │
| └─────┴─────┴────────┘
|
| Get the bottom 2 rows by column `a` or `b`.
|
| >>> df.select(
| ... pl.all().bottom_k_by("a", 2).name.suffix("_btm_by_a"),
| ... pl.all().bottom_k_by("b", 2).name.suffix("_btm_by_b"),
| ... )
| shape: (2, 6)
| ┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
| │ a_btm_by_a ┆ b_btm_by_a ┆ c_btm_by_a ┆ a_btm_by_b ┆ b_btm_by_b ┆ c_btm_by_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞════════════╪════════════╪════════════╪════════════╪════════════╪════════════╡
| │ 1 ┆ 6 ┆ Apple ┆ 6 ┆ 1 ┆ Banana │
| │ 2 ┆ 5 ┆ Orange ┆ 5 ┆ 2 ┆ Banana │
| └────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
|
| Get the bottom 2 rows by multiple columns with given order.
|
| >>> df.select(
| ... pl.all()
| ... .bottom_k_by(["c", "a"], 2, reverse=[False, True])
| ... .name.suffix("_by_ca"),
| ... pl.all()
| ... .bottom_k_by(["c", "b"], 2, reverse=[False, True])
| ... .name.suffix("_by_cb"),
| ... )
| shape: (2, 6)
| ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
| │ a_by_ca ┆ b_by_ca ┆ c_by_ca ┆ a_by_cb ┆ b_by_cb ┆ c_by_cb │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
| │ 4 ┆ 3 ┆ Apple ┆ 1 ┆ 6 ┆ Apple │
| │ 3 ┆ 4 ┆ Apple ┆ 3 ┆ 4 ┆ Apple │
| └─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
|
| Get the bottom 2 rows by column `a` in each group.
|
| >>> (
| ... df.group_by("c", maintain_order=True)
| ... .agg(pl.all().bottom_k_by("a", 2))
| ... .explode(pl.all().exclude("c"))
| ... )
| shape: (5, 3)
| ┌────────┬─────┬─────┐
| │ c ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ Apple ┆ 1 ┆ 6 │
| │ Apple ┆ 3 ┆ 4 │
| │ Orange ┆ 2 ┆ 5 │
| │ Banana ┆ 5 ┆ 2 │
| │ Banana ┆ 6 ┆ 1 │
| └────────┴─────┴─────┘
|
| cast(self, dtype: 'PolarsDataType | pl.DataTypeExpr | type[Any]', *, strict: 'bool' = True, wrap_numerical: 'bool' = False) -> 'Expr'
| Cast between data types.
|
| Parameters
| ----------
| dtype
| DataType to cast to.
| strict
| Raise if cast is invalid on rows after predicates are pushed down.
| If `False`, invalid casts will produce null values.
| wrap_numerical
| If True numeric casts wrap overflowing values instead of
| marking the cast as invalid.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["4", "5", "6"],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("a").cast(pl.Float64),
| ... pl.col("b").cast(pl.Int32),
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ i32 │
| ╞═════╪═════╡
| │ 1.0 ┆ 4 │
| │ 2.0 ┆ 5 │
| │ 3.0 ┆ 6 │
| └─────┴─────┘
|
| cbrt(self) -> 'Expr'
| Compute the cube root of the elements.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").cbrt())
| shape: (3, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 1.259921 │
| │ 1.587401 │
| └──────────┘
|
| ceil(self) -> 'Expr'
| Rounds up to the nearest integer value.
|
| Only works on floating point Series.
|
| See Also
| --------
| floor : Round down to the nearest integer.
| round : Round to the nearest integer.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.3, 0.5, 1.0, 1.1]})
| >>> df.select(pl.col("a").ceil())
| shape: (4, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| │ 1.0 │
| │ 1.0 │
| │ 2.0 │
| └─────┘
|
| clip(self, lower_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None, upper_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None) -> 'Expr'
| Set values outside the given boundaries to the boundary value.
|
| Parameters
| ----------
| lower_bound
| Lower bound. Accepts expression input. Non-expression inputs are
| parsed as literals. Strings are parsed as column names.
| upper_bound
| Upper bound. Accepts expression input. Non-expression inputs are
| parsed as literals. Strings are parsed as column names.
|
| See Also
| --------
| when
|
| Notes
| -----
| This method only works for numeric and temporal columns. To clip other data
| types, consider writing a `when-then-otherwise` expression. See :func:`when`.
|
| Examples
| --------
| Specifying both a lower and upper bound:
|
| >>> df = pl.DataFrame({"a": [-50, 5, 50, None]})
| >>> df.with_columns(clip=pl.col("a").clip(1, 10))
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ clip │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ -50 ┆ 1 │
| │ 5 ┆ 5 │
| │ 50 ┆ 10 │
| │ null ┆ null │
| └──────┴──────┘
|
| Specifying only a single bound:
|
| >>> df.with_columns(clip=pl.col("a").clip(upper_bound=10))
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ clip │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ -50 ┆ -50 │
| │ 5 ┆ 5 │
| │ 50 ┆ 10 │
| │ null ┆ null │
| └──────┴──────┘
|
| Using columns as bounds:
|
| >>> df = pl.DataFrame(
| ... {"a": [-50, 5, 50, None], "low": [10, 1, 0, 0], "up": [20, 4, 3, 2]}
| ... )
| >>> df.with_columns(clip=pl.col("a").clip("low", "up"))
| shape: (4, 4)
| ┌──────┬─────┬─────┬──────┐
| │ a ┆ low ┆ up ┆ clip │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞══════╪═════╪═════╪══════╡
| │ -50 ┆ 10 ┆ 20 ┆ 10 │
| │ 5 ┆ 1 ┆ 4 ┆ 4 │
| │ 50 ┆ 0 ┆ 3 ┆ 3 │
| │ null ┆ 0 ┆ 2 ┆ null │
| └──────┴─────┴─────┴──────┘
|
| cos(self) -> 'Expr'
| Compute the element-wise value for the cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 2]})
| >>> df.select(pl.col("a").cos())
| shape: (2, 1)
| ┌────────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════════╡
| │ 1.0 │
| │ 6.1232e-17 │
| └────────────┘
| >>> df = pl.DataFrame({"a": [0.0, 90]})
| >>> df.select(pl.col("a").radians().cos())
| shape: (2, 1)
| ┌────────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════════╡
| │ 1.0 │
| │ 6.1232e-17 │
| └────────────┘
|
| cosh(self) -> 'Expr'
| Compute the element-wise value for the hyperbolic cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").cosh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.543081 │
| └──────────┘
|
| cot(self) -> 'Expr'
| Compute the element-wise value for the cotangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 4]})
| >>> df.select(pl.col("a").cot())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ inf │
| │ 1.0 │
| └─────┘
| >>> df = pl.DataFrame({"a": [0.0, 45]})
| >>> df.select(pl.col("a").radians().cot())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ inf │
| │ 1.0 │
| └─────┘
|
| count(self) -> 'Expr'
| Return the number of non-null elements in the column.
|
| Returns
| -------
| Expr
| Expression of data type :class:`UInt32`.
|
| See Also
| --------
| len
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [None, 4, 4]})
| >>> df.select(pl.all().count())
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 3 ┆ 2 │
| └─────┴─────┘
|
| cum_count(self, *, reverse: 'bool' = False) -> 'Expr'
| Return the cumulative count of the non-null values in the column.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": ["x", "k", None, "d"]})
| >>> df.with_columns(
| ... pl.col("a").cum_count().alias("cum_count"),
| ... pl.col("a").cum_count(reverse=True).alias("cum_count_reverse"),
| ... )
| shape: (4, 3)
| ┌──────┬───────────┬───────────────────┐
| │ a ┆ cum_count ┆ cum_count_reverse │
| │ --- ┆ --- ┆ --- │
| │ str ┆ u32 ┆ u32 │
| ╞══════╪═══════════╪═══════════════════╡
| │ x ┆ 1 ┆ 3 │
| │ k ┆ 2 ┆ 2 │
| │ null ┆ 2 ┆ 1 │
| │ d ┆ 3 ┆ 1 │
| └──────┴───────────┴───────────────────┘
|
| cum_max(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative max computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 2]})
| >>> df.with_columns(
| ... pl.col("a").cum_max().alias("cum_max"),
| ... pl.col("a").cum_max(reverse=True).alias("cum_max_reverse"),
| ... )
| shape: (3, 3)
| ┌─────┬─────────┬─────────────────┐
| │ a ┆ cum_max ┆ cum_max_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════════╪═════════════════╡
| │ 1 ┆ 1 ┆ 3 │
| │ 3 ┆ 3 ┆ 3 │
| │ 2 ┆ 3 ┆ 2 │
| └─────┴─────────┴─────────────────┘
|
|
| Null values are excluded, but can also be filled by calling
| `fill_null(strategy="forward")`.
|
| >>> df = pl.DataFrame({"values": [None, 10, None, 8, 9, None, 16, None]})
| >>> df.with_columns(
| ... pl.col("values").cum_max().alias("cum_max"),
| ... pl.col("values")
| ... .cum_max()
| ... .fill_null(strategy="forward")
| ... .alias("cum_max_all_filled"),
| ... )
| shape: (8, 3)
| ┌────────┬─────────┬────────────────────┐
| │ values ┆ cum_max ┆ cum_max_all_filled │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞════════╪═════════╪════════════════════╡
| │ null ┆ null ┆ null │
| │ 10 ┆ 10 ┆ 10 │
| │ null ┆ null ┆ 10 │
| │ 8 ┆ 10 ┆ 10 │
| │ 9 ┆ 10 ┆ 10 │
| │ null ┆ null ┆ 10 │
| │ 16 ┆ 16 ┆ 16 │
| │ null ┆ null ┆ 16 │
| └────────┴─────────┴────────────────────┘
|
| cum_min(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative min computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [3, 1, 2]})
| >>> df.with_columns(
| ... pl.col("a").cum_min().alias("cum_min"),
| ... pl.col("a").cum_min(reverse=True).alias("cum_min_reverse"),
| ... )
| shape: (3, 3)
| ┌─────┬─────────┬─────────────────┐
| │ a ┆ cum_min ┆ cum_min_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════════╪═════════════════╡
| │ 3 ┆ 3 ┆ 1 │
| │ 1 ┆ 1 ┆ 1 │
| │ 2 ┆ 1 ┆ 2 │
| └─────┴─────────┴─────────────────┘
|
| cum_prod(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative product computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.with_columns(
| ... pl.col("a").cum_prod().alias("cum_prod"),
| ... pl.col("a").cum_prod(reverse=True).alias("cum_prod_reverse"),
| ... )
| shape: (4, 3)
| ┌─────┬──────────┬──────────────────┐
| │ a ┆ cum_prod ┆ cum_prod_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪══════════╪══════════════════╡
| │ 1 ┆ 1 ┆ 24 │
| │ 2 ┆ 2 ┆ 24 │
| │ 3 ┆ 6 ┆ 12 │
| │ 4 ┆ 24 ┆ 4 │
| └─────┴──────────┴──────────────────┘
|
| cum_sum(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative sum computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.with_columns(
| ... pl.col("a").cum_sum().alias("cum_sum"),
| ... pl.col("a").cum_sum(reverse=True).alias("cum_sum_reverse"),
| ... )
| shape: (4, 3)
| ┌─────┬─────────┬─────────────────┐
| │ a ┆ cum_sum ┆ cum_sum_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════════╪═════════════════╡
| │ 1 ┆ 1 ┆ 10 │
| │ 2 ┆ 3 ┆ 9 │
| │ 3 ┆ 6 ┆ 7 │
| │ 4 ┆ 10 ┆ 4 │
| └─────┴─────────┴─────────────────┘
|
| Null values are excluded, but can also be filled by calling
| `fill_null(strategy="forward")`.
|
| >>> df = pl.DataFrame({"values": [None, 10, None, 8, 9, None, 16, None]})
| >>> df.with_columns(
| ... pl.col("values").cum_sum().alias("value_cum_sum"),
| ... pl.col("values")
| ... .cum_sum()
| ... .fill_null(strategy="forward")
| ... .alias("value_cum_sum_all_filled"),
| ... )
| shape: (8, 3)
| ┌────────┬───────────────┬──────────────────────────┐
| │ values ┆ value_cum_sum ┆ value_cum_sum_all_filled │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞════════╪═══════════════╪══════════════════════════╡
| │ null ┆ null ┆ null │
| │ 10 ┆ 10 ┆ 10 │
| │ null ┆ null ┆ 10 │
| │ 8 ┆ 18 ┆ 18 │
| │ 9 ┆ 27 ┆ 27 │
| │ null ┆ null ┆ 27 │
| │ 16 ┆ 43 ┆ 43 │
| │ null ┆ null ┆ 43 │
| └────────┴───────────────┴──────────────────────────┘
|
| cumulative_eval(self, expr: 'Expr', *, min_samples: 'int' = 1) -> 'Expr'
| Run an expression over a sliding window that increases `1` slot every iteration.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| expr
| Expression to evaluate
| min_samples
| Number of valid values there should be in the window before the expression
| is evaluated. valid values = `length - null_count`
|
| Warnings
| --------
| This can be really slow as it can have `O(n^2)` complexity. Don't use this
| for operations that visit all elements.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1, 2, 3, 4, 5]})
| >>> df.select(
| ... [
| ... pl.col("values").cumulative_eval(
| ... pl.element().first() - pl.element().last() ** 2
| ... )
| ... ]
| ... )
| shape: (5, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 0 │
| │ -3 │
| │ -8 │
| │ -15 │
| │ -24 │
| └────────┘
|
| cut(self, breaks: 'Sequence[float]', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, include_breaks: 'bool' = False) -> 'Expr'
| Bin continuous values into discrete categories.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| breaks
| List of unique cut points.
| labels
| Names of the categories. The number of labels must be equal to the number
| of cut points plus one.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from an
| :class:`Enum` to a :class:`Struct`.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Enum` if `include_breaks` is set to
| `False` (default), otherwise an expression of data type :class:`Struct`.
|
| See Also
| --------
| qcut
|
| Examples
| --------
| Divide a column into three categories.
|
| >>> df = pl.DataFrame({"foo": [-2, -1, 0, 1, 2]})
| >>> df.with_columns(
| ... pl.col("foo").cut([-1, 1], labels=["a", "b", "c"]).alias("cut")
| ... )
| shape: (5, 2)
| ┌─────┬──────┐
| │ foo ┆ cut │
| │ --- ┆ --- │
| │ i64 ┆ enum │
| ╞═════╪══════╡
| │ -2 ┆ a │
| │ -1 ┆ a │
| │ 0 ┆ b │
| │ 1 ┆ b │
| │ 2 ┆ c │
| └─────┴──────┘
|
| Add both the category and the breakpoint.
|
| >>> df.with_columns(
| ... pl.col("foo").cut([-1, 1], include_breaks=True).alias("cut")
| ... ).unnest("cut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ enum │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| degrees(self) -> 'Expr'
| Convert from radians to degrees.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> import math
| >>> df = pl.DataFrame({"a": [x * math.pi for x in range(-4, 5)]})
| >>> df.select(pl.col("a").degrees())
| shape: (9, 1)
| ┌────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════╡
| │ -720.0 │
| │ -540.0 │
| │ -360.0 │
| │ -180.0 │
| │ 0.0 │
| │ 180.0 │
| │ 360.0 │
| │ 540.0 │
| │ 720.0 │
| └────────┘
|
| diff(self, n: 'int | IntoExpr' = 1, null_behavior: 'NullBehavior' = 'ignore') -> 'Expr'
| Calculate the first discrete difference between shifted items.
|
| Parameters
| ----------
| n
| Number of slots to shift.
| null_behavior : {'ignore', 'drop'}
| How to handle null values.
|
| Examples
| --------
| >>> df = pl.DataFrame({"int": [20, 10, 30, 25, 35]})
| >>> df.with_columns(change=pl.col("int").diff())
| shape: (5, 2)
| ┌─────┬────────┐
| │ int ┆ change │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪════════╡
| │ 20 ┆ null │
| │ 10 ┆ -10 │
| │ 30 ┆ 20 │
| │ 25 ┆ -5 │
| │ 35 ┆ 10 │
| └─────┴────────┘
|
| >>> df.with_columns(change=pl.col("int").diff(n=2))
| shape: (5, 2)
| ┌─────┬────────┐
| │ int ┆ change │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪════════╡
| │ 20 ┆ null │
| │ 10 ┆ null │
| │ 30 ┆ 10 │
| │ 25 ┆ 15 │
| │ 35 ┆ 5 │
| └─────┴────────┘
|
| >>> df.select(pl.col("int").diff(n=2, null_behavior="drop").alias("diff"))
| shape: (3, 1)
| ┌──────┐
| │ diff │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 10 │
| │ 15 │
| │ 5 │
| └──────┘
|
| dot(self, other: 'Expr | str_') -> 'Expr'
| Compute the dot/inner product between two Expressions.
|
| Parameters
| ----------
| other
| Expression to compute dot product with.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 5], "b": [2, 4, 6]})
| >>> df.select(pl.col("a").dot(pl.col("b")))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 44 │
| └─────┘
|
| drop_nans(self) -> 'Expr'
| Drop all floating point NaN values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nulls
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To drop null values, use :func:`drop_nulls`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, None, 3.0, float("nan")]})
| >>> df.select(pl.col("a").drop_nans())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 1.0 │
| │ null │
| │ 3.0 │
| └──────┘
|
| drop_nulls(self) -> 'Expr'
| Drop all null values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nans
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To drop NaN values, use :func:`drop_nans`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, None, 3.0, float("nan")]})
| >>> df.select(pl.col("a").drop_nulls())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| │ 3.0 │
| │ NaN │
| └─────┘
|
| entropy(self, base: 'float' = 2.718281828459045, *, normalize: 'bool' = True) -> 'Expr'
| Computes the entropy.
|
| Uses the formula `-sum(pk * log(pk))` where `pk` are discrete probabilities.
|
| Parameters
| ----------
| base
| Given base, defaults to `e`
| normalize
| Normalize pk if it doesn't sum to 1.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").entropy(base=2))
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.459148 │
| └──────────┘
| >>> df.select(pl.col("a").entropy(base=2, normalize=False))
| shape: (1, 1)
| ┌───────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═══════════╡
| │ -6.754888 │
| └───────────┘
|
| eq(self, other: 'Any') -> 'Expr'
| Method equivalent of equality operator `expr == other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0],
| ... "y": [2.0, 2.0, float("nan"), 4.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").eq(pl.col("y")).alias("x == y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x == y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 1.0 ┆ 2.0 ┆ false │
| │ 2.0 ┆ 2.0 ┆ true │
| │ NaN ┆ NaN ┆ true │
| │ 4.0 ┆ 4.0 ┆ true │
| └─────┴─────┴────────┘
|
| eq_missing(self, other: 'Any') -> 'Expr'
| Method equivalent of equality operator `expr == other` where `None == None`.
|
| This differs from default `eq` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0, None, None],
| ... "y": [2.0, 2.0, float("nan"), 4.0, 5.0, None],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").eq(pl.col("y")).alias("x eq y"),
| ... pl.col("x").eq_missing(pl.col("y")).alias("x eq_missing y"),
| ... )
| shape: (6, 4)
| ┌──────┬──────┬────────┬────────────────┐
| │ x ┆ y ┆ x eq y ┆ x eq_missing y │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪══════╪════════╪════════════════╡
| │ 1.0 ┆ 2.0 ┆ false ┆ false │
| │ 2.0 ┆ 2.0 ┆ true ┆ true │
| │ NaN ┆ NaN ┆ true ┆ true │
| │ 4.0 ┆ 4.0 ┆ true ┆ true │
| │ null ┆ 5.0 ┆ null ┆ false │
| │ null ┆ null ┆ null ┆ true │
| └──────┴──────┴────────┴────────────────┘
|
| ewm_mean(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving average.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_mean(com=1, ignore_nulls=False))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 1.666667 │
| │ 2.428571 │
| └──────────┘
|
| ewm_mean_by(self, by: 'str_ | IntoExpr', *, half_life: 'str_ | timedelta') -> 'Expr'
| Compute time-based exponentially weighted moving average.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMA is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \alpha_i &= 1 - \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= \alpha_i x_i + (1 - \alpha_i) y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Times to calculate average by. Should be ``DateTime``, ``Date``, ``UInt64``,
| ``UInt32``, ``Int64``, or ``Int32`` data type.
| half_life
| Unit over which observation decays to half its value.
|
| Can be created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 day)
| - 1w (1 week)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| Note that `half_life` is treated as a constant duration - calendar
| durations such as months (or even days in the time-zone-aware case)
| are not supported, please express your duration in an approximately
| equivalent number of hours (e.g. '370h' instead of '1mo').
|
| Returns
| -------
| Expr
| :class:`.Float16` if input is `Float16`, class:`.Float32` if input is
| `Float32`, otherwise class:`.Float64`.
|
| Examples
| --------
| >>> from datetime import date, timedelta
| >>> df = pl.DataFrame(
| ... {
| ... "values": [0, 1, 2, None, 4],
| ... "times": [
| ... date(2020, 1, 1),
| ... date(2020, 1, 3),
| ... date(2020, 1, 10),
| ... date(2020, 1, 15),
| ... date(2020, 1, 17),
| ... ],
| ... }
| ... ).sort("times")
| >>> df.with_columns(
| ... result=pl.col("values").ewm_mean_by("times", half_life="4d"),
| ... )
| shape: (5, 3)
| ┌────────┬────────────┬──────────┐
| │ values ┆ times ┆ result │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ date ┆ f64 │
| ╞════════╪════════════╪══════════╡
| │ 0 ┆ 2020-01-01 ┆ 0.0 │
| │ 1 ┆ 2020-01-03 ┆ 0.292893 │
| │ 2 ┆ 2020-01-10 ┆ 1.492474 │
| │ null ┆ 2020-01-15 ┆ null │
| │ 4 ┆ 2020-01-17 ┆ 3.254508 │
| └────────┴────────────┴──────────┘
|
| ewm_std(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving standard deviation.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_std(com=1, ignore_nulls=False))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ null │
| │ 0.707107 │
| │ 0.963624 │
| └──────────┘
|
| ewm_sum(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_sum(alpha=0.5))
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 1.0 │
| │ 2.5 │
| │ 4.25 │
| └──────┘
|
| ewm_sum_by(self, by: 'str_ | IntoExpr', *, half_life: 'str_ | timedelta') -> 'Expr'
| Compute time-based exponentially-weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMS is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \lambda_i &= \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= x_i + \lambda_i y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Column to use as reference for the time decay.
| half_life
| Half-life of the exponential decay.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "values": [1, 2, 3, 4, 5],
| ... "times": [0, 1, 2, 5, 6],
| ... }
| ... )
| >>> df.select(
| ... pl.col("values").ewm_sum_by("times", half_life="1i"),
| ... )
| shape: (5, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 2.5 │
| │ 4.25 │
| │ 4.53125 │
| │ 7.265625 │
| └──────────┘
|
| ewm_var(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving variance.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_var(com=1, ignore_nulls=False))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ null │
| │ 0.5 │
| │ 0.928571 │
| └──────────┘
|
| exclude(self, columns: 'str_ | PolarsDataType | Collection[str_ | PolarsDataType]', *more_columns: 'str_ | PolarsDataType') -> 'Expr'
| Exclude columns from a multi-column expression.
|
| Only works after a wildcard or regex column selection, and you cannot provide
| both string column names *and* dtypes (you may prefer to use selectors instead).
|
| Parameters
| ----------
| columns
| The name or datatype of the column(s) to exclude. Accepts regular expression
| input. Regular expressions should start with `^` and end with `$`.
| *more_columns
| Additional names or datatypes of columns to exclude, specified as positional
| arguments.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "aa": [1, 2, 3],
| ... "ba": ["a", "b", None],
| ... "cc": [None, 2.5, 1.5],
| ... }
| ... )
| >>> df
| shape: (3, 3)
| ┌─────┬──────┬──────┐
| │ aa ┆ ba ┆ cc │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ f64 │
| ╞═════╪══════╪══════╡
| │ 1 ┆ a ┆ null │
| │ 2 ┆ b ┆ 2.5 │
| │ 3 ┆ null ┆ 1.5 │
| └─────┴──────┴──────┘
|
| Exclude by column name(s):
|
| >>> df.select(pl.all().exclude("ba"))
| shape: (3, 2)
| ┌─────┬──────┐
| │ aa ┆ cc │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ null │
| │ 2 ┆ 2.5 │
| │ 3 ┆ 1.5 │
| └─────┴──────┘
|
| Exclude by regex, e.g. removing all columns whose names end with the letter "a":
|
| >>> df.select(pl.all().exclude("^.*a$"))
| shape: (3, 1)
| ┌──────┐
| │ cc │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ 2.5 │
| │ 1.5 │
| └──────┘
|
| Exclude by dtype(s), e.g. removing all columns of type Int64 or Float64:
|
| >>> df.select(pl.all().exclude([pl.Int64, pl.Float64]))
| shape: (3, 1)
| ┌──────┐
| │ ba │
| │ --- │
| │ str │
| ╞══════╡
| │ a │
| │ b │
| │ null │
| └──────┘
|
| exp(self) -> 'Expr'
| Compute the exponential, element-wise.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").exp())
| shape: (3, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 2.718282 │
| │ 7.389056 │
| │ 54.59815 │
| └──────────┘
|
| explode(self, *, empty_as_null: 'bool' = <object object at 0x00000280D5528F20>, keep_nulls: 'bool' = True) -> 'Expr'
| Explode a list expression.
|
| This means that every item is expanded to a new row.
|
| Parameters
| ----------
| empty_as_null
| Explode an empty list/array into a `null`.
| keep_nulls
| Explode a `null` list/array into a `null`.
|
| Returns
| -------
| Expr
| Expression with the data type of the list elements.
|
| See Also
| --------
| Expr.list.explode : Explode a list column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["a", "b"],
| ... "values": [
| ... [1, 2],
| ... [3, 4],
| ... ],
| ... }
| ... )
| >>> df.select(pl.col("values").explode(empty_as_null=False))
| shape: (4, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 1 │
| │ 2 │
| │ 3 │
| │ 4 │
| └────────┘
|
| extend_constant(self, value: 'IntoExpr', n: 'int | IntoExprColumn') -> 'Expr'
| Extremely fast method for extending the Series with 'n' copies of a value.
|
| Parameters
| ----------
| value
| A constant literal value or a unit expression with which to extend the
| expression result Series; can pass None to extend with nulls.
| n
| The number of additional values that will be added.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1, 2, 3]})
| >>> df.select((pl.col("values") - 1).extend_constant(99, n=2))
| shape: (5, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 0 │
| │ 1 │
| │ 2 │
| │ 99 │
| │ 99 │
| └────────┘
|
| fill_nan(self, value: 'int | float | Expr | None') -> 'Expr'
| Fill floating point NaN value with a fill value.
|
| Parameters
| ----------
| value
| Value used to fill NaN values.
|
| See Also
| --------
| fill_null
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To fill null values, use :func:`fill_null`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1.0, None, float("nan")],
| ... "b": [4.0, float("nan"), 6],
| ... }
| ... )
| >>> df.with_columns(pl.col("b").fill_nan(0))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪═════╡
| │ 1.0 ┆ 4.0 │
| │ null ┆ 0.0 │
| │ NaN ┆ 6.0 │
| └──────┴─────┘
|
| fill_null(self, value: 'Any | Expr | None' = None, strategy: 'FillNullStrategy | None' = None, limit: 'int | None' = None) -> 'Expr'
| Fill null values using the specified value or strategy.
|
| To interpolate over null values see interpolate.
| See the examples below to fill nulls with an expression.
|
| Parameters
| ----------
| value
| Value used to fill null values.
| strategy : {None, 'forward', 'backward', 'min', 'max', 'mean', 'zero', 'one'}
| Strategy used to fill null values.
| limit
| Number of consecutive null values to fill when using the 'forward' or
| 'backward' strategy.
|
| See Also
| --------
| backward_fill
| fill_nan
| forward_fill
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To fill NaN values, use :func:`fill_nan`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": [4, None, 6],
| ... }
| ... )
| >>> df.with_columns(pl.col("b").fill_null(strategy="zero"))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 0 │
| │ null ┆ 6 │
| └──────┴─────┘
| >>> df.with_columns(pl.col("b").fill_null(99))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 99 │
| │ null ┆ 6 │
| └──────┴─────┘
| >>> df.with_columns(pl.col("b").fill_null(strategy="forward"))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 4 │
| │ null ┆ 6 │
| └──────┴─────┘
| >>> df.with_columns(pl.col("b").fill_null(pl.col("b").median()))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞══════╪═════╡
| │ 1 ┆ 4.0 │
| │ 2 ┆ 5.0 │
| │ null ┆ 6.0 │
| └──────┴─────┘
| >>> df.with_columns(pl.all().fill_null(pl.all().median()))
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 1.0 ┆ 4.0 │
| │ 2.0 ┆ 5.0 │
| │ 1.5 ┆ 6.0 │
| └─────┴─────┘
|
| filter(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn]', **constraints: 'Any') -> 'Expr'
| Filter the expression based on one or more predicate expressions.
|
| The original order of the remaining elements is preserved.
|
| Elements where the filter does not evaluate to True are discarded, including
| nulls.
|
| Mostly useful in an aggregation context. If you want to filter on a DataFrame
| level, use `LazyFrame.filter`.
|
| Parameters
| ----------
| predicates
| Expression(s) that evaluates to a boolean Series.
| constraints
| Column filters; use `name = value` to filter columns by the supplied value.
| Each constraint will behave the same as `pl.col(name).eq(value)`, and
| be implicitly joined with the other filter conditions using `&`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group_col": ["g1", "g1", "g2"],
| ... "b": [1, 2, 3],
| ... }
| ... )
| >>> df.group_by("group_col").agg(
| ... lt=pl.col("b").filter(pl.col("b") < 2).sum(),
| ... gte=pl.col("b").filter(pl.col("b") >= 2).sum(),
| ... ).sort("group_col")
| shape: (2, 3)
| ┌───────────┬─────┬─────┐
| │ group_col ┆ lt ┆ gte │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════════╪═════╪═════╡
| │ g1 ┆ 1 ┆ 2 │
| │ g2 ┆ 0 ┆ 3 │
| └───────────┴─────┴─────┘
|
| Filter expressions can also take constraints as keyword arguments.
|
| >>> df = pl.DataFrame(
| ... {
| ... "key": ["a", "a", "a", "a", "b", "b", "b", "b", "b"],
| ... "n": [1, 2, 2, 3, 1, 3, 3, 2, 3],
| ... },
| ... )
| >>> df.group_by("key").agg(
| ... n_1=pl.col("n").filter(n=1).sum(),
| ... n_2=pl.col("n").filter(n=2).sum(),
| ... n_3=pl.col("n").filter(n=3).sum(),
| ... ).sort(by="key")
| shape: (2, 4)
| ┌─────┬─────┬─────┬─────┐
| │ key ┆ n_1 ┆ n_2 ┆ n_3 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╡
| │ a ┆ 1 ┆ 4 ┆ 3 │
| │ b ┆ 1 ┆ 2 ┆ 9 │
| └─────┴─────┴─────┴─────┘
|
| first(self, *, ignore_nulls: 'bool' = False) -> 'Expr'
| Get the first value.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the first non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [None, 1, 2]})
| >>> df.select(pl.col("a").first())
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| └──────┘
| >>> df.select(pl.col("a").first(ignore_nulls=True))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| └─────┘
|
| flatten(self) -> 'Expr'
| Flatten a list or string column.
|
| Alias for :func:`Expr.list.explode`.
|
| .. deprecated:: 1.38
| `Expr.flatten()` is deprecated and will be removed in version 2.0.
| Use `Expr.list.explode(keep_nulls=False, empty_as_null=False)` instead,
| which provides the behavior you likely expect.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["a", "b", "b"],
| ... "values": [[1, 2], [2, 3], [4]],
| ... }
| ... )
| >>> df.group_by("group").agg(pl.col("values").flatten()) # doctest: +SKIP
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ values │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪═══════════╡
| │ a ┆ [1, 2] │
| │ b ┆ [2, 3, 4] │
| └───────┴───────────┘
|
| floor(self) -> 'Expr'
| Rounds down to the nearest integer value.
|
| Only works on floating point Series.
|
| See Also
| --------
| ceil : Round up to the nearest integer value.
| round : Round to the nearest integer.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.3, 0.5, 1.0, 1.1]})
| >>> df.select(pl.col("a").floor())
| shape: (4, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 0.0 │
| │ 1.0 │
| │ 1.0 │
| └─────┘
|
| floordiv(self, other: 'Any') -> 'Expr'
| Method equivalent of integer division operator `expr // other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| See Also
| --------
| truediv
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 3, 4, 5]})
| >>> df.with_columns(
| ... pl.col("x").truediv(2).alias("x/2"),
| ... pl.col("x").floordiv(2).alias("x//2"),
| ... )
| shape: (5, 3)
| ┌─────┬─────┬──────┐
| │ x ┆ x/2 ┆ x//2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ i64 │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 0.5 ┆ 0 │
| │ 2 ┆ 1.0 ┆ 1 │
| │ 3 ┆ 1.5 ┆ 1 │
| │ 4 ┆ 2.0 ┆ 2 │
| │ 5 ┆ 2.5 ┆ 2 │
| └─────┴─────┴──────┘
|
| Note that Polars' `floordiv` is subtly different from Python's floor division.
| For example, consider 6.0 floor-divided by 0.1.
| Python gives:
|
| >>> 6.0 // 0.1
| 59.0
|
| because `0.1` is not represented internally as that exact value,
| but a slightly larger value.
| So the result of the division is slightly less than 60,
| meaning the flooring operation returns 59.0.
|
| Polars instead first does the floating-point division,
| resulting in a floating-point value of 60.0,
| and then performs the flooring operation using :any:`floor`:
|
| >>> df = pl.DataFrame({"x": [6.0, 6.03]})
| >>> df.with_columns(
| ... pl.col("x").truediv(0.1).alias("x/0.1"),
| ... ).with_columns(
| ... pl.col("x/0.1").floor().alias("x/0.1 floor"),
| ... )
| shape: (2, 3)
| ┌──────┬───────┬─────────────┐
| │ x ┆ x/0.1 ┆ x/0.1 floor │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞══════╪═══════╪═════════════╡
| │ 6.0 ┆ 60.0 ┆ 60.0 │
| │ 6.03 ┆ 60.3 ┆ 60.0 │
| └──────┴───────┴─────────────┘
|
| yielding the more intuitive result 60.0.
| The row with x = 6.03 is included to demonstrate
| the effect of the flooring operation.
|
| `floordiv` combines those two steps
| to give the same result with one expression:
|
| >>> df.with_columns(
| ... pl.col("x").floordiv(0.1).alias("x//0.1"),
| ... )
| shape: (2, 2)
| ┌──────┬────────┐
| │ x ┆ x//0.1 │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪════════╡
| │ 6.0 ┆ 60.0 │
| │ 6.03 ┆ 60.0 │
| └──────┴────────┘
|
| forward_fill(self, limit: 'int | None' = None) -> 'Expr'
| Fill missing values with the last non-null value.
|
| This is an alias of `.fill_null(strategy="forward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to forward fill.
|
| See Also
| --------
| backward_fill
| fill_null
| shift
|
| gather(self, indices: 'int | Sequence[int] | IntoExpr | Series | np.ndarray[Any, Any]', *, null_on_oob: 'bool' = False) -> 'Expr'
| Take values by index.
|
| Parameters
| ----------
| indices
| An expression that leads to a UInt32 dtyped Series.
| null_on_oob
| Behavior if an index is out of bounds:
|
| - True -> set the result to null
| - False -> raise an error
|
| Returns
| -------
| Expr
| Expression of the same data type.
|
| See Also
| --------
| Expr.get : Take a single value
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": [
| ... "one",
| ... "one",
| ... "one",
| ... "two",
| ... "two",
| ... "two",
| ... ],
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.group_by("group", maintain_order=True).agg(
| ... pl.col("value").gather([2, 1])
| ... )
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪═══════════╡
| │ one ┆ [2, 98] │
| │ two ┆ [4, 99] │
| └───────┴───────────┘
|
| Use `null_on_oob=True` to return null for out-of-bounds indices.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").gather([0, 1, 10], null_on_oob=True))
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 1 │
| │ 2 │
| │ null │
| └──────┘
|
| gather_every(self, n: 'int', offset: 'int' = 0) -> 'Expr'
| Take every nth value in the Series and return as a new Series.
|
| Parameters
| ----------
| n
| Gather every *n*-th row.
| offset
| Starting index.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7, 8, 9]})
| >>> df.select(pl.col("foo").gather_every(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 4 │
| │ 7 │
| └─────┘
|
| >>> df.select(pl.col("foo").gather_every(3, offset=1))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 5 │
| │ 8 │
| └─────┘
|
| ge(self, other: 'Any') -> 'Expr'
| Method equivalent of "greater than or equal" operator `expr >= other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5.0, 4.0, float("nan"), 2.0],
| ... "y": [5.0, 3.0, float("nan"), 1.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").ge(pl.col("y")).alias("x >= y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x >= y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 5.0 ┆ 5.0 ┆ true │
| │ 4.0 ┆ 3.0 ┆ true │
| │ NaN ┆ NaN ┆ true │
| │ 2.0 ┆ 1.0 ┆ true │
| └─────┴─────┴────────┘
|
| get(self, index: 'int | Expr', *, null_on_oob: 'bool' = False) -> 'Expr'
| Return a single value by index.
|
| Parameters
| ----------
| index
| An expression that evaluates to an integer.
| Negative indexing is supported.
|
| null_on_oob
| Behavior if an index is out of bounds:
|
| - True -> set the result to null
| - False -> raise an error
|
| Returns
| -------
| Expr
| Expression of the same data type.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": [
| ... "one",
| ... "one",
| ... "one",
| ... "two",
| ... "two",
| ... "two",
| ... ],
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.group_by("group", maintain_order=True).agg(pl.col("value").get(1))
| shape: (2, 2)
| ┌───────┬───────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═══════╡
| │ one ┆ 98 │
| │ two ┆ 99 │
| └───────┴───────┘
|
| gt(self, other: 'Any') -> 'Expr'
| Method equivalent of "greater than" operator `expr > other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5.0, 4.0, float("nan"), 2.0],
| ... "y": [5.0, 3.0, float("nan"), 1.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").gt(pl.col("y")).alias("x > y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────┐
| │ x ┆ y ┆ x > y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪═══════╡
| │ 5.0 ┆ 5.0 ┆ false │
| │ 4.0 ┆ 3.0 ┆ true │
| │ NaN ┆ NaN ┆ false │
| │ 2.0 ┆ 1.0 ┆ true │
| └─────┴─────┴───────┘
|
| has_nulls(self) -> 'Expr'
| Check whether the expression contains one or more null values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 1, None],
| ... "b": [10, None, 300],
| ... "c": [350, 650, 850],
| ... }
| ... )
| >>> df.select(pl.all().has_nulls())
| shape: (1, 3)
| ┌──────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪══════╪═══════╡
| │ true ┆ true ┆ false │
| └──────┴──────┴───────┘
|
| hash(self, seed: 'int' = 0, seed_1: 'int | None' = None, seed_2: 'int | None' = None, seed_3: 'int | None' = None) -> 'Expr'
| Hash the elements in the selection.
|
| The hash value is of type `UInt64`.
|
| Parameters
| ----------
| seed
| Random seed parameter. Defaults to 0.
| seed_1
| Random seed parameter. Defaults to `seed` if not set.
| seed_2
| Random seed parameter. Defaults to `seed` if not set.
| seed_3
| Random seed parameter. Defaults to `seed` if not set.
|
| Notes
| -----
| This implementation of `hash` does not guarantee stable results
| across different Polars versions. Its stability is only guaranteed within a
| single version.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": ["x", None, "z"],
| ... }
| ... )
| >>> df.with_columns(pl.all().hash(10, 20, 30, 40)) # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌──────────────────────┬──────────────────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u64 ┆ u64 │
| ╞══════════════════════╪══════════════════════╡
| │ 9774092659964970114 ┆ 13614470193936745724 │
| │ 1101441246220388612 ┆ 11638928888656214026 │
| │ 11638928888656214026 ┆ 13382926553367784577 │
| └──────────────────────┴──────────────────────┘
|
| head(self, n: 'int | Expr' = 10) -> 'Expr'
| Get the first `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7]})
| >>> df.select(pl.col("foo").head(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| hist(self, bins: 'IntoExpr | None' = None, *, bin_count: 'int | None' = None, include_category: 'bool' = False, include_breakpoint: 'bool' = False) -> 'Expr'
| Bin values into buckets and count their occurrences.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| bins
| Bin edges. If None given, we determine the edges based on the data.
| bin_count
| If `bins` is not provided, `bin_count` uniform bins are created that fully
| encompass the data.
| include_breakpoint
| Include a column that indicates the upper breakpoint.
| include_category
| Include a column that shows the intervals as categories.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 8, 8, 2, 1, 3]})
| >>> df.select(pl.col("a").hist(bins=[1, 2, 3]))
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 3 │
| │ 2 │
| └─────┘
| >>> df.select(
| ... pl.col("a").hist(
| ... bins=[1, 2, 3], include_breakpoint=True, include_category=True
| ... )
| ... )
| shape: (2, 1)
| ┌──────────────────────┐
| │ a │
| │ --- │
| │ struct[3] │
| ╞══════════════════════╡
| │ {2.0,"[1.0, 2.0]",3} │
| │ {3.0,"(2.0, 3.0]",2} │
| └──────────────────────┘
|
| implode(self, *, maintain_order: 'bool' = True) -> 'Expr'
| Aggregate values into a list.
|
| The returned list itself is a scalar value of `list` dtype.
|
| Parameters
| ----------
| maintain_order
| Whether to preserve the order of elements in the list. Setting this
| to `False` can improve performance, especially within `group_by`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": [4, 5, 6],
| ... }
| ... )
| >>> df.select(pl.all().implode())
| shape: (1, 2)
| ┌───────────┬───────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ list[i64] ┆ list[i64] │
| ╞═══════════╪═══════════╡
| │ [1, 2, 3] ┆ [4, 5, 6] │
| └───────────┴───────────┘
|
| index_of(self, element: 'IntoExpr') -> 'Expr'
| Get the index of the first occurrence of a value, or ``None`` if it's not found.
|
| Parameters
| ----------
| element
| Value to find.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, None, 17]})
| >>> df.select(
| ... [
| ... pl.col("a").index_of(17).alias("seventeen"),
| ... pl.col("a").index_of(None).alias("null"),
| ... pl.col("a").index_of(55).alias("fiftyfive"),
| ... ]
| ... )
| shape: (1, 3)
| ┌───────────┬──────┬───────────┐
| │ seventeen ┆ null ┆ fiftyfive │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═══════════╪══════╪═══════════╡
| │ 2 ┆ 1 ┆ null │
| └───────────┴──────┴───────────┘
|
| inspect(self, fmt: 'str_' = '{}') -> 'Expr'
| Print the value that this expression evaluates to and pass on the value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 1, 2]})
| >>> df.select(pl.col("foo").cum_sum().inspect("value is: {}").alias("bar"))
| value is: shape: (3,)
| Series: 'foo' [i64]
| [
| 1
| 2
| 4
| ]
| shape: (3, 1)
| ┌─────┐
| │ bar │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 4 │
| └─────┘
|
| interpolate(self, method: 'InterpolationMethod' = 'linear') -> 'Expr'
| Interpolate intermediate values.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| method : {'linear', 'nearest'}
| Interpolation method.
|
| Examples
| --------
| Fill null values using linear interpolation.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, None, 3],
| ... "b": [1.0, float("nan"), 3.0],
| ... }
| ... )
| >>> df.select(pl.all().interpolate())
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 1.0 ┆ 1.0 │
| │ 2.0 ┆ NaN │
| │ 3.0 ┆ 3.0 │
| └─────┴─────┘
|
| Fill null values using nearest interpolation.
|
| >>> df.select(pl.all().interpolate("nearest"))
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 1.0 │
| │ 3 ┆ NaN │
| │ 3 ┆ 3.0 │
| └─────┴─────┘
|
| Regrid data to a new grid.
|
| >>> df_original_grid = pl.DataFrame(
| ... {
| ... "grid_points": [1, 3, 10],
| ... "values": [2.0, 6.0, 20.0],
| ... }
| ... ) # Interpolate from this to the new grid
| >>> df_new_grid = pl.DataFrame({"grid_points": range(1, 11)})
| >>> df_new_grid.join(
| ... df_original_grid, on="grid_points", how="left", coalesce=True
| ... ).with_columns(pl.col("values").interpolate())
| shape: (10, 2)
| ┌─────────────┬────────┐
| │ grid_points ┆ values │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════════════╪════════╡
| │ 1 ┆ 2.0 │
| │ 2 ┆ 4.0 │
| │ 3 ┆ 6.0 │
| │ 4 ┆ 8.0 │
| │ 5 ┆ 10.0 │
| │ 6 ┆ 12.0 │
| │ 7 ┆ 14.0 │
| │ 8 ┆ 16.0 │
| │ 9 ┆ 18.0 │
| │ 10 ┆ 20.0 │
| └─────────────┴────────┘
|
| interpolate_by(self, by: 'IntoExpr') -> 'Expr'
| Fill null values using interpolation based on another column.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| by
| Column to interpolate values based on.
|
| Examples
| --------
| Fill null values using linear interpolation.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, None, None, 3],
| ... "b": [1, 2, 7, 8],
| ... }
| ... )
| >>> df.with_columns(a_interpolated=pl.col("a").interpolate_by("b"))
| shape: (4, 3)
| ┌──────┬─────┬────────────────┐
| │ a ┆ b ┆ a_interpolated │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞══════╪═════╪════════════════╡
| │ 1 ┆ 1 ┆ 1.0 │
| │ null ┆ 2 ┆ 1.285714 │
| │ null ┆ 7 ┆ 2.714286 │
| │ 3 ┆ 8 ┆ 3.0 │
| └──────┴─────┴────────────────┘
|
| is_between(self, lower_bound: 'IntoExpr', upper_bound: 'IntoExpr', closed: 'ClosedInterval' = 'both') -> 'Expr'
| Check if this expression is between the given lower and upper bounds.
|
| Parameters
| ----------
| lower_bound
| Lower bound value. Accepts expression input. Strings are parsed as column
| names, other non-expression inputs are parsed as literals.
| upper_bound
| Upper bound value. Accepts expression input. Strings are parsed as column
| names, other non-expression inputs are parsed as literals.
| closed : {'both', 'left', 'right', 'none'}
| Define which sides of the interval are closed (inclusive).
|
| Notes
| -----
| If the value of the `lower_bound` is greater than that of the `upper_bound`
| then the result will be False, as no value can satisfy the condition.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"num": [1, 2, 3, 4, 5]})
| >>> df.with_columns(pl.col("num").is_between(2, 4).alias("is_between"))
| shape: (5, 2)
| ┌─────┬────────────┐
| │ num ┆ is_between │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪════════════╡
| │ 1 ┆ false │
| │ 2 ┆ true │
| │ 3 ┆ true │
| │ 4 ┆ true │
| │ 5 ┆ false │
| └─────┴────────────┘
|
| Use the `closed` argument to include or exclude the values at the bounds:
|
| >>> df.with_columns(
| ... pl.col("num").is_between(2, 4, closed="left").alias("is_between")
| ... )
| shape: (5, 2)
| ┌─────┬────────────┐
| │ num ┆ is_between │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪════════════╡
| │ 1 ┆ false │
| │ 2 ┆ true │
| │ 3 ┆ true │
| │ 4 ┆ false │
| │ 5 ┆ false │
| └─────┴────────────┘
|
| You can also use strings as well as numeric/temporal values (note: ensure that
| string literals are wrapped with `lit` so as not to conflate them with
| column names):
|
| >>> df = pl.DataFrame({"a": ["a", "b", "c", "d", "e"]})
| >>> df.with_columns(
| ... pl.col("a")
| ... .is_between(pl.lit("a"), pl.lit("c"), closed="both")
| ... .alias("is_between")
| ... )
| shape: (5, 2)
| ┌─────┬────────────┐
| │ a ┆ is_between │
| │ --- ┆ --- │
| │ str ┆ bool │
| ╞═════╪════════════╡
| │ a ┆ true │
| │ b ┆ true │
| │ c ┆ true │
| │ d ┆ false │
| │ e ┆ false │
| └─────┴────────────┘
|
| Use column expressions as lower/upper bounds, comparing to a literal value:
|
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5], "b": [5, 4, 3, 2, 1]})
| >>> df.with_columns(
| ... pl.lit(3).is_between(pl.col("a"), pl.col("b")).alias("between_ab")
| ... )
| shape: (5, 3)
| ┌─────┬─────┬────────────┐
| │ a ┆ b ┆ between_ab │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ bool │
| ╞═════╪═════╪════════════╡
| │ 1 ┆ 5 ┆ true │
| │ 2 ┆ 4 ┆ true │
| │ 3 ┆ 3 ┆ true │
| │ 4 ┆ 2 ┆ false │
| │ 5 ┆ 1 ┆ false │
| └─────┴─────┴────────────┘
|
| is_close(self, other: 'IntoExpr', *, abs_tol: 'float' = 0.0, rel_tol: 'float' = 1e-09, nans_equal: 'bool' = False) -> 'Expr'
| Check if this expression is close, i.e. almost equal, to the other expression.
|
| Two values `a` and `b` are considered close if the following condition holds:
|
| .. math::
| |a-b| \le max \{ \text{rel_tol} \cdot max \{ |a|, |b| \}, \text{abs_tol} \}
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
| abs_tol
| Absolute tolerance. This is the maximum allowed absolute difference between
| two values. Must be non-negative.
| rel_tol
| Relative tolerance. This is the maximum allowed difference between two
| values, relative to the larger absolute value. Must be non-negative.
| nans_equal
| Whether NaN values should be considered equal.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Notes
| -----
| The implementation of this method is symmetric and mirrors the behavior of
| :meth:`math.isclose`. Specifically note that this behavior is different to
| :meth:`numpy.isclose`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.5, 2.0, 2.5], "b": [1.55, 2.2, 3.0]})
| >>> df.with_columns(pl.col("a").is_close("b", abs_tol=0.1).alias("is_close"))
| shape: (3, 3)
| ┌─────┬──────┬──────────┐
| │ a ┆ b ┆ is_close │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪══════╪══════════╡
| │ 1.5 ┆ 1.55 ┆ true │
| │ 2.0 ┆ 2.2 ┆ false │
| │ 2.5 ┆ 3.0 ┆ false │
| └─────┴──────┴──────────┘
|
| is_duplicated(self) -> 'Expr'
| Return a boolean mask indicating duplicated values.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
| >>> df.select(pl.col("a").is_duplicated())
| shape: (3, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ true │
| │ true │
| │ false │
| └───────┘
|
| is_empty(self, *, ignore_nulls: 'bool' = False) -> 'Expr'
| Return whether the column is empty.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| ignore_nulls
| If true a column containing only nulls will also be considered empty.
| The default is false.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [None, None]})
| >>> df.select(
| ... a=pl.col.x.is_empty(),
| ... b=pl.col.x.drop_nulls().is_empty(),
| ... c=pl.col.x.is_empty(ignore_nulls=True),
| ... )
| shape: (1, 3)
| ┌───────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞═══════╪══════╪══════╡
| │ false ┆ true ┆ true │
| └───────┴──────┴──────┘
|
| is_finite(self) -> 'Expr'
| Returns a boolean Series indicating which values are finite.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1.0, 2],
| ... "B": [3.0, float("inf")],
| ... }
| ... )
| >>> df.select(pl.all().is_finite())
| shape: (2, 2)
| ┌──────┬───────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ bool ┆ bool │
| ╞══════╪═══════╡
| │ true ┆ true │
| │ true ┆ false │
| └──────┴───────┘
|
| is_first_distinct(self) -> 'Expr'
| Return a boolean mask indicating the first occurrence of each distinct value.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 3, 2]})
| >>> df.with_columns(pl.col("a").is_first_distinct().alias("first"))
| shape: (5, 2)
| ┌─────┬───────┐
| │ a ┆ first │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪═══════╡
| │ 1 ┆ true │
| │ 1 ┆ false │
| │ 2 ┆ true │
| │ 3 ┆ true │
| │ 2 ┆ false │
| └─────┴───────┘
|
| is_in(self, other: 'Expr | Collection[Any] | Series', *, nulls_equal: 'bool' = False) -> 'Expr'
| Check if elements of this expression are present in the other Series.
|
| Parameters
| ----------
| other
| Series or sequence of primitive type.
| nulls_equal : bool, default False
| If True, treat null as a distinct value. Null values will not propagate.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"sets": [[1, 2, 3], [1, 2], [9, 10]], "optional_members": [1, 2, 3]}
| ... )
| >>> df.with_columns(contains=pl.col("optional_members").is_in("sets"))
| shape: (3, 3)
| ┌───────────┬──────────────────┬──────────┐
| │ sets ┆ optional_members ┆ contains │
| │ --- ┆ --- ┆ --- │
| │ list[i64] ┆ i64 ┆ bool │
| ╞═══════════╪══════════════════╪══════════╡
| │ [1, 2, 3] ┆ 1 ┆ true │
| │ [1, 2] ┆ 2 ┆ true │
| │ [9, 10] ┆ 3 ┆ false │
| └───────────┴──────────────────┴──────────┘
|
| is_infinite(self) -> 'Expr'
| Returns a boolean Series indicating which values are infinite.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1.0, 2],
| ... "B": [3.0, float("inf")],
| ... }
| ... )
| >>> df.select(pl.all().is_infinite())
| shape: (2, 2)
| ┌───────┬───────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ bool ┆ bool │
| ╞═══════╪═══════╡
| │ false ┆ false │
| │ false ┆ true │
| └───────┴───────┘
|
| is_last_distinct(self) -> 'Expr'
| Return a boolean mask indicating the last occurrence of each distinct value.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 3, 2]})
| >>> df.with_columns(pl.col("a").is_last_distinct().alias("last"))
| shape: (5, 2)
| ┌─────┬───────┐
| │ a ┆ last │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪═══════╡
| │ 1 ┆ false │
| │ 1 ┆ true │
| │ 2 ┆ false │
| │ 3 ┆ true │
| │ 2 ┆ true │
| └─────┴───────┘
|
| is_nan(self) -> 'Expr'
| Returns a boolean Series indicating which values are NaN.
|
| Notes
| -----
| Floating point `NaN` (Not A Number) should not be confused
| with missing data represented as `Null/None`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(pl.col(pl.Float64).is_nan().name.suffix("_isnan"))
| shape: (5, 3)
| ┌──────┬─────┬─────────┐
| │ a ┆ b ┆ b_isnan │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪═════╪═════════╡
| │ 1 ┆ 1.0 ┆ false │
| │ 2 ┆ 2.0 ┆ false │
| │ null ┆ NaN ┆ true │
| │ 1 ┆ 1.0 ┆ false │
| │ 5 ┆ 5.0 ┆ false │
| └──────┴─────┴─────────┘
|
| is_not_nan(self) -> 'Expr'
| Returns a boolean Series indicating which values are not NaN.
|
| Notes
| -----
| Floating point `NaN` (Not A Number) should not be confused
| with missing data represented as `Null/None`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(pl.col(pl.Float64).is_not_nan().name.suffix("_is_not_nan"))
| shape: (5, 3)
| ┌──────┬─────┬──────────────┐
| │ a ┆ b ┆ b_is_not_nan │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪═════╪══════════════╡
| │ 1 ┆ 1.0 ┆ true │
| │ 2 ┆ 2.0 ┆ true │
| │ null ┆ NaN ┆ false │
| │ 1 ┆ 1.0 ┆ true │
| │ 5 ┆ 5.0 ┆ true │
| └──────┴─────┴──────────────┘
|
| is_not_null(self) -> 'Expr'
| Returns a boolean Series indicating which values are not null.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.all().is_not_null().name.suffix("_not_null") # nan != null
| ... )
| shape: (5, 4)
| ┌──────┬─────┬────────────┬────────────┐
| │ a ┆ b ┆ a_not_null ┆ b_not_null │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪═════╪════════════╪════════════╡
| │ 1 ┆ 1.0 ┆ true ┆ true │
| │ 2 ┆ 2.0 ┆ true ┆ true │
| │ null ┆ NaN ┆ false ┆ true │
| │ 1 ┆ 1.0 ┆ true ┆ true │
| │ 5 ┆ 5.0 ┆ true ┆ true │
| └──────┴─────┴────────────┴────────────┘
|
| is_null(self) -> 'Expr'
| Returns a boolean Series indicating which values are null.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(pl.all().is_null().name.suffix("_isnull")) # nan != null
| shape: (5, 4)
| ┌──────┬─────┬──────────┬──────────┐
| │ a ┆ b ┆ a_isnull ┆ b_isnull │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪═════╪══════════╪══════════╡
| │ 1 ┆ 1.0 ┆ false ┆ false │
| │ 2 ┆ 2.0 ┆ false ┆ false │
| │ null ┆ NaN ┆ true ┆ false │
| │ 1 ┆ 1.0 ┆ false ┆ false │
| │ 5 ┆ 5.0 ┆ false ┆ false │
| └──────┴─────┴──────────┴──────────┘
|
| is_sorted(self, *, descending: 'bool | None' = False, nulls_last: 'bool | None' = False) -> 'Expr'
| Checks if an expression is sorted.
|
| If `descending` and/or `nulls_last` are None, it will check `True` and `False`
| for the unspecified option(s), and return `True` if the expression is sorted
| under any combination of those settings.
|
| Parameters
| ----------
| descending
| Checks if the expression is sorted in descending order.
| Defaults to False.
| nulls_last
| Consider null values as being ordered last when checking sortedness.
| Defaults to False.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| Check if a column is sorted in ascending order.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.select(pl.col("a").is_sorted())
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ bool │
| ╞══════╡
| │ true │
| └──────┘
|
| Check if a column is sorted in descending order.
|
| >>> df = pl.DataFrame({"a": [4, 3, 2, 1]})
| >>> df.select(pl.col("a").is_sorted(descending=True))
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ bool │
| ╞══════╡
| │ true │
| └──────┘
|
| Check if a column is sorted in either direction.
|
| >>> df = pl.DataFrame({"a": [4, 3, 2, 1]})
| >>> df.select(pl.col("a").is_sorted(descending=None))
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ bool │
| ╞══════╡
| │ true │
| └──────┘
|
| is_unique(self) -> 'Expr'
| Get mask of unique values.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
| >>> df.select(pl.col("a").is_unique())
| shape: (3, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ false │
| │ true │
| └───────┘
|
| item(self, *, allow_empty: 'bool' = False) -> 'Expr'
| Get the single value.
|
| This raises an error if there is not exactly one value.
|
| Parameters
| ----------
| allow_empty
| Allow having no values to return `null`.
|
| See Also
| --------
| :meth:`Expr.get` : Get a single value by index.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1]})
| >>> df.select(pl.col("a").item())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| └─────┘
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").item())
| Traceback (most recent call last):
| ...
| polars.exceptions.ComputeError: aggregation 'item' expected a single value, got 3 values
| ...
| >>> df.head(0).select(pl.col("a").item(allow_empty=True))
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| └──────┘
|
| kurtosis(self, *, fisher: 'bool' = True, bias: 'bool' = True) -> 'Expr'
| Compute the kurtosis (Fisher or Pearson) of a dataset.
|
| Kurtosis is the fourth central moment divided by the square of the
| variance. If Fisher's definition is used, then 3.0 is subtracted from
| the result to give 0.0 for a normal distribution.
| If bias is False then the kurtosis is calculated using k statistics to
| eliminate bias coming from biased moment estimators.
|
| See scipy.stats for more information
|
| Parameters
| ----------
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").kurtosis())
| shape: (1, 1)
| ┌───────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═══════════╡
| │ -1.153061 │
| └───────────┘
|
| last(self, *, ignore_nulls: 'bool' = False) -> 'Expr'
| Get the last value.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the last non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 2]})
| >>> df.select(pl.col("a").last())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| └─────┘
|
| le(self, other: 'Any') -> 'Expr'
| Method equivalent of "less than or equal" operator `expr <= other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5.0, 4.0, float("nan"), 0.5],
| ... "y": [5.0, 3.5, float("nan"), 2.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").le(pl.col("y")).alias("x <= y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x <= y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 5.0 ┆ 5.0 ┆ true │
| │ 4.0 ┆ 3.5 ┆ false │
| │ NaN ┆ NaN ┆ true │
| │ 0.5 ┆ 2.0 ┆ true │
| └─────┴─────┴────────┘
|
| len(self) -> 'Expr'
| Return the number of elements in the column.
|
| Null values count towards the total.
|
| Returns
| -------
| Expr
| Expression of data type :class:`UInt32`.
|
| See Also
| --------
| count
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [None, 4, 4]})
| >>> df.select(pl.all().len())
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 3 ┆ 3 │
| └─────┴─────┘
|
| limit(self, n: 'int | Expr' = 10) -> 'Expr'
| Get the first `n` rows (alias for :func:`Expr.head`).
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7]})
| >>> df.select(pl.col("foo").limit(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| log(self, base: 'float | IntoExpr' = 2.718281828459045) -> 'Expr'
| Compute the logarithm to a given base.
|
| Parameters
| ----------
| base
| Given base, defaults to `e`
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").log(base=2))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.0 │
| │ 1.0 │
| │ 1.584963 │
| └──────────┘
|
| log10(self) -> 'Expr'
| Compute the base 10 logarithm of the input array, element-wise.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").log10())
| shape: (3, 1)
| ┌─────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞═════════╡
| │ 0.0 │
| │ 0.30103 │
| │ 0.60206 │
| └─────────┘
|
| log1p(self) -> 'Expr'
| Compute the natural logarithm of each element plus one.
|
| This computes `log(1 + x)` but is more numerically stable for `x` close to zero.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").log1p())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.693147 │
| │ 1.098612 │
| │ 1.386294 │
| └──────────┘
|
| lower_bound(self) -> 'Expr'
| Calculate the lower bound.
|
| Returns a unit Series with the lowest value possible for the dtype of this
| expression.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").lower_bound())
| shape: (1, 1)
| ┌──────────────────────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════════════════════╡
| │ -9223372036854775808 │
| └──────────────────────┘
|
| lt(self, other: 'Any') -> 'Expr'
| Method equivalent of "less than" operator `expr < other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 3.0],
| ... "y": [2.0, 2.0, float("nan"), 4.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").lt(pl.col("y")).alias("x < y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────┐
| │ x ┆ y ┆ x < y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪═══════╡
| │ 1.0 ┆ 2.0 ┆ true │
| │ 2.0 ┆ 2.0 ┆ false │
| │ NaN ┆ NaN ┆ false │
| │ 3.0 ┆ 4.0 ┆ true │
| └─────┴─────┴───────┘
|
| map_batches(self, function: 'Callable[[Series], Series | Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, agg_list: 'bool' = False, is_elementwise: 'bool' = False, returns_scalar: 'bool' = False) -> 'Expr'
| Apply a custom python function to a whole Series or sequence of Series.
|
| The output of this custom function is presumed to be either a Series,
| or a NumPy array (in which case it will be automatically converted into
| a Series), or a scalar that will be converted into a Series. If the
| result is a scalar and you want it to stay as a scalar, pass in
| ``returns_scalar=True``. If you want to apply a
| custom function elementwise over single values, see :func:`map_elements`.
| A reasonable use case for `map` functions is transforming the values
| represented by an expression using a third-party library.
|
| Parameters
| ----------
| function
| Lambda/function to apply.
| return_dtype
| Datatype of the output Series.
|
| It is recommended to set this whenever possible. If this is `None`, it tries
| to infer the datatype by calling the function with dummy data and looking at
| the output.
| agg_list
| First implode when in a group-by aggregation.
|
| .. deprecated:: 1.32.0
|
| Use `expr.implode().map_batches(..)` instead.
| is_elementwise
| Set to true if the operations is elementwise for better performance
| and optimization.
|
| An elementwise operations has unit or equal length for all inputs
| and can be ran sequentially on slices without results being affected.
| returns_scalar
| If the function returns a scalar, by default it will be wrapped in
| a list in the output, since the assumption is that the function
| always returns something Series-like. If you want to keep the
| result as a scalar, set this argument to True.
|
| Notes
| -----
| A UDF passed to `map_batches` must be pure, meaning that it cannot modify
| or depend on state other than its arguments. Polars may call the function
| with arbitrary input data.
|
| See Also
| --------
| map_elements
| replace
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "sine": [0.0, 1.0, 0.0, -1.0],
| ... "cosine": [1.0, 0.0, -1.0, 0.0],
| ... }
| ... )
| >>> df.select(
| ... pl.all().map_batches(
| ... lambda x: x.to_numpy().argmax(),
| ... returns_scalar=True,
| ... )
| ... )
| shape: (1, 2)
| ┌──────┬────────┐
| │ sine ┆ cosine │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪════════╡
| │ 1 ┆ 0 │
| └──────┴────────┘
|
| Here's an example of a function that returns a scalar, where we want it
| to stay as a scalar:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [0, 1, 0, 1],
| ... "b": [1, 2, 3, 4],
| ... }
| ... )
| >>> df.group_by("a").agg(
| ... pl.col("b").map_batches(
| ... lambda x: x.max(), returns_scalar=True, return_dtype=pl.self_dtype()
| ... )
| ... ) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 0 ┆ 3 │
| └─────┴─────┘
|
| Call a function that takes multiple arguments by creating a `struct` and
| referencing its fields inside the function call.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [5, 1, 0, 3],
| ... "b": [4, 2, 3, 4],
| ... }
| ... )
| >>> df.with_columns(
| ... a_times_b=pl.struct("a", "b").map_batches(
| ... lambda x: np.multiply(x.struct.field("a"), x.struct.field("b")),
| ... return_dtype=pl.Int64,
| ... )
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────────┐
| │ a ┆ b ┆ a_times_b │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═══════════╡
| │ 5 ┆ 4 ┆ 20 │
| │ 1 ┆ 2 ┆ 2 │
| │ 0 ┆ 3 ┆ 0 │
| │ 3 ┆ 4 ┆ 12 │
| └─────┴─────┴───────────┘
|
| map_elements(self, function: 'Callable[[Any], Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, skip_nulls: 'bool' = True, pass_name: 'bool' = False, strategy: 'MapElementsStrategy' = 'thread_local', returns_scalar: 'bool' = False) -> 'Expr'
| Map a custom/user-defined function (UDF) to each element of a column.
|
| .. warning::
| This method is much slower than the native expressions API.
| Only use it if you cannot implement your logic otherwise.
|
| Suppose that the function is: `x ↦ sqrt(x)`:
|
| - For mapping elements of a series, consider:
| `pl.col("col_name").sqrt()`.
| - For mapping inner elements of lists, consider:
| `pl.col("col_name").list.eval(pl.element().sqrt())`.
| - For mapping elements of struct fields, consider:
| `pl.col("col_name").struct.field("field_name").sqrt()`.
|
| If you want to replace the original column or field,
| consider :meth:`.with_columns <polars.DataFrame.with_columns>`
| and :meth:`.with_fields <polars.Expr.struct.with_fields>`.
|
| Parameters
| ----------
| function
| Lambda/function to map.
| return_dtype
| Datatype of the output Series.
|
| It is recommended to set this whenever possible. If this is `None`, it tries
| to infer the datatype by calling the function with dummy data and looking at
| the output.
| skip_nulls
| Don't map the function over values that contain nulls (this is faster).
| pass_name
| Pass the Series name to the custom function (this is more expensive).
| returns_scalar
| .. deprecated:: 1.32.0
| Is ignored and will be removed in 2.0.
| strategy : {'thread_local', 'threading'}
| The threading strategy to use.
|
| - 'thread_local': run the python function on a single thread.
| - 'threading': run the python function on separate threads. Use with
| care as this can slow performance. This might only speed up
| your code if the amount of work per element is significant
| and the python function releases the GIL (e.g. via calling
| a c function)
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Notes
| -----
| * Using `map_elements` is strongly discouraged as you will be effectively
| running python "for" loops, which will be very slow. Wherever possible you
| should prefer the native expression API to achieve the best performance.
|
| * If your function is expensive and you don't want it to be called more than
| once for a given input, consider applying an `@lru_cache` decorator to it.
| If your data is suitable you may achieve *significant* speedups.
|
| * Window function application using `over` is considered a GroupBy context
| here, so `map_elements` can be used to map functions over window groups.
|
| * A UDF passed to `map_elements` must be pure, meaning that it cannot modify or
| depend on state other than its arguments. Polars may call the function
| with arbitrary input data.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 1],
| ... "b": ["a", "b", "c", "c"],
| ... }
| ... )
|
| The function is applied to each element of column `'a'`:
|
| >>> df.with_columns( # doctest: +SKIP
| ... pl.col("a")
| ... .map_elements(lambda x: x * 2, return_dtype=pl.self_dtype())
| ... .alias("a_times_2"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────────┐
| │ a ┆ b ┆ a_times_2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ i64 │
| ╞═════╪═════╪═══════════╡
| │ 1 ┆ a ┆ 2 │
| │ 2 ┆ b ┆ 4 │
| │ 3 ┆ c ┆ 6 │
| │ 1 ┆ c ┆ 2 │
| └─────┴─────┴───────────┘
|
| Tip: it is better to implement this with an expression:
|
| >>> df.with_columns(
| ... (pl.col("a") * 2).alias("a_times_2"),
| ... ) # doctest: +IGNORE_RESULT
|
| >>> (
| ... df.lazy()
| ... .group_by("b")
| ... .agg(
| ... pl.col("a")
| ... .implode()
| ... .map_elements(lambda x: x.sum(), return_dtype=pl.Int64)
| ... )
| ... .collect()
| ... ) # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌─────┬─────┐
| │ b ┆ a │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ b ┆ 2 │
| │ c ┆ 4 │
| └─────┴─────┘
|
| Tip: again, it is better to implement this with an expression:
|
| >>> (
| ... df.lazy()
| ... .group_by("b", maintain_order=True)
| ... .agg(pl.col("a").sum())
| ... .collect()
| ... ) # doctest: +IGNORE_RESULT
|
| Window function application using `over` will behave as a GroupBy
| context, with your function receiving individual window groups:
|
| >>> df = pl.DataFrame(
| ... {
| ... "key": ["x", "x", "y", "x", "y", "z"],
| ... "val": [1, 1, 1, 1, 1, 1],
| ... }
| ... )
| >>> df.with_columns(
| ... scaled=pl.col("val")
| ... .implode()
| ... .map_elements(lambda s: s * len(s), return_dtype=pl.List(pl.Int64))
| ... .explode(empty_as_null=False)
| ... .over("key"),
| ... ).sort("key")
| shape: (6, 3)
| ┌─────┬─────┬────────┐
| │ key ┆ val ┆ scaled │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪════════╡
| │ x ┆ 1 ┆ 3 │
| │ x ┆ 1 ┆ 3 │
| │ x ┆ 1 ┆ 3 │
| │ y ┆ 1 ┆ 2 │
| │ y ┆ 1 ┆ 2 │
| │ z ┆ 1 ┆ 1 │
| └─────┴─────┴────────┘
|
| Note that this function would *also* be better-implemented natively:
|
| >>> df.with_columns(
| ... scaled=(pl.col("val") * pl.col("val").count()).over("key"),
| ... ).sort("key") # doctest: +IGNORE_RESULT
|
| max(self) -> 'Expr'
| Get maximum value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0]})
| >>> df.select(pl.col("a").max())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
|
| max_by(self, by: 'IntoExpr') -> 'Expr'
| Get maximum value, ordered by another expression.
|
| If the by expression has multiple values equal to the maximum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the largest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0], "b": ["x", "y", "z"]})
| >>> df.select(pl.col("b").max_by("a"))
| shape: (1, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ str │
| ╞═════╡
| │ z │
| └─────┘
|
| mean(self) -> 'Expr'
| Get mean value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").mean())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| └─────┘
|
| median(self) -> 'Expr'
| Get median value using linear interpolation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").median())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| └─────┘
|
| min(self) -> 'Expr'
| Get minimum value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0]})
| >>> df.select(pl.col("a").min())
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ -1.0 │
| └──────┘
|
| min_by(self, by: 'IntoExpr') -> 'Expr'
| Get minimum value, ordered by another expression.
|
| If the by expression has multiple values equal to the minimum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the smallest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0], "b": ["x", "y", "z"]})
| >>> df.select(pl.col("b").min_by("a"))
| shape: (1, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ str │
| ╞═════╡
| │ x │
| └─────┘
|
| mod(self, other: 'Any') -> 'Expr'
| Method equivalent of modulus operator `expr % other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [0, 1, 2, 3, 4]})
| >>> df.with_columns(pl.col("x").mod(2).alias("x%2"))
| shape: (5, 2)
| ┌─────┬─────┐
| │ x ┆ x%2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 0 ┆ 0 │
| │ 1 ┆ 1 │
| │ 2 ┆ 0 │
| │ 3 ┆ 1 │
| │ 4 ┆ 0 │
| └─────┴─────┘
|
| mode(self, *, maintain_order: 'bool' = False) -> 'Expr'
| Compute the most occurring value(s).
|
| Can return multiple Values.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 1, 2, 3],
| ... "b": [1, 1, 2, 2],
| ... }
| ... )
| >>> df.select(pl.all().mode().first()) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| └─────┴─────┘
|
| mul(self, other: 'Any') -> 'Expr'
| Method equivalent of multiplication operator `expr * other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 4, 8, 16]})
| >>> df.with_columns(
| ... pl.col("x").mul(2).alias("x*2"),
| ... pl.col("x").mul(pl.col("x").log(2)).alias("x * xlog2"),
| ... )
| shape: (5, 3)
| ┌─────┬─────┬───────────┐
| │ x ┆ x*2 ┆ x * xlog2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪═════╪═══════════╡
| │ 1 ┆ 2 ┆ 0.0 │
| │ 2 ┆ 4 ┆ 2.0 │
| │ 4 ┆ 8 ┆ 8.0 │
| │ 8 ┆ 16 ┆ 24.0 │
| │ 16 ┆ 32 ┆ 64.0 │
| └─────┴─────┴───────────┘
|
| n_unique(self) -> 'Expr'
| Count unique values.
|
| Notes
| -----
| `null` is considered to be a unique value for the purposes of this operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 1, 2, 2, 3], "y": [1, 1, 1, None, None]})
| >>> df.select(
| ... x_unique=pl.col("x").n_unique(),
| ... y_unique=pl.col("y").n_unique(),
| ... )
| shape: (1, 2)
| ┌──────────┬──────────┐
| │ x_unique ┆ y_unique │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞══════════╪══════════╡
| │ 3 ┆ 2 │
| └──────────┴──────────┘
|
| nan_max(self) -> 'Expr'
| Get maximum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.0, float("nan")]})
| >>> df.select(pl.col("a").nan_max())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ NaN │
| └─────┘
|
| nan_min(self) -> 'Expr'
| Get minimum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.0, float("nan")]})
| >>> df.select(pl.col("a").nan_min())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ NaN │
| └─────┘
|
| ne(self, other: 'Any') -> 'Expr'
| Method equivalent of inequality operator `expr != other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0],
| ... "y": [2.0, 2.0, float("nan"), 4.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").ne(pl.col("y")).alias("x != y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x != y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 1.0 ┆ 2.0 ┆ true │
| │ 2.0 ┆ 2.0 ┆ false │
| │ NaN ┆ NaN ┆ false │
| │ 4.0 ┆ 4.0 ┆ false │
| └─────┴─────┴────────┘
|
| ne_missing(self, other: 'Any') -> 'Expr'
| Method equivalent of equality operator `expr != other` where `None == None`.
|
| This differs from default `ne` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0, None, None],
| ... "y": [2.0, 2.0, float("nan"), 4.0, 5.0, None],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").ne(pl.col("y")).alias("x ne y"),
| ... pl.col("x").ne_missing(pl.col("y")).alias("x ne_missing y"),
| ... )
| shape: (6, 4)
| ┌──────┬──────┬────────┬────────────────┐
| │ x ┆ y ┆ x ne y ┆ x ne_missing y │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪══════╪════════╪════════════════╡
| │ 1.0 ┆ 2.0 ┆ true ┆ true │
| │ 2.0 ┆ 2.0 ┆ false ┆ false │
| │ NaN ┆ NaN ┆ false ┆ false │
| │ 4.0 ┆ 4.0 ┆ false ┆ false │
| │ null ┆ 5.0 ┆ null ┆ true │
| │ null ┆ null ┆ null ┆ false │
| └──────┴──────┴────────┴────────────────┘
|
| neg(self) -> 'Expr'
| Method equivalent of unary minus operator `-expr`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 2, None]})
| >>> df.with_columns(pl.col("a").neg())
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 1 │
| │ 0 │
| │ -2 │
| │ null │
| └──────┘
|
| not_(self) -> 'Expr'
| Method equivalent of bitwise "not" operator `~expr`.
|
| This has the effect of negating logical boolean expressions,
| but operates bitwise on integers.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "label": ["aa", "bb", "cc", "dd", "ee"],
| ... "valid": [True, False, None, False, True],
| ... "int_code": [1, 0, 2, None, -1],
| ... }
| ... )
|
| Apply "not" to boolean expression (negates the value) and
| integer expression (operates bitwise):
|
| >>> df.with_columns(
| ... not_valid=pl.col("valid").not_(),
| ... not_int_code=pl.col("int_code").not_(),
| ... )
| shape: (5, 5)
| ┌───────┬───────┬──────────┬───────────┬──────────────┐
| │ label ┆ valid ┆ int_code ┆ not_valid ┆ not_int_code │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ bool ┆ i64 ┆ bool ┆ i64 │
| ╞═══════╪═══════╪══════════╪═══════════╪══════════════╡
| │ aa ┆ true ┆ 1 ┆ false ┆ -2 │
| │ bb ┆ false ┆ 0 ┆ true ┆ -1 │
| │ cc ┆ null ┆ 2 ┆ null ┆ -3 │
| │ dd ┆ false ┆ null ┆ true ┆ null │
| │ ee ┆ true ┆ -1 ┆ false ┆ 0 │
| └───────┴───────┴──────────┴───────────┴──────────────┘
|
| null_count(self) -> 'Expr'
| Count null values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 1, None],
| ... "b": [10, None, 300],
| ... "c": [350, 650, 850],
| ... }
| ... )
| >>> df.select(pl.all().null_count())
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 2 ┆ 1 ┆ 0 │
| └─────┴─────┴─────┘
|
| or_(self, *others: 'Any') -> 'Expr'
| Method equivalent of bitwise "or" operator `expr | other | ...`.
|
| This has the effect of combining logical boolean expressions,
| but operates bitwise on integers.
|
| Parameters
| ----------
| *others
| One or more integer or boolean expressions to evaluate/combine.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5, 6, 7, 4, 8],
| ... "y": [1.5, 2.5, 1.0, 4.0, -5.75],
| ... "z": [-9, 2, -1, 4, 8],
| ... }
| ... )
|
| Combine logical "or" conditions:
|
| >>> df.select(
| ... (pl.col("x") == pl.col("y"))
| ... .or_(
| ... pl.col("x") == pl.col("y"),
| ... pl.col("y") == pl.col("z"),
| ... pl.col("y").cast(int) == pl.col("z"),
| ... )
| ... .alias("any")
| ... )
| shape: (5, 1)
| ┌───────┐
| │ any │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ true │
| │ false │
| │ true │
| │ false │
| └───────┘
|
| Bitwise "or" operation on integer columns:
|
| >>> df.select("x", "z", x_or_z=pl.col("x").or_(pl.col("z")))
| shape: (5, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ z ┆ x_or_z │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪════════╡
| │ 5 ┆ -9 ┆ -9 │
| │ 6 ┆ 2 ┆ 6 │
| │ 7 ┆ -1 ┆ -1 │
| │ 4 ┆ 4 ┆ 4 │
| │ 8 ┆ 8 ┆ 8 │
| └─────┴─────┴────────┘
|
| over(self, partition_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, *more_exprs: 'IntoExpr', order_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, descending: 'bool' = False, nulls_last: 'bool' = False, mapping_strategy: 'WindowMappingStrategy' = 'group_to_rows') -> 'Expr'
| Compute expressions over the given groups.
|
| This expression is similar to performing a group by aggregation and joining the
| result back into the original DataFrame.
|
| The outcome is similar to how `window functions
| <https://www.postgresql.org/docs/current/tutorial-window.html>`_
| work in PostgreSQL.
|
| Parameters
| ----------
| partition_by
| Column(s) to group by. Accepts expression input. Strings are parsed as
| column names.
| *more_exprs
| Additional columns to group by, specified as positional arguments.
| order_by
| Order rows within each partition group before evaluating the expression.
| Useful for order-sensitive operations such as
| :func:`cum_sum` or :func:`diff`.
| descending
| In case 'order_by' is given, indicate whether to order in
| ascending or descending order.
| nulls_last
| In case 'order_by' is given, indicate whether to order
| the nulls in last position.
| mapping_strategy: {'group_to_rows', 'join', 'explode'}
| - group_to_rows
| If the aggregation results in multiple values per group, map them back
| to their row position in the DataFrame. This can only be done if each
| group yields the same number of elements before aggregation as after. If
| the aggregation results in one scalar value per group, this value will
| be mapped to every row.
| - join
| If the aggregation may result in multiple values per group, join the
| values as 'List<group_dtype>' to each row position. Warning: this can be
| memory intensive. If the aggregation always results in one scalar value
| per group, join this value as '<group_dtype>' to each row position.
| - explode
| If the aggregation may result in multiple values per group, map each
| value to a new row, similar to the results of `group_by` + `agg` +
| `explode`. If the aggregation always results in one scalar value per
| group, map this value to one row position. Sorting of the given groups
| is required if the groups are not part of the window operation for the
| operation, otherwise the result would not make sense. This operation
| changes the number of rows.
|
| Examples
| --------
| Pass the name of a column to compute the expression over that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "a", "b", "b", "b"],
| ... "b": [1, 2, 3, 5, 3],
| ... "c": [5, 4, 3, 2, 1],
| ... }
| ... )
| >>> df.with_columns(c_max=pl.col("c").max().over("a"))
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────┐
| │ a ┆ b ┆ c ┆ c_max │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ a ┆ 1 ┆ 5 ┆ 5 │
| │ a ┆ 2 ┆ 4 ┆ 5 │
| │ b ┆ 3 ┆ 3 ┆ 3 │
| │ b ┆ 5 ┆ 2 ┆ 3 │
| │ b ┆ 3 ┆ 1 ┆ 3 │
| └─────┴─────┴─────┴───────┘
|
| Expression input is also supported.
|
| >>> df.with_columns(c_max=pl.col("c").max().over(pl.col("b") // 2))
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────┐
| │ a ┆ b ┆ c ┆ c_max │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ a ┆ 1 ┆ 5 ┆ 5 │
| │ a ┆ 2 ┆ 4 ┆ 4 │
| │ b ┆ 3 ┆ 3 ┆ 4 │
| │ b ┆ 5 ┆ 2 ┆ 2 │
| │ b ┆ 3 ┆ 1 ┆ 4 │
| └─────┴─────┴─────┴───────┘
|
| Group by multiple columns by passing multiple column names or expressions.
|
| >>> df.with_columns(c_min=pl.col("c").min().over("a", pl.col("b") % 2))
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────┐
| │ a ┆ b ┆ c ┆ c_min │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ a ┆ 1 ┆ 5 ┆ 5 │
| │ a ┆ 2 ┆ 4 ┆ 4 │
| │ b ┆ 3 ┆ 3 ┆ 1 │
| │ b ┆ 5 ┆ 2 ┆ 1 │
| │ b ┆ 3 ┆ 1 ┆ 1 │
| └─────┴─────┴─────┴───────┘
|
| Mapping strategy `join` joins the values by group.
|
| >>> df.with_columns(
| ... c_pairs=pl.col("c").head(2).over("a", mapping_strategy="join")
| ... )
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────────┐
| │ a ┆ b ┆ c ┆ c_pairs │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ list[i64] │
| ╞═════╪═════╪═════╪═══════════╡
| │ a ┆ 1 ┆ 5 ┆ [5, 4] │
| │ a ┆ 2 ┆ 4 ┆ [5, 4] │
| │ b ┆ 3 ┆ 3 ┆ [3, 2] │
| │ b ┆ 5 ┆ 2 ┆ [3, 2] │
| │ b ┆ 3 ┆ 1 ┆ [3, 2] │
| └─────┴─────┴─────┴───────────┘
|
| Mapping strategy `explode` maps the values to new rows, changing the shape.
|
| >>> df.select(
| ... c_first_2=pl.col("c").head(2).over("a", mapping_strategy="explode")
| ... )
| shape: (4, 1)
| ┌───────────┐
| │ c_first_2 │
| │ --- │
| │ i64 │
| ╞═══════════╡
| │ 5 │
| │ 4 │
| │ 3 │
| │ 2 │
| └───────────┘
|
| You can use non-elementwise expressions with `over` too. By default they are
| evaluated using row-order, but you can specify a different one using `order_by`.
|
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "store_id": ["a", "a", "b", "b"],
| ... "date": [
| ... date(2024, 9, 18),
| ... date(2024, 9, 17),
| ... date(2024, 9, 18),
| ... date(2024, 9, 16),
| ... ],
| ... "sales": [7, 9, 8, 10],
| ... }
| ... )
| >>> df.with_columns(
| ... cumulative_sales=pl.col("sales")
| ... .cum_sum()
| ... .over("store_id", order_by="date")
| ... )
| shape: (4, 4)
| ┌──────────┬────────────┬───────┬──────────────────┐
| │ store_id ┆ date ┆ sales ┆ cumulative_sales │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 ┆ i64 │
| ╞══════════╪════════════╪═══════╪══════════════════╡
| │ a ┆ 2024-09-18 ┆ 7 ┆ 16 │
| │ a ┆ 2024-09-17 ┆ 9 ┆ 9 │
| │ b ┆ 2024-09-18 ┆ 8 ┆ 18 │
| │ b ┆ 2024-09-16 ┆ 10 ┆ 10 │
| └──────────┴────────────┴───────┴──────────────────┘
|
| If you don't require that the group order be preserved, then the more performant
| option is to use `mapping_strategy='explode'` - be careful however to only ever
| use this in a `select` statement, not a `with_columns` one.
|
| >>> window = {
| ... "partition_by": "store_id",
| ... "order_by": "date",
| ... "mapping_strategy": "explode",
| ... }
| >>> df.select(
| ... pl.all().over(**window),
| ... cumulative_sales=pl.col("sales").cum_sum().over(**window),
| ... )
| shape: (4, 4)
| ┌──────────┬────────────┬───────┬──────────────────┐
| │ store_id ┆ date ┆ sales ┆ cumulative_sales │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 ┆ i64 │
| ╞══════════╪════════════╪═══════╪══════════════════╡
| │ a ┆ 2024-09-17 ┆ 9 ┆ 9 │
| │ a ┆ 2024-09-18 ┆ 7 ┆ 16 │
| │ b ┆ 2024-09-16 ┆ 10 ┆ 10 │
| │ b ┆ 2024-09-18 ┆ 8 ┆ 18 │
| └──────────┴────────────┴───────┴──────────────────┘
|
| pct_change(self, n: 'int | IntoExprColumn' = 1) -> 'Expr'
| Computes percentage change between values.
|
| Percentage change (as fraction) between current element and most-recent
| non-null element at least `n` period(s) before the current element.
|
| Computes the change from the previous row by default.
|
| Parameters
| ----------
| n
| periods to shift for forming percent change.
|
| Notes
| -----
| Null values are preserved. If you're coming from pandas, this matches
| their ``fill_method=None`` behaviour.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [10, 11, 12, None, 12],
| ... }
| ... )
| >>> df.with_columns(pl.col("a").pct_change().alias("pct_change"))
| shape: (5, 2)
| ┌──────┬────────────┐
| │ a ┆ pct_change │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞══════╪════════════╡
| │ 10 ┆ null │
| │ 11 ┆ 0.1 │
| │ 12 ┆ 0.090909 │
| │ null ┆ null │
| │ 12 ┆ null │
| └──────┴────────────┘
|
| peak_max(self) -> 'Expr'
| Get a boolean mask of the local maximum peaks.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5]})
| >>> df.select(pl.col("a").peak_max())
| shape: (5, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ false │
| │ false │
| │ false │
| │ true │
| └───────┘
|
| peak_min(self) -> 'Expr'
| Get a boolean mask of the local minimum peaks.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [4, 1, 3, 2, 5]})
| >>> df.select(pl.col("a").peak_min())
| shape: (5, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ true │
| │ false │
| │ true │
| │ false │
| └───────┘
|
| pipe(self, function: 'Callable[Concatenate[Expr, P], T]', *args: 'P.args', **kwargs: 'P.kwargs') -> 'T'
| Offers a structured way to apply a sequence of user-defined functions (UDFs).
|
| Parameters
| ----------
| function
| Callable; will receive the expression as the first parameter,
| followed by any given args/kwargs.
| *args
| Arguments to pass to the UDF.
| **kwargs
| Keyword arguments to pass to the UDF.
|
| Examples
| --------
| >>> def extract_number(expr: pl.Expr) -> pl.Expr:
| ... """Extract the digits from a string."""
| ... return expr.str.extract(r"\d+", 0).cast(pl.Int64)
| >>>
| >>> def scale_negative_even(expr: pl.Expr, *, n: int = 1) -> pl.Expr:
| ... """Set even numbers negative, and scale by a user-supplied value."""
| ... expr = pl.when(expr % 2 == 0).then(-expr).otherwise(expr)
| ... return expr * n
| >>>
| >>> df = pl.DataFrame({"val": ["a: 1", "b: 2", "c: 3", "d: 4"]})
| >>> df.with_columns(
| ... udfs=(
| ... pl.col("val").pipe(extract_number).pipe(scale_negative_even, n=5)
| ... ),
| ... )
| shape: (4, 2)
| ┌──────┬──────┐
| │ val ┆ udfs │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞══════╪══════╡
| │ a: 1 ┆ 5 │
| │ b: 2 ┆ -10 │
| │ c: 3 ┆ 15 │
| │ d: 4 ┆ -20 │
| └──────┴──────┘
|
| pow(self, exponent: 'IntoExprColumn | int | float') -> 'Expr'
| Method equivalent of exponentiation operator `expr ** exponent`.
|
| If the exponent is float, the result follows the dtype of exponent.
| Otherwise, it follows dtype of base.
|
| Parameters
| ----------
| exponent
| Numeric literal or expression exponent value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 4, 8]})
| >>> df.with_columns(
| ... pl.col("x").pow(3).alias("cube"),
| ... pl.col("x").pow(pl.col("x").log(2)).alias("x ** xlog2"),
| ... )
| shape: (4, 3)
| ┌─────┬──────┬────────────┐
| │ x ┆ cube ┆ x ** xlog2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪══════╪════════════╡
| │ 1 ┆ 1 ┆ 1.0 │
| │ 2 ┆ 8 ┆ 2.0 │
| │ 4 ┆ 64 ┆ 16.0 │
| │ 8 ┆ 512 ┆ 512.0 │
| └─────┴──────┴────────────┘
|
| Raising an integer to a positive integer results in an integer - in order
| to raise to a negative integer, you can cast either the base or the exponent
| to float first:
|
| >>> df.with_columns(
| ... x_squared=pl.col("x").pow(2),
| ... x_inverse=pl.col("x").pow(-1.0),
| ... )
| shape: (4, 3)
| ┌─────┬───────────┬───────────┐
| │ x ┆ x_squared ┆ x_inverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪═══════════╪═══════════╡
| │ 1 ┆ 1 ┆ 1.0 │
| │ 2 ┆ 4 ┆ 0.5 │
| │ 4 ┆ 16 ┆ 0.25 │
| │ 8 ┆ 64 ┆ 0.125 │
| └─────┴───────────┴───────────┘
|
| product(self) -> 'Expr'
| Compute the product of an expression.
|
| Notes
| -----
| If there are no non-null values, then the output is `1`.
| If you would prefer empty products to return `None`, you can
| use `pl.when(expr.count()>0).then(expr.product())` instead
| of `expr.product()`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").product())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 6 │
| └─────┘
|
| qcut(self, quantiles: 'Sequence[float] | int', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, allow_duplicates: 'bool' = False, include_breaks: 'bool' = False) -> 'Expr'
| Bin continuous values into discrete categories based on their quantiles.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| quantiles
| Either a list of quantile probabilities between 0 and 1 or a positive
| integer determining the number of bins with uniform probability.
| labels
| Names of the categories. The number of labels must be equal to the number
| of categories.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| allow_duplicates
| If set to `True`, duplicates in the resulting quantiles are dropped,
| rather than raising a `DuplicateError`. This can happen even with unique
| probabilities, depending on the data.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from a
| :class:`Categorical` to a :class:`Struct`.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Categorical` if `include_breaks` is set to
| `False` (default), otherwise an expression of data type :class:`Struct`.
|
| See Also
| --------
| cut
|
| Examples
| --------
| Divide a column into three categories according to pre-defined quantile
| probabilities.
|
| >>> df = pl.DataFrame({"foo": [-2, -1, 0, 1, 2]})
| >>> df.with_columns(
| ... pl.col("foo").qcut([0.25, 0.75], labels=["a", "b", "c"]).alias("qcut")
| ... )
| shape: (5, 2)
| ┌─────┬──────┐
| │ foo ┆ qcut │
| │ --- ┆ --- │
| │ i64 ┆ cat │
| ╞═════╪══════╡
| │ -2 ┆ a │
| │ -1 ┆ a │
| │ 0 ┆ b │
| │ 1 ┆ b │
| │ 2 ┆ c │
| └─────┴──────┘
|
| Divide a column into two categories using uniform quantile probabilities.
|
| >>> df.with_columns(
| ... pl.col("foo")
| ... .qcut(2, labels=["low", "high"], left_closed=True)
| ... .alias("qcut")
| ... )
| shape: (5, 2)
| ┌─────┬──────┐
| │ foo ┆ qcut │
| │ --- ┆ --- │
| │ i64 ┆ cat │
| ╞═════╪══════╡
| │ -2 ┆ low │
| │ -1 ┆ low │
| │ 0 ┆ high │
| │ 1 ┆ high │
| │ 2 ┆ high │
| └─────┴──────┘
|
| Add both the category and the breakpoint.
|
| >>> df.with_columns(
| ... pl.col("foo").qcut([0.25, 0.75], include_breaks=True).alias("qcut")
| ... ).unnest("qcut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ cat │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| quantile(self, quantile: 'float | list_[float] | Expr', interpolation: 'QuantileMethod' = 'nearest') -> 'Expr'
| Get quantile value.
|
| Parameters
| ----------
| quantile
| Quantile(s) between 0.0 and 1.0. Can be a single float or a list of floats.
|
| - If a single float, returns a single f64 value per row.
| - If a list of floats, returns a list of f64 values per row (one value per quantile).
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0, 1, 2, 3, 4, 5]})
| >>> df.select(pl.col("a").quantile(0.3))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 2.0 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="higher"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 2.0 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="lower"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="midpoint"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.5 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="linear"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.5 │
| └─────┘
| >>> df.select(pl.col("a").quantile([0.25, 0.75], interpolation="linear"))
| shape: (1, 1)
| ┌──────────────┐
| │ a │
| │ --- │
| │ list[f64] │
| ╞══════════════╡
| │ [1.25, 3.75] │
| └──────────────┘
|
| radians(self) -> 'Expr'
| Convert from degrees to radians.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-720, -540, -360, -180, 0, 180, 360, 540, 720]})
| >>> df.select(pl.col("a").radians())
| shape: (9, 1)
| ┌────────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════════╡
| │ -12.566371 │
| │ -9.424778 │
| │ -6.283185 │
| │ -3.141593 │
| │ 0.0 │
| │ 3.141593 │
| │ 6.283185 │
| │ 9.424778 │
| │ 12.566371 │
| └────────────┘
|
| rank(self, method: 'RankMethod' = 'average', *, descending: 'bool' = False, seed: 'int | None' = None) -> 'Expr'
| Assign ranks to data, dealing with ties appropriately.
|
| Parameters
| ----------
| method : {'average', 'min', 'max', 'dense', 'ordinal', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'ordinal' : All values are given a distinct rank, corresponding to
| the order that the values occur in the Series.
| - 'random' : Like 'ordinal', but the rank for ties is not dependent
| on the order that the values occur in the Series.
| descending
| Rank in descending order.
| seed
| If `method="random"`, use this as seed.
|
| Notes
| -----
| If you're coming from SQL, you may be expecting null values to be ranked last.
| Polars, however, only ranks non-null values and preserves the null ones.
|
| Examples
| --------
| The 'average' method:
|
| >>> df = pl.DataFrame({"a": [3, 6, 1, 1, 6]})
| >>> df.select(pl.col("a").rank())
| shape: (5, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 3.0 │
| │ 4.5 │
| │ 1.5 │
| │ 1.5 │
| │ 4.5 │
| └─────┘
|
| The 'ordinal' method:
|
| >>> df = pl.DataFrame({"a": [3, 6, 1, 1, 6]})
| >>> df.select(pl.col("a").rank("ordinal"))
| shape: (5, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 3 │
| │ 4 │
| │ 1 │
| │ 2 │
| │ 5 │
| └─────┘
|
| Use 'rank' with 'over' to rank within groups:
|
| >>> df = pl.DataFrame({"a": [1, 1, 2, 2, 2], "b": [6, 7, 5, 14, 11]})
| >>> df.with_columns(pl.col("b").rank().over("a").alias("rank"))
| shape: (5, 3)
| ┌─────┬─────┬──────┐
| │ a ┆ b ┆ rank │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 6 ┆ 1.0 │
| │ 1 ┆ 7 ┆ 2.0 │
| │ 2 ┆ 5 ┆ 1.0 │
| │ 2 ┆ 14 ┆ 3.0 │
| │ 2 ┆ 11 ┆ 2.0 │
| └─────┴─────┴──────┘
|
| Divide by the length or number of non-null values
| to compute the percentile rank.
|
| >>> df = pl.DataFrame({"a": [6, 7, None, 14, 11]})
| >>> df.with_columns(
| ... pct=pl.col("a").rank() / pl.len(),
| ... pct_valid=pl.col("a").rank() / pl.count("a"),
| ... )
| shape: (5, 3)
| ┌──────┬──────┬───────────┐
| │ a ┆ pct ┆ pct_valid │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪═══════════╡
| │ 6 ┆ 0.2 ┆ 0.25 │
| │ 7 ┆ 0.4 ┆ 0.5 │
| │ null ┆ null ┆ null │
| │ 14 ┆ 0.8 ┆ 1.0 │
| │ 11 ┆ 0.6 ┆ 0.75 │
| └──────┴──────┴───────────┘
|
| rechunk(self) -> 'Expr'
| Create a single chunk of memory for this Series.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
|
| Create a Series with 3 nulls, append column `a`, then rechunk.
|
| >>> df.select(pl.repeat(None, 3).append(pl.col("a")).rechunk())
| shape: (6, 1)
| ┌────────┐
| │ repeat │
| │ --- │
| │ i64 │
| ╞════════╡
| │ null │
| │ null │
| │ null │
| │ 1 │
| │ 1 │
| │ 2 │
| └────────┘
|
| register_plugin(self, *, lib: 'str_', symbol: 'str_', args: 'list_[IntoExpr] | None' = None, kwargs: 'dict[Any, Any] | None' = None, is_elementwise: 'bool' = False, input_wildcard_expansion: 'bool' = False, returns_scalar: 'bool' = False, cast_to_supertypes: 'bool' = False, pass_name_to_apply: 'bool' = False, changes_length: 'bool' = False) -> 'Expr'
| Register a plugin function.
|
| .. deprecated:: 0.20.16
| Use :func:`polars.plugins.register_plugin_function` instead.
|
| See the `user guide <https://docs.pola.rs/user-guide/plugins/>`_
| for more information about plugins.
|
| Warnings
| --------
| This method is deprecated. Use the new `polars.plugins.register_plugin_function`
| function instead.
|
| This is highly unsafe as this will call the C function loaded by
| `lib::symbol`.
|
| The parameters you set dictate how Polars will handle the function.
| Make sure they are correct!
|
| Parameters
| ----------
| lib
| Library to load.
| symbol
| Function to load.
| args
| Arguments (other than self) passed to this function.
| These arguments have to be of type Expression.
| kwargs
| Non-expression arguments. They must be JSON serializable.
| is_elementwise
| If the function only operates on scalars
| this will trigger fast paths.
| input_wildcard_expansion
| Expand expressions as input of this function.
| returns_scalar
| Automatically explode on unit length if it ran as final aggregation.
| this is the case for aggregations like `sum`, `min`, `covariance` etc.
| cast_to_supertypes
| Cast the input datatypes to their supertype.
| pass_name_to_apply
| if set, then the `Series` passed to the function in the group_by operation
| will ensure the name is set. This is an extra heap allocation per group.
| changes_length
| For example a `unique` or a `slice`
|
| reinterpret(self, *, signed: 'bool | None' = None, dtype: 'PolarsDataType | None' = None) -> 'Expr'
| Reinterpret the underlying bits as a signed/unsigned integer or float.
|
| This operation is only allowed for numeric types of the same size.
| For lower bits numbers, you can safely use the cast operation.
|
| Either `signed` or `dtype` can be specified.
| Defaults to `signed=True` otherwise.
|
| Parameters
| ----------
| signed
| If True, reinterpret as signed integer. Otherwise, reinterpret
| as unsigned integer.
| dtype
| DataType to reinterpret to.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 1, 2], dtype=pl.UInt64)
| >>> df = pl.DataFrame([s])
| >>> df.select(
| ... [
| ... pl.col("a").reinterpret(dtype=pl.Int64).alias("reinterpreted"),
| ... pl.col("a").alias("original"),
| ... ]
| ... )
| shape: (3, 2)
| ┌───────────────┬──────────┐
| │ reinterpreted ┆ original │
| │ --- ┆ --- │
| │ i64 ┆ u64 │
| ╞═══════════════╪══════════╡
| │ 1 ┆ 1 │
| │ 1 ┆ 1 │
| │ 2 ┆ 2 │
| └───────────────┴──────────┘
|
| repeat_by(self, by: 'pl.Series | Expr | str_ | int') -> 'Expr'
| Repeat the elements in this Series as specified in the given expression.
|
| The repeated elements are expanded into a `List`.
|
| Parameters
| ----------
| by
| Numeric column that determines how often the values will be repeated.
| The column will be coerced to UInt32. Give this dtype to make the coercion a
| no-op.
|
| Returns
| -------
| Expr
| Expression of data type :class:`List`, where the inner data type is equal
| to the original data type.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["x", "y", "z"],
| ... "n": [1, 2, 3],
| ... }
| ... )
| >>> df.select(pl.col("a").repeat_by("n"))
| shape: (3, 1)
| ┌─────────────────┐
| │ a │
| │ --- │
| │ list[str] │
| ╞═════════════════╡
| │ ["x"] │
| │ ["y", "y"] │
| │ ["z", "z", "z"] │
| └─────────────────┘
|
| replace(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | None' = None) -> 'Expr'
| Replace the given values by different values of the same data type.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Length must match the length of `old` or have length 1.
|
| default
| Set values that were not replaced to this value.
| Defaults to keeping the original value.
| Accepts expression input. Non-expression inputs are parsed as literals.
|
| .. deprecated:: 1.0.0
| Use :meth:`replace_strict` instead to set a default while replacing
| values.
|
| return_dtype
| The data type of the resulting expression. If set to `None` (default),
| the data type of the original column is preserved.
|
| .. deprecated:: 1.0.0
| Use :meth:`replace_strict` instead to set a return data type while
| replacing values, or explicitly call :meth:`cast` on the output.
|
| See Also
| --------
| replace_strict
| str.replace
|
| Examples
| --------
| Replace a single value by another value. Values that were not replaced remain
| unchanged.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3]})
| >>> df.with_columns(replaced=pl.col("a").replace(2, 100))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 100 │
| │ 2 ┆ 100 │
| │ 3 ┆ 3 │
| └─────┴──────────┘
|
| Replace multiple values by passing sequences to the `old` and `new` parameters.
|
| >>> df.with_columns(replaced=pl.col("a").replace([2, 3], [100, 200]))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 100 │
| │ 2 ┆ 100 │
| │ 3 ┆ 200 │
| └─────┴──────────┘
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {2: 100, 3: 200}
| >>> df.with_columns(replaced=pl.col("a").replace(mapping))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 100 │
| │ 2 ┆ 100 │
| │ 3 ┆ 200 │
| └─────┴──────────┘
|
| The original data type is preserved when replacing by values of a different
| data type. Use :meth:`replace_strict` to replace and change the return data
| type.
|
| >>> df = pl.DataFrame({"a": ["x", "y", "z"]})
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> df.with_columns(replaced=pl.col("a").replace(mapping))
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
|
| Expression input is supported.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3], "b": [1.5, 2.5, 5.0, 1.0]})
| >>> df.with_columns(
| ... replaced=pl.col("a").replace(
| ... old=pl.col("a").max(),
| ... new=pl.col("b").sum(),
| ... )
| ... )
| shape: (4, 3)
| ┌─────┬─────┬──────────┐
| │ a ┆ b ┆ replaced │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ i64 │
| ╞═════╪═════╪══════════╡
| │ 1 ┆ 1.5 ┆ 1 │
| │ 2 ┆ 2.5 ┆ 2 │
| │ 2 ┆ 5.0 ┆ 2 │
| │ 3 ┆ 1.0 ┆ 10 │
| └─────┴─────┴──────────┘
|
| replace_strict(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None) -> 'Expr'
| Replace all values by different values.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace_strict(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Length must match the length of `old` or have length 1.
| default
| Set values that were not replaced to this value. If no default is specified,
| (default), an error is raised if any values were not replaced.
| Accepts expression input. Non-expression inputs are parsed as literals.
| return_dtype
| The data type of the resulting expression. If set to `None` (default),
| the data type is determined automatically based on the other inputs.
|
| Raises
| ------
| InvalidOperationError
| If any non-null values in the original column were not replaced, and no
| `default` was specified.
|
| See Also
| --------
| replace
| str.replace
|
| Examples
| --------
| Replace values by passing sequences to the `old` and `new` parameters.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3]})
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict([1, 2, 3], [100, 200, 300])
| ... )
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 100 │
| │ 2 ┆ 200 │
| │ 2 ┆ 200 │
| │ 3 ┆ 300 │
| └─────┴──────────┘
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {1: 100, 2: 200, 3: 300}
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 100 │
| │ 2 ┆ 200 │
| │ 2 ┆ 200 │
| │ 3 ┆ 300 │
| └─────┴──────────┘
|
| By default, an error is raised if any non-null values were not replaced.
| Specify a default to set all values that were not matched.
|
| >>> mapping = {2: 200, 3: 300}
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict(mapping)
| ... ) # doctest: +SKIP
| Traceback (most recent call last):
| ...
| polars.exceptions.InvalidOperationError: incomplete mapping specified for `replace_strict`
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping, default=-1))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ -1 │
| │ 2 ┆ 200 │
| │ 2 ┆ 200 │
| │ 3 ┆ 300 │
| └─────┴──────────┘
|
| Replacing by values of a different data type sets the return type based on
| a combination of the `new` data type and the `default` data type.
|
| >>> df = pl.DataFrame({"a": ["x", "y", "z"]})
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping))
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping, default="x"))
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
|
| Set the `return_dtype` parameter to control the resulting data type directly.
|
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict(mapping, return_dtype=pl.UInt8)
| ... )
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ u8 │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
|
| Expression input is supported for all parameters.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3], "b": [1.5, 2.5, 5.0, 1.0]})
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict(
| ... old=pl.col("a").max(),
| ... new=pl.col("b").sum(),
| ... default=pl.col("b"),
| ... )
| ... )
| shape: (4, 3)
| ┌─────┬─────┬──────────┐
| │ a ┆ b ┆ replaced │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 │
| ╞═════╪═════╪══════════╡
| │ 1 ┆ 1.5 ┆ 1.5 │
| │ 2 ┆ 2.5 ┆ 2.5 │
| │ 2 ┆ 5.0 ┆ 5.0 │
| │ 3 ┆ 1.0 ┆ 10.0 │
| └─────┴─────┴──────────┘
|
| reshape(self, dimensions: 'tuple[int, ...]') -> 'Expr'
| Reshape this Expr to a flat column or an Array column.
|
| Parameters
| ----------
| dimensions
| Tuple of the dimension sizes. If -1 is used as the value for the
| first dimension, that dimension is inferred.
| Because the size of the Column may not be known in advance, it is
| only possible to use -1 for the first dimension.
|
| Returns
| -------
| Expr
| If a single dimension is given, results in an expression of the original
| data type.
| If a multiple dimensions are given, results in an expression of data type
| :class:`Array` with shape `dimensions`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7, 8, 9]})
| >>> square = df.select(pl.col("foo").reshape((3, 3)))
| >>> square
| shape: (3, 1)
| ┌───────────────┐
| │ foo │
| │ --- │
| │ array[i64, 3] │
| ╞═══════════════╡
| │ [1, 2, 3] │
| │ [4, 5, 6] │
| │ [7, 8, 9] │
| └───────────────┘
| >>> square.select(pl.col("foo").reshape((9,)))
| shape: (9, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| │ 4 │
| │ 5 │
| │ 6 │
| │ 7 │
| │ 8 │
| │ 9 │
| └─────┘
|
| See Also
| --------
| Expr.list.explode : Explode a list column.
|
| reverse(self) -> 'Expr'
| Reverse the selection.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1, 2, 3, 4, 5],
| ... "fruits": ["banana", "banana", "apple", "apple", "banana"],
| ... "B": [5, 4, 3, 2, 1],
| ... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"],
| ... }
| ... )
| >>> df.select(
| ... [
| ... pl.all(),
| ... pl.all().reverse().name.suffix("_reverse"),
| ... ]
| ... )
| shape: (5, 8)
| ┌─────┬────────┬─────┬────────┬───────────┬────────────────┬───────────┬──────────────┐
| │ A ┆ fruits ┆ B ┆ cars ┆ A_reverse ┆ fruits_reverse ┆ B_reverse ┆ cars_reverse │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ i64 ┆ str ┆ i64 ┆ str ┆ i64 ┆ str │
| ╞═════╪════════╪═════╪════════╪═══════════╪════════════════╪═══════════╪══════════════╡
| │ 1 ┆ banana ┆ 5 ┆ beetle ┆ 5 ┆ banana ┆ 1 ┆ beetle │
| │ 2 ┆ banana ┆ 4 ┆ audi ┆ 4 ┆ apple ┆ 2 ┆ beetle │
| │ 3 ┆ apple ┆ 3 ┆ beetle ┆ 3 ┆ apple ┆ 3 ┆ beetle │
| │ 4 ┆ apple ┆ 2 ┆ beetle ┆ 2 ┆ banana ┆ 4 ┆ audi │
| │ 5 ┆ banana ┆ 1 ┆ beetle ┆ 1 ┆ banana ┆ 5 ┆ beetle │
| └─────┴────────┴─────┴────────┴───────────┴────────────────┴───────────┴──────────────┘
|
| rle(self) -> 'Expr'
| Compress the column data using run-length encoding.
|
| Run-length encoding (RLE) encodes data by storing each *run* of identical values
| as a single value and its length.
|
| Returns
| -------
| Expr
| Expression of data type `Struct` with fields `len` of data type `UInt32`
| and `value` of the original data type.
|
| See Also
| --------
| rle_id
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 1, None, 1, 3, 3]})
| >>> df.select(pl.col("a").rle()).unnest("a")
| shape: (6, 2)
| ┌─────┬───────┐
| │ len ┆ value │
| │ --- ┆ --- │
| │ u32 ┆ i64 │
| ╞═════╪═══════╡
| │ 2 ┆ 1 │
| │ 1 ┆ 2 │
| │ 1 ┆ 1 │
| │ 1 ┆ null │
| │ 1 ┆ 1 │
| │ 2 ┆ 3 │
| └─────┴───────┘
|
| rle_id(self) -> 'Expr'
| Get a distinct integer ID for each run of identical values.
|
| The ID starts at 0 and increases by one each time the value of the column
| changes.
|
| Returns
| -------
| Expr
| Expression of data type `UInt32`.
|
| See Also
| --------
| rle
|
| Notes
| -----
| This functionality is especially useful for defining a new group for every time
| a column's value changes, rather than for every distinct value of that column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 1, 1, 1],
| ... "b": ["x", "x", None, "y", "y"],
| ... }
| ... )
| >>> df.with_columns(
| ... rle_id_a=pl.col("a").rle_id(),
| ... rle_id_ab=pl.struct("a", "b").rle_id(),
| ... )
| shape: (5, 4)
| ┌─────┬──────┬──────────┬───────────┐
| │ a ┆ b ┆ rle_id_a ┆ rle_id_ab │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ u32 ┆ u32 │
| ╞═════╪══════╪══════════╪═══════════╡
| │ 1 ┆ x ┆ 0 ┆ 0 │
| │ 2 ┆ x ┆ 1 ┆ 1 │
| │ 1 ┆ null ┆ 2 ┆ 2 │
| │ 1 ┆ y ┆ 2 ┆ 3 │
| │ 1 ┆ y ┆ 2 ┆ 3 │
| └─────┴──────┴──────────┴───────────┘
|
| rolling(self, index_column: 'IntoExprColumn', *, period: 'str_ | timedelta', offset: 'str_ | timedelta | None' = None, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Create rolling groups based on a temporal or integer column.
|
| If you have a time series `<t_0, t_1, ..., t_n>`, then by default the
| windows created will be
|
| * (t_0 - period, t_0]
| * (t_1 - period, t_1]
| * ...
| * (t_n - period, t_n]
|
| whereas if you pass a non-default `offset`, then the windows will be
|
| * (t_0 + offset, t_0 + offset + period]
| * (t_1 + offset, t_1 + offset + period]
| * ...
| * (t_n + offset, t_n + offset + period]
|
| The `period` and `offset` arguments are created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order.
| In case of a rolling group by on indices, dtype needs to be one of
| {UInt32, UInt64, Int32, Int64}. Note that the first three get temporarily
| cast to Int64, so if performance matters use an Int64 column.
| period
| Length of the window - must be non-negative.
| offset
| Offset of the window. Default is `-period`.
| closed : {'right', 'left', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
|
| Examples
| --------
| >>> dates = [
| ... "2020-01-01 13:45:48",
| ... "2020-01-01 16:42:13",
| ... "2020-01-01 16:45:09",
| ... "2020-01-02 18:12:48",
| ... "2020-01-03 19:45:32",
| ... "2020-01-08 23:16:43",
| ... ]
| >>> df = pl.DataFrame({"dt": dates, "a": [3, 7, 5, 9, 2, 1]}).with_columns(
| ... pl.col("dt").str.strptime(pl.Datetime).set_sorted()
| ... )
| >>> df.with_columns(
| ... sum_a=pl.sum("a").rolling(index_column="dt", period="2d"),
| ... min_a=pl.min("a").rolling(index_column="dt", period="2d"),
| ... max_a=pl.max("a").rolling(index_column="dt", period="2d"),
| ... )
| shape: (6, 5)
| ┌─────────────────────┬─────┬───────┬───────┬───────┐
| │ dt ┆ a ┆ sum_a ┆ min_a ┆ max_a │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════════════════════╪═════╪═══════╪═══════╪═══════╡
| │ 2020-01-01 13:45:48 ┆ 3 ┆ 3 ┆ 3 ┆ 3 │
| │ 2020-01-01 16:42:13 ┆ 7 ┆ 10 ┆ 3 ┆ 7 │
| │ 2020-01-01 16:45:09 ┆ 5 ┆ 15 ┆ 3 ┆ 7 │
| │ 2020-01-02 18:12:48 ┆ 9 ┆ 24 ┆ 3 ┆ 9 │
| │ 2020-01-03 19:45:32 ┆ 2 ┆ 11 ┆ 2 ┆ 9 │
| │ 2020-01-08 23:16:43 ┆ 1 ┆ 1 ┆ 1 ┆ 1 │
| └─────────────────────┴─────┴───────┴───────┴───────┘
|
| rolling_kurtosis(self, window_size: 'int', *, fisher: 'bool' = True, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling kurtosis.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Expr.kurtosis
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 4, 2, 9]})
| >>> df.select(pl.col("a").rolling_kurtosis(3))
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ null │
| │ -1.5 │
| │ -1.5 │
| └──────┘
|
| rolling_map(self, function: 'Callable[[Series], Any]', window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a custom rolling window function.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| function
| Custom aggregation function.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Warnings
| --------
| Computing custom functions is extremely slow. Use specialized rolling
| functions such as :func:`Expr.rolling_sum` if at all possible.
|
| Examples
| --------
| >>> from numpy import nansum
| >>> df = pl.DataFrame({"a": [11.0, 2.0, 9.0, float("nan"), 8.0]})
| >>> df.select(pl.col("a").rolling_map(nansum, window_size=3))
| shape: (5, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ null │
| │ 22.0 │
| │ 11.0 │
| │ 17.0 │
| └──────┘
|
| rolling_max(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling max (moving max) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their max.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_max=pl.col("A").rolling_max(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_max │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 2.0 │
| │ 3.0 ┆ 3.0 │
| │ 4.0 ┆ 4.0 │
| │ 5.0 ┆ 5.0 │
| │ 6.0 ┆ 6.0 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_max=pl.col("A").rolling_max(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_max │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.25 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ 3.75 │
| │ 6.0 ┆ 4.5 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_max=pl.col("A").rolling_max(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_max │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 3.0 │
| │ 3.0 ┆ 4.0 │
| │ 4.0 ┆ 5.0 │
| │ 5.0 ┆ 6.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_max_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling max based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling max with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_max=pl.col("index").rolling_max_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_max │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 2 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 3 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 4 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 20 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 21 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 22 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 23 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 24 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling max with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_max=pl.col("index").rolling_max_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_max │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 2 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 3 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 4 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 20 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 21 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 22 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 23 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 24 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_mean(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling mean (moving mean) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their mean. Weights
| are normalized to sum to 1.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window, after being normalized to sum to
| 1.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_mean=pl.col("A").rolling_mean(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────┐
| │ A ┆ rolling_mean │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.5 │
| │ 4.0 ┆ 3.5 │
| │ 5.0 ┆ 4.5 │
| │ 6.0 ┆ 5.5 │
| └─────┴──────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_mean=pl.col("A").rolling_mean(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────┐
| │ A ┆ rolling_mean │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.75 │
| │ 3.0 ┆ 2.75 │
| │ 4.0 ┆ 3.75 │
| │ 5.0 ┆ 4.75 │
| │ 6.0 ┆ 5.75 │
| └─────┴──────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_mean=pl.col("A").rolling_mean(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────┐
| │ A ┆ rolling_mean │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 2.0 │
| │ 3.0 ┆ 3.0 │
| │ 4.0 ┆ 4.0 │
| │ 5.0 ┆ 5.0 │
| │ 6.0 ┆ null │
| └─────┴──────────────┘
|
| rolling_mean_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling mean based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling mean with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_mean=pl.col("index").rolling_mean_by(
| ... "date", window_size="2h"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬──────────────────┐
| │ index ┆ date ┆ rolling_row_mean │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪══════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.5 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.5 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.5 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.5 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.5 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.5 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.5 │
| └───────┴─────────────────────┴──────────────────┘
|
| Compute the rolling mean with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_mean=pl.col("index").rolling_mean_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬──────────────────┐
| │ index ┆ date ┆ rolling_row_mean │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪══════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.0 │
| └───────┴─────────────────────┴──────────────────┘
|
| rolling_median(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling median.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their median.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_median=pl.col("A").rolling_median(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬────────────────┐
| │ A ┆ rolling_median │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.5 │
| │ 4.0 ┆ 3.5 │
| │ 5.0 ┆ 4.5 │
| │ 6.0 ┆ 5.5 │
| └─────┴────────────────┘
|
| Specify weights for the values in each window:
|
| >>> df.with_columns(
| ... rolling_median=pl.col("A").rolling_median(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬────────────────┐
| │ A ┆ rolling_median │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.5 │
| │ 4.0 ┆ 3.5 │
| │ 5.0 ┆ 4.5 │
| │ 6.0 ┆ 5.5 │
| └─────┴────────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_median=pl.col("A").rolling_median(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬────────────────┐
| │ A ┆ rolling_median │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 2.0 │
| │ 3.0 ┆ 3.0 │
| │ 4.0 ┆ 4.0 │
| │ 5.0 ┆ 5.0 │
| │ 6.0 ┆ null │
| └─────┴────────────────┘
|
| rolling_median_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Compute a rolling median based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling median with the temporal windows closed on the right:
|
| >>> df_temporal.with_columns(
| ... rolling_row_median=pl.col("index").rolling_median_by(
| ... "date", window_size="2h"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬────────────────────┐
| │ index ┆ date ┆ rolling_row_median │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.5 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.5 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.5 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.5 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.5 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.5 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.5 │
| └───────┴─────────────────────┴────────────────────┘
|
| rolling_min(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling min (moving min) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their min.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_min=pl.col("A").rolling_min(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_min │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 2.0 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ 4.0 │
| │ 6.0 ┆ 5.0 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_min=pl.col("A").rolling_min(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_min │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.25 │
| │ 3.0 ┆ 0.5 │
| │ 4.0 ┆ 0.75 │
| │ 5.0 ┆ 1.0 │
| │ 6.0 ┆ 1.25 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_min=pl.col("A").rolling_min(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_min │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 2.0 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ 4.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_min_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling min based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling min with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_min=pl.col("index").rolling_min_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_min │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_quantile(self, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', window_size: 'int' = 2, weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling quantile.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their quantile.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.25, window_size=4
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ null │
| │ 4.0 ┆ 2.0 │
| │ 5.0 ┆ 3.0 │
| │ 6.0 ┆ 4.0 │
| └─────┴──────────────────┘
|
| Specify weights for the values in each window:
|
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.25, window_size=4, weights=[0.2, 0.4, 0.4, 0.2]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ null │
| │ 4.0 ┆ 2.0 │
| │ 5.0 ┆ 3.0 │
| │ 6.0 ┆ 4.0 │
| └─────┴──────────────────┘
|
| Specify weights and interpolation method
|
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.25,
| ... window_size=4,
| ... weights=[0.2, 0.4, 0.4, 0.2],
| ... interpolation="linear",
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ null │
| │ 4.0 ┆ 1.625 │
| │ 5.0 ┆ 2.625 │
| │ 6.0 ┆ 3.625 │
| └─────┴──────────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.2, window_size=5, center=True
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ 2.0 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ null │
| │ 6.0 ┆ null │
| └─────┴──────────────────┘
|
| rolling_quantile_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Compute a rolling quantile based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling quantile with the temporal windows closed on the right:
|
| >>> df_temporal.with_columns(
| ... rolling_row_quantile=pl.col("index").rolling_quantile_by(
| ... "date", window_size="2h", quantile=0.3
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬──────────────────────┐
| │ index ┆ date ┆ rolling_row_quantile │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪══════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.0 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.0 │
| └───────┴─────────────────────┴──────────────────────┘
|
| rolling_rank(self, window_size: 'int', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling rank.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| A window of length `window_size` will traverse the array. The values
| that fill this window will be ranked according to the `method`
| parameter. The resulting values will be the rank of the value that is
| at the end of the sliding window.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Returns
| -------
| Expr
| An Expr of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 4, 4, 1, 9]})
| >>> df.select(pl.col("a").rolling_rank(3, method="average"))
| shape: (5, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ null │
| │ 2.5 │
| │ 1.0 │
| │ 3.0 │
| └──────┘
|
| rolling_rank_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Compute a rolling rank based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Returns
| -------
| Expr
| An Expr of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| rolling_skew(self, window_size: 'int', *, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling skew.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| bias
| If False, the calculations are corrected for statistical bias.
| bias: bool = True,
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Expr.skew
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 4, 2, 9]})
| >>> df.select(pl.col("a").rolling_skew(3))
| shape: (4, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ null │
| │ null │
| │ 0.381802 │
| │ 0.47033 │
| └──────────┘
|
| Note how the values match the following:
|
| >>> pl.Series([1, 4, 2]).skew(), pl.Series([4, 2, 9]).skew()
| (0.38180177416060584, 0.47033046033698594)
|
| rolling_std(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Expr'
| Compute a rolling standard deviation.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their std. Weights
| are normalized to sum to 1.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window after being normalized to sum to
| 1.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_std=pl.col("A").rolling_std(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_std │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.707107 │
| │ 3.0 ┆ 0.707107 │
| │ 4.0 ┆ 0.707107 │
| │ 5.0 ┆ 0.707107 │
| │ 6.0 ┆ 0.707107 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_std=pl.col("A").rolling_std(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_std │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.433013 │
| │ 3.0 ┆ 0.433013 │
| │ 4.0 ┆ 0.433013 │
| │ 5.0 ┆ 0.433013 │
| │ 6.0 ┆ 0.433013 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_std=pl.col("A").rolling_std(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_std │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 1.0 │
| │ 4.0 ┆ 1.0 │
| │ 5.0 ┆ 1.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_std_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Expr'
| Compute a rolling standard deviation based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling std with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_std=pl.col("index").rolling_std_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_std │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.707107 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 0.707107 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 0.707107 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 0.707107 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 0.707107 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 0.707107 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 0.707107 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 0.707107 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 0.707107 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling std with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_std=pl.col("index").rolling_std_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_std │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.707107 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 1.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 1.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 1.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 1.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 1.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 1.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 1.0 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_sum(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling sum (moving sum) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their sum.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_sum=pl.col("A").rolling_sum(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_sum │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 3.0 │
| │ 3.0 ┆ 5.0 │
| │ 4.0 ┆ 7.0 │
| │ 5.0 ┆ 9.0 │
| │ 6.0 ┆ 11.0 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_sum=pl.col("A").rolling_sum(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_sum │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.75 │
| │ 3.0 ┆ 2.75 │
| │ 4.0 ┆ 3.75 │
| │ 5.0 ┆ 4.75 │
| │ 6.0 ┆ 5.75 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_sum=pl.col("A").rolling_sum(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_sum │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 6.0 │
| │ 3.0 ┆ 9.0 │
| │ 4.0 ┆ 12.0 │
| │ 5.0 ┆ 15.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_sum_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 0, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling sum based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling sum with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_sum=pl.col("index").rolling_sum_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_sum │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 3 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 7 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 39 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 41 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 43 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 45 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 47 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling sum with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_sum=pl.col("index").rolling_sum_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_sum │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 3 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 6 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 9 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 57 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 60 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 63 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 66 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 69 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_var(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Expr'
| Compute a rolling variance.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their var. Weights
| are normalized to sum to 1.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window after being normalized to sum to
| 1.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_var=pl.col("A").rolling_var(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_var │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.5 │
| │ 3.0 ┆ 0.5 │
| │ 4.0 ┆ 0.5 │
| │ 5.0 ┆ 0.5 │
| │ 6.0 ┆ 0.5 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_var=pl.col("A").rolling_var(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_var │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.1875 │
| │ 3.0 ┆ 0.1875 │
| │ 4.0 ┆ 0.1875 │
| │ 5.0 ┆ 0.1875 │
| │ 6.0 ┆ 0.1875 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_var=pl.col("A").rolling_var(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_var │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 1.0 │
| │ 4.0 ┆ 1.0 │
| │ 5.0 ┆ 1.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_var_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Expr'
| Compute a rolling variance based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling var with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_var=pl.col("index").rolling_var_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_var │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 0.5 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 0.5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 0.5 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 0.5 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 0.5 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 0.5 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 0.5 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 0.5 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling var with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_var=pl.col("index").rolling_var_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_var │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 1.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 1.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 1.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 1.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 1.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 1.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 1.0 │
| └───────┴─────────────────────┴─────────────────┘
|
| round(self, decimals: 'int' = 0, mode: 'RoundMode' = 'half_to_even') -> 'Expr'
| Round underlying floating point data by `decimals` digits.
|
| Parameters
| ----------
| decimals
| Number of decimals to round by.
| mode : {'half_to_even', 'half_away_from_zero', 'to_zero'}
| The rounding strategy used. A "rounded value" is a value with at most
| `decimals` decimal places (e.g. integers when ``decimals=0``, multiples
| of 0.1 when ``decimals=1``, 0.01 when ``decimals=2``, and so on).
|
| Strategies that start with ``half_`` round all values to the *nearest*
| rounded value, only using the strategy to break ties when a value falls
| exactly between two rounded values (e.g. 0.5 when ``decimals=0``, 0.05
| when ``decimals=1``). Other rounding strategies specify explicitly
| which rounded value is chosen and always apply (not just for tiebreaks).
|
| * *half_to_even* (default)
| Round to the nearest value; break ties by choosing the nearest
| **even** value. For example, 0.5 rounds to 0, 1.5 rounds to 2,
| 2.5 rounds to 2. Also known as "banker's rounding"; this is the
| default because it tends to minimise cumulative rounding bias.
| * *half_away_from_zero*
| Round to the nearest value; break ties by rounding **away from
| zero**. For example, 0.5 rounds to 1, -0.5 rounds to -1, 2.5
| rounds to 3. Also known as "commercial rounding".
| * *to_zero*
| Always round (truncate) **towards zero**, discarding the fractional
| part beyond `decimals`. For example, 0.9 rounds to 0, -0.9 rounds
| to 0, 1.29 rounds to 1.2 (with ``decimals=1``). Equivalent to the
| :meth:`truncate` method.
|
| See Also
| --------
| ceil : Round up to the nearest integer.
| floor : Round down to the nearest integer.
| round_sig_figs : Round to a given number of significant figures.
| truncate : Truncate to a given number of decimals.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.33, 0.52, 1.02, 1.17]})
| >>> df.select(pl.col("a").round(1))
| shape: (4, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.3 │
| │ 0.5 │
| │ 1.0 │
| │ 1.2 │
| └─────┘
|
| >>> df = pl.DataFrame(
| ... {
| ... "f64": [-3.5, -2.5, -1.5, -0.5, 0.5, 1.5, 2.5, 3.5],
| ... "d": ["-3.5", "-2.5", "-1.5", "-0.5", "0.5", "1.5", "2.5", "3.5"],
| ... },
| ... schema_overrides={"d": pl.Decimal(scale=1)},
| ... )
| >>> df.with_columns(
| ... pl.all().round(mode="half_away_from_zero").name.suffix("_away"),
| ... pl.all().round(mode="half_to_even").name.suffix("_to_even"),
| ... )
| shape: (8, 6)
| ┌──────┬───────────────┬──────────┬───────────────┬─────────────┬───────────────┐
| │ f64 ┆ d ┆ f64_away ┆ d_away ┆ f64_to_even ┆ d_to_even │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ decimal[38,1] ┆ f64 ┆ decimal[38,1] ┆ f64 ┆ decimal[38,1] │
| ╞══════╪═══════════════╪══════════╪═══════════════╪═════════════╪═══════════════╡
| │ -3.5 ┆ -3.5 ┆ -4.0 ┆ -4.0 ┆ -4.0 ┆ -4.0 │
| │ -2.5 ┆ -2.5 ┆ -3.0 ┆ -3.0 ┆ -2.0 ┆ -2.0 │
| │ -1.5 ┆ -1.5 ┆ -2.0 ┆ -2.0 ┆ -2.0 ┆ -2.0 │
| │ -0.5 ┆ -0.5 ┆ -1.0 ┆ -1.0 ┆ -0.0 ┆ 0.0 │
| │ 0.5 ┆ 0.5 ┆ 1.0 ┆ 1.0 ┆ 0.0 ┆ 0.0 │
| │ 1.5 ┆ 1.5 ┆ 2.0 ┆ 2.0 ┆ 2.0 ┆ 2.0 │
| │ 2.5 ┆ 2.5 ┆ 3.0 ┆ 3.0 ┆ 2.0 ┆ 2.0 │
| │ 3.5 ┆ 3.5 ┆ 4.0 ┆ 4.0 ┆ 4.0 ┆ 4.0 │
| └──────┴───────────────┴──────────┴───────────────┴─────────────┴───────────────┘
|
| round_sig_figs(self, digits: 'int') -> 'Expr'
| Round to a number of significant figures.
|
| Parameters
| ----------
| digits
| Number of significant figures to round to.
|
| See Also
| --------
| ceil : Round up to the nearest integer.
| floor : Round down to the nearest integer.
| round : Round to a given number of decimals.
| truncate : Truncate to a given number of decimals.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.01234, 3.333, 1234.0]})
| >>> df.with_columns(pl.col("a").round_sig_figs(2).alias("round_sig_figs"))
| shape: (3, 2)
| ┌─────────┬────────────────┐
| │ a ┆ round_sig_figs │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════════╪════════════════╡
| │ 0.01234 ┆ 0.012 │
| │ 3.333 ┆ 3.3 │
| │ 1234.0 ┆ 1200.0 │
| └─────────┴────────────────┘
|
| sample(self, n: 'int | IntoExprColumn | None' = None, *, fraction: 'float | IntoExprColumn | None' = None, with_replacement: 'bool' = False, shuffle: 'bool | None' = None, seed: 'int | None' = None) -> 'Expr'
| Sample from this expression.
|
| Parameters
| ----------
| n
| Number of items to return. Cannot be used with `fraction`. Defaults to 1 if
| `fraction` is None.
| fraction
| Fraction of items to return. Cannot be used with `n`.
| with_replacement
| Allow values to be sampled more than once.
| shuffle
| Determines the order of the sampled elements.
| If True, sampled elements are explicitly shuffled.
| If False, the relative order of the sampled elements is preserved.
| (i.e. they appear in the same order as the original input).
| If None (default), no ordering guarantee; uses the most performant
| algorithm.
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated for each sample operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(
| ... pl.col("a").sample(
| ... fraction=1.0, with_replacement=True, shuffle=False, seed=1
| ... )
| ... )
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 3 │
| │ 3 │
| │ 1 │
| └─────┘
|
| search_sorted(self, element: 'IntoExpr | np.ndarray[Any, Any]', side: 'SearchSortedSide' = 'any', *, descending: 'bool' = False) -> 'Expr'
| Find indices where elements should be inserted to maintain order.
|
| .. math:: a[i-1] < v <= a[i]
|
| Parameters
| ----------
| element
| Expression or scalar value.
| side : {'any', 'left', 'right'}
| If 'any', the index of the first suitable location found is given.
| If 'left', the index of the leftmost suitable location found is given.
| If 'right', return the rightmost suitable location found is given.
| descending
| Boolean indicating whether the values are descending or not (they
| are required to be sorted either way).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "values": [1, 2, 3, 5],
| ... }
| ... )
| >>> df.select(
| ... [
| ... pl.col("values").search_sorted(0).alias("zero"),
| ... pl.col("values").search_sorted(3).alias("three"),
| ... pl.col("values").search_sorted(6).alias("six"),
| ... ]
| ... )
| shape: (1, 3)
| ┌──────┬───────┬─────┐
| │ zero ┆ three ┆ six │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞══════╪═══════╪═════╡
| │ 0 ┆ 2 ┆ 4 │
| └──────┴───────┴─────┘
|
| set_sorted(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Expr'
| Flags the expression as 'sorted'.
|
| Enables downstream code to user fast paths for sorted arrays. It is
| recommended to also set whether `nulls_last` is `True` or `False`, as
| this enables many internal optimizations.
|
| Parameters
| ----------
| descending
| Whether the `Series` order is descending.
| nulls_last
| Whether the nulls are at the end.
|
| Warnings
| --------
| This can lead to incorrect results if the data is NOT sorted!!
| Use with care!
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1, 2, 3]})
| >>> df.select(pl.col("values").set_sorted().max())
| shape: (1, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 3 │
| └────────┘
|
| shift(self, n: 'int | IntoExprColumn' = 1, *, fill_value: 'IntoExpr | None' = None) -> 'Expr'
| Shift values by the given number of indices.
|
| Parameters
| ----------
| n
| Number of indices to shift forward. If a negative value is passed, values
| are shifted in the opposite direction instead.
| fill_value
| Fill the resulting null values with this scalar value.
|
| Notes
| -----
| This method is similar to the `LAG` operation in SQL when the value for `n`
| is positive. With a negative value for `n`, it is similar to `LEAD`.
|
| See Also
| --------
| fill_null
|
| Examples
| --------
| By default, values are shifted forward by one index.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.with_columns(shift=pl.col("a").shift())
| shape: (4, 2)
| ┌─────┬───────┐
| │ a ┆ shift │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═══════╡
| │ 1 ┆ null │
| │ 2 ┆ 1 │
| │ 3 ┆ 2 │
| │ 4 ┆ 3 │
| └─────┴───────┘
|
| Pass a negative value to shift in the opposite direction instead.
|
| >>> df.with_columns(shift=pl.col("a").shift(-2))
| shape: (4, 2)
| ┌─────┬───────┐
| │ a ┆ shift │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═══════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| │ 3 ┆ null │
| │ 4 ┆ null │
| └─────┴───────┘
|
| Specify `fill_value` to fill the resulting null values.
|
| >>> df.with_columns(shift=pl.col("a").shift(-2, fill_value=100))
| shape: (4, 2)
| ┌─────┬───────┐
| │ a ┆ shift │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═══════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| │ 3 ┆ 100 │
| │ 4 ┆ 100 │
| └─────┴───────┘
|
| shrink_dtype(self) -> 'Expr'
| Shrink numeric columns to the minimal required datatype.
|
| Shrink to the dtype needed to fit the extrema of this [`Series`].
| This can be used to reduce memory pressure.
|
| .. versionchanged:: 1.33.0
| Deprecated and turned into a no-op. The operation does not match the
| Polars data-model during lazy execution since the output datatype
| cannot be known without inspecting the data.
|
| Use `Series.shrink_dtype` instead.
|
| Examples
| --------
| >>> pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": [1, 2, 2 << 32],
| ... "c": [-1, 2, 1 << 30],
| ... "d": [-112, 2, 112],
| ... "e": [-112, 2, 129],
| ... "f": ["a", "b", "c"],
| ... "g": [0.1, 1.32, 0.12],
| ... "h": [True, None, False],
| ... }
| ... ).select(pl.all().shrink_dtype()) # doctest: +SKIP
| shape: (3, 8)
| ┌─────┬────────────┬────────────┬──────┬──────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ d ┆ e ┆ f ┆ g ┆ h │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i8 ┆ i64 ┆ i32 ┆ i8 ┆ i16 ┆ str ┆ f32 ┆ bool │
| ╞═════╪════════════╪════════════╪══════╪══════╪═════╪══════╪═══════╡
| │ 1 ┆ 1 ┆ -1 ┆ -112 ┆ -112 ┆ a ┆ 0.1 ┆ true │
| │ 2 ┆ 2 ┆ 2 ┆ 2 ┆ 2 ┆ b ┆ 1.32 ┆ null │
| │ 3 ┆ 8589934592 ┆ 1073741824 ┆ 112 ┆ 129 ┆ c ┆ 0.12 ┆ false │
| └─────┴────────────┴────────────┴──────┴──────┴─────┴──────┴───────┘
|
| shuffle(self, seed: 'int | None' = None) -> 'Expr'
| Shuffle the contents of this expression.
|
| Note this is shuffled independently of any other column or Expression. If you
| want each row to stay the same use df.sample(shuffle=True)
|
| Parameters
| ----------
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated each time the shuffle is called.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").shuffle(seed=1))
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 3 │
| │ 1 │
| └─────┘
|
| sign(self) -> 'Expr'
| Compute the element-wise sign function on numeric types.
|
| The returned value is computed as follows:
|
| * -1 if x < 0.
| * 1 if x > 0.
| * x otherwise (typically 0, but could be NaN if the input is).
|
| Null values are preserved as-is, and the dtype of the input is preserved.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-9.0, -0.0, 0.0, 4.0, float("nan"), None]})
| >>> df.select(pl.col.a.sign())
| shape: (6, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ -1.0 │
| │ -0.0 │
| │ 0.0 │
| │ 1.0 │
| │ NaN │
| │ null │
| └──────┘
|
| sin(self) -> 'Expr'
| Compute the element-wise value for the sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 2]})
| >>> df.select(pl.col("a").sin())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
| >>> df = pl.DataFrame({"a": [0.0, 90]})
| >>> df.select(pl.col("a").radians().sin())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
|
| sinh(self) -> 'Expr'
| Compute the element-wise value for the hyperbolic sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").sinh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.175201 │
| └──────────┘
|
| skew(self, *, bias: 'bool' = True) -> 'Expr'
| Compute the sample skewness of a data set.
|
| For normally distributed data, the skewness should be about zero. For
| unimodal continuous distributions, a skewness value greater than zero means
| that there is more weight in the right tail of the distribution. The
| function `skewtest` can be used to determine if the skewness value
| is close enough to zero, statistically speaking.
|
|
| See scipy.stats for more information.
|
| Parameters
| ----------
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Notes
| -----
| The sample skewness is computed as the Fisher-Pearson coefficient
| of skewness, i.e.
|
| .. math:: g_1=\frac{m_3}{m_2^{3/2}}
|
| where
|
| .. math:: m_i=\frac{1}{N}\sum_{n=1}^N(x[n]-\bar{x})^i
|
| is the biased sample :math:`i\texttt{th}` central moment, and
| :math:`\bar{x}` is
| the sample mean. If `bias` is False, the calculations are
| corrected for bias and the value computed is the adjusted
| Fisher-Pearson standardized moment coefficient, i.e.
|
| .. math::
| G_1 = \frac{k_3}{k_2^{3/2}} = \frac{\sqrt{N(N-1)}}{N-2}\frac{m_3}{m_2^{3/2}}
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").skew())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.343622 │
| └──────────┘
|
| slice(self, offset: 'int | Expr', length: 'int | Expr | None' = None) -> 'Expr'
| Get a slice of this expression.
|
| Parameters
| ----------
| offset
| Start index. Negative indexing is supported.
| length
| Length of the slice. If set to `None`, all rows starting at the offset
| will be selected.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [8, 9, 10, 11],
| ... "b": [None, 4, 4, 4],
| ... }
| ... )
| >>> df.select(pl.all().slice(1, 2))
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 9 ┆ 4 │
| │ 10 ┆ 4 │
| └─────┴─────┘
|
| sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Expr'
| Sort this column.
|
| When used in a projection/selection context, the whole column is sorted.
| When used in a group by context, the groups are sorted.
|
| Parameters
| ----------
| descending
| Sort in descending order.
| nulls_last
| Place null values last.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, None, 3, 2],
| ... }
| ... )
| >>> df.select(pl.col("a").sort())
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| │ 1 │
| │ 2 │
| │ 3 │
| └──────┘
| >>> df.select(pl.col("a").sort(descending=True))
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| │ 3 │
| │ 2 │
| │ 1 │
| └──────┘
| >>> df.select(pl.col("a").sort(nulls_last=True))
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 1 │
| │ 2 │
| │ 3 │
| │ null │
| └──────┘
|
| When sorting in a group by context, the groups are sorted.
|
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["one", "one", "one", "two", "two", "two"],
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.group_by("group").agg(pl.col("value").sort()) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌───────┬────────────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪════════════╡
| │ two ┆ [3, 4, 99] │
| │ one ┆ [1, 2, 98] │
| └───────┴────────────┘
|
| sort_by(self, by: 'IntoExpr | Iterable[IntoExpr]', *more_by: 'IntoExpr', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False, multithreaded: 'bool' = True, maintain_order: 'bool' = False) -> 'Expr'
| Sort this column by the ordering of other columns.
|
| When used in a projection/selection context, the whole column is sorted.
| When used in a group by context, the groups are sorted.
|
| Parameters
| ----------
| by
| Column(s) to sort by. Accepts expression input. Strings are parsed as column
| names.
| *more_by
| Additional columns to sort by, specified as positional arguments.
| descending
| Sort in descending order. When sorting by multiple columns, can be specified
| per column by passing a sequence of booleans.
| nulls_last
| Place null values last; can specify a single boolean applying to all columns
| or a sequence of booleans for per-column control.
| multithreaded
| Sort using multiple threads.
| maintain_order
| Whether the order should be maintained if elements are equal.
|
| Examples
| --------
| Pass a single column name to sort by that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["a", "a", "b", "b"],
| ... "value1": [1, 3, 4, 2],
| ... "value2": [8, 7, 6, 5],
| ... }
| ... )
| >>> df.select(pl.col("group").sort_by("value1"))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ a │
| │ b │
| │ a │
| │ b │
| └───────┘
|
| Sorting by expressions is also supported.
|
| >>> df.select(pl.col("group").sort_by(pl.col("value1") + pl.col("value2")))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ b │
| │ a │
| │ a │
| │ b │
| └───────┘
|
| Sort by multiple columns by passing a list of columns.
|
| >>> df.select(pl.col("group").sort_by(["value1", "value2"], descending=True))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ b │
| │ a │
| │ b │
| │ a │
| └───────┘
|
| Or use positional arguments to sort by multiple columns in the same way.
|
| >>> df.select(pl.col("group").sort_by("value1", "value2"))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ a │
| │ b │
| │ a │
| │ b │
| └───────┘
|
| When sorting in a group by context, the groups are sorted.
|
| >>> df.group_by("group").agg(
| ... pl.col("value1").sort_by("value2")
| ... ) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ value1 │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪═══════════╡
| │ a ┆ [3, 1] │
| │ b ┆ [2, 4] │
| └───────┴───────────┘
|
| Take a single row from each group where a column attains its minimal value
| within that group.
|
| >>> df.group_by("group").agg(
| ... pl.all().sort_by("value2").first()
| ... ) # doctest: +IGNORE_RESULT
| shape: (2, 3)
| ┌───────┬────────┬────────┐
| │ group ┆ value1 ┆ value2 |
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 |
| ╞═══════╪════════╪════════╡
| │ a ┆ 3 ┆ 7 |
| │ b ┆ 2 ┆ 5 |
| └───────┴────────┴────────┘
|
| sqrt(self) -> 'Expr'
| Compute the square root of the elements.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").sqrt())
| shape: (3, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 1.414214 │
| │ 2.0 │
| └──────────┘
|
| std(self, ddof: 'int' = 1) -> 'Expr'
| Get standard deviation.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").std())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
|
| sub(self, other: 'Any') -> 'Expr'
| Method equivalent of subtraction operator `expr - other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [0, 1, 2, 3, 4]})
| >>> df.with_columns(
| ... pl.col("x").sub(2).alias("x-2"),
| ... pl.col("x").sub(pl.col("x").cum_sum()).alias("x-expr"),
| ... )
| shape: (5, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ x-2 ┆ x-expr │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪════════╡
| │ 0 ┆ -2 ┆ 0 │
| │ 1 ┆ -1 ┆ 0 │
| │ 2 ┆ 0 ┆ -1 │
| │ 3 ┆ 1 ┆ -3 │
| │ 4 ┆ 2 ┆ -6 │
| └─────┴─────┴────────┘
|
| sum(self) -> 'Expr'
| Get sum value.
|
| Notes
| -----
| * Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
| * If there are no non-null values, then the output is `0`.
| If you would prefer empty sums to return `None`, you can
| use `pl.when(expr.count()>0).then(expr.sum())` instead
| of `expr.sum()`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").sum())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 0 │
| └─────┘
|
| tail(self, n: 'int | Expr' = 10) -> 'Expr'
| Get the last `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7]})
| >>> df.select(pl.col("foo").tail(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 5 │
| │ 6 │
| │ 7 │
| └─────┘
|
| tan(self) -> 'Expr'
| Compute the element-wise value for the tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 4]})
| >>> df.select(pl.col("a").tan())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
| >>> df = pl.DataFrame({"a": [0.0, 45]})
| >>> df.select(pl.col("a").radians().tan())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
|
| tanh(self) -> 'Expr'
| Compute the element-wise value for the hyperbolic tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").tanh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.761594 │
| └──────────┘
|
| to_physical(self) -> 'Expr'
| Cast to physical representation of the logical dtype.
|
| - :func:`polars.datatypes.Date` -> :func:`polars.datatypes.Int32`
| - :func:`polars.datatypes.Datetime` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Time` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Duration` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Categorical` -> :func:`polars.datatypes.UInt32`
| - `List(inner)` -> `List(physical of inner)`
| - `Array(inner)` -> `Struct(physical of inner)`
| - `Struct(fields)` -> `Array(physical of fields)`
|
| Other data types will be left unchanged.
|
| Warnings
| --------
| The physical representations are an implementation detail
| and not guaranteed to be stable.
|
| Examples
| --------
| Replicating the pandas
| `pd.factorize
| <https://pandas.pydata.org/docs/reference/api/pandas.factorize.html>`_
| function.
|
| >>> pl.DataFrame({"vals": ["a", "x", None, "a"]}).with_columns(
| ... pl.col("vals").cast(pl.Categorical),
| ... pl.col("vals")
| ... .cast(pl.Categorical)
| ... .to_physical()
| ... .alias("vals_physical"),
| ... )
| shape: (4, 2)
| ┌──────┬───────────────┐
| │ vals ┆ vals_physical │
| │ --- ┆ --- │
| │ cat ┆ u32 │
| ╞══════╪═══════════════╡
| │ a ┆ 0 │
| │ x ┆ 1 │
| │ null ┆ null │
| │ a ┆ 0 │
| └──────┴───────────────┘
|
| top_k(self, k: 'int | IntoExprColumn' = 5) -> 'Expr'
| Return the `k` largest elements.
|
| Non-null elements are always preferred over null elements. The output
| is not guaranteed to be in any particular order, call :func:`sort`
| after this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k_by
| bottom_k
| bottom_k_by
|
| Examples
| --------
| Get the 5 largest values in series.
|
| >>> df = pl.DataFrame({"value": [1, 98, 2, 3, 99, 4]})
| >>> df.select(
| ... pl.col("value").top_k().alias("top_k"),
| ... pl.col("value").bottom_k().alias("bottom_k"),
| ... )
| shape: (5, 2)
| ┌───────┬──────────┐
| │ top_k ┆ bottom_k │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪══════════╡
| │ 4 ┆ 1 │
| │ 98 ┆ 98 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| │ 99 ┆ 4 │
| └───────┴──────────┘
|
| top_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int | IntoExprColumn' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Expr'
| Return the elements corresponding to the `k` largest elements of the `by` column(s).
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed to `reverse`.
|
| Parameters
| ----------
| by
| Column(s) used to determine the largest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` smallest elements of the `by` column(s) (instead of the `k`
| largest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| bottom_k
| bottom_k_by
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [6, 5, 4, 3, 2, 1],
| ... "c": ["Apple", "Orange", "Apple", "Apple", "Banana", "Banana"],
| ... }
| ... )
| >>> df
| shape: (6, 3)
| ┌─────┬─────┬────────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪════════╡
| │ 1 ┆ 6 ┆ Apple │
| │ 2 ┆ 5 ┆ Orange │
| │ 3 ┆ 4 ┆ Apple │
| │ 4 ┆ 3 ┆ Apple │
| │ 5 ┆ 2 ┆ Banana │
| │ 6 ┆ 1 ┆ Banana │
| └─────┴─────┴────────┘
|
| Get the top 2 rows by column `a` or `b`.
|
| >>> df.select(
| ... pl.all().top_k_by("a", 2).name.suffix("_top_by_a"),
| ... pl.all().top_k_by("b", 2).name.suffix("_top_by_b"),
| ... )
| shape: (2, 6)
| ┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
| │ a_top_by_a ┆ b_top_by_a ┆ c_top_by_a ┆ a_top_by_b ┆ b_top_by_b ┆ c_top_by_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞════════════╪════════════╪════════════╪════════════╪════════════╪════════════╡
| │ 6 ┆ 1 ┆ Banana ┆ 1 ┆ 6 ┆ Apple │
| │ 5 ┆ 2 ┆ Banana ┆ 2 ┆ 5 ┆ Orange │
| └────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
|
| Get the top 2 rows by multiple columns with given order.
|
| >>> df.select(
| ... pl.all()
| ... .top_k_by(["c", "a"], 2, reverse=[False, True])
| ... .name.suffix("_by_ca"),
| ... pl.all()
| ... .top_k_by(["c", "b"], 2, reverse=[False, True])
| ... .name.suffix("_by_cb"),
| ... )
| shape: (2, 6)
| ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
| │ a_by_ca ┆ b_by_ca ┆ c_by_ca ┆ a_by_cb ┆ b_by_cb ┆ c_by_cb │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
| │ 2 ┆ 5 ┆ Orange ┆ 2 ┆ 5 ┆ Orange │
| │ 5 ┆ 2 ┆ Banana ┆ 6 ┆ 1 ┆ Banana │
| └─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
|
| Get the top 2 rows by column `a` in each group.
|
| >>> (
| ... df.group_by("c", maintain_order=True)
| ... .agg(pl.all().top_k_by("a", 2))
| ... .explode(pl.all().exclude("c"))
| ... )
| shape: (5, 3)
| ┌────────┬─────┬─────┐
| │ c ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ Apple ┆ 4 ┆ 3 │
| │ Apple ┆ 3 ┆ 4 │
| │ Orange ┆ 2 ┆ 5 │
| │ Banana ┆ 6 ┆ 1 │
| │ Banana ┆ 5 ┆ 2 │
| └────────┴─────┴─────┘
|
| truediv(self, other: 'Any') -> 'Expr'
| Method equivalent of float division operator `expr / other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Notes
| -----
| Zero-division behaviour follows IEEE-754:
|
| 0/0: Invalid operation - mathematically undefined, returns NaN.
| n/0: On finite operands gives an exact infinite result, eg: ±infinity.
|
| See Also
| --------
| floordiv
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={"x": [-2, -1, 0, 1, 2], "y": [0.5, 0.0, 0.0, -4.0, -0.5]}
| ... )
| >>> df.with_columns(
| ... pl.col("x").truediv(2).alias("x/2"),
| ... pl.col("x").truediv(pl.col("y")).alias("x/y"),
| ... )
| shape: (5, 4)
| ┌─────┬──────┬──────┬───────┐
| │ x ┆ y ┆ x/2 ┆ x/y │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 ┆ f64 │
| ╞═════╪══════╪══════╪═══════╡
| │ -2 ┆ 0.5 ┆ -1.0 ┆ -4.0 │
| │ -1 ┆ 0.0 ┆ -0.5 ┆ -inf │
| │ 0 ┆ 0.0 ┆ 0.0 ┆ NaN │
| │ 1 ┆ -4.0 ┆ 0.5 ┆ -0.25 │
| │ 2 ┆ -0.5 ┆ 1.0 ┆ -4.0 │
| └─────┴──────┴──────┴───────┘
|
| truncate(self, decimals: 'int' = 0) -> 'Expr'
| Truncate numeric data toward zero to `decimals` number of decimal places.
|
| Parameters
| ----------
| decimals
| Number of decimal places to truncate to.
|
| Notes
| -----
| * Truncation discards the fractional part beyond the given number of decimals.
| For example, when rounding to 0 decimals 0.25, -0.25, 0.99, and -0.99 will
| all round to 0. When rounding to 1 decimal 1.9999 rounds to 1.9 and -1.9999
| rounds to -1.9. There is no tiebreak behaviour at midpoint values as there
| is with :meth:`round` so 0.5 and -0.5 will also round to 0 when decimals=1.
|
| * This method performs numeric truncation. For truncating temporal
| data (dates/datetimes), use :func:`Expr.dt.truncate` instead.
|
| See Also
| --------
| ceil : Round up to the nearest integer.
| floor : Round down to the nearest integer.
| round : Round to a given number of decimals.
| round_sig_figs : Round to a given number of significant figures.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-9.9999, 0.12345, 1.0251, 8.8765]})
| >>> df.with_columns(
| ... t0=pl.col("n").truncate(0),
| ... t1=pl.col("n").truncate(1),
| ... t2=pl.col("n").truncate(2),
| ... t3=pl.col("n").truncate(3),
| ... t4=pl.col("n").truncate(4),
| ... )
| shape: (4, 6)
| ┌─────────┬──────┬──────┬───────┬────────┬─────────┐
| │ n ┆ t0 ┆ t1 ┆ t2 ┆ t3 ┆ t4 │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
| ╞═════════╪══════╪══════╪═══════╪════════╪═════════╡
| │ -9.9999 ┆ -9.0 ┆ -9.9 ┆ -9.99 ┆ -9.999 ┆ -9.9999 │
| │ 0.12345 ┆ 0.0 ┆ 0.1 ┆ 0.12 ┆ 0.123 ┆ 0.1234 │
| │ 1.0251 ┆ 1.0 ┆ 1.0 ┆ 1.02 ┆ 1.025 ┆ 1.025 │
| │ 8.8765 ┆ 8.0 ┆ 8.8 ┆ 8.87 ┆ 8.876 ┆ 8.8765 │
| └─────────┴──────┴──────┴───────┴────────┴─────────┘
|
| unique(self, *, maintain_order: 'bool' = False) -> 'Expr'
| Get unique values of this expression.
|
| `null` is considered to be a unique value for the purposes of this operation.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
| >>> df.select(pl.col("a").unique()) # doctest: +IGNORE_RESULT
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 1 │
| └─────┘
| >>> df.select(pl.col("a").unique(maintain_order=True))
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| └─────┘
|
| unique_counts(self) -> 'Expr'
| Return a count of the unique values in the order of appearance.
|
| This method differs from `value_counts` in that it does not return the
| values, only the counts and might be faster
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "id": ["a", "b", "b", "c", "c", "c"],
| ... }
| ... )
| >>> df.select(pl.col("id").unique_counts())
| shape: (3, 1)
| ┌─────┐
| │ id │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Note that `group_by` can be used to generate counts.
|
| >>> df.group_by("id", maintain_order=True).len().select("len")
| shape: (3, 1)
| ┌─────┐
| │ len │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| To add counts as a new column `pl.len()` can be used as a window function.
|
| >>> df.with_columns(pl.len().over("id"))
| shape: (6, 2)
| ┌─────┬─────┐
| │ id ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ b ┆ 2 │
| │ b ┆ 2 │
| │ c ┆ 3 │
| │ c ┆ 3 │
| │ c ┆ 3 │
| └─────┴─────┘
|
| upper_bound(self) -> 'Expr'
| Calculate the upper bound.
|
| Returns a unit Series with the highest value possible for the dtype of this
| expression.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").upper_bound())
| shape: (1, 1)
| ┌─────────────────────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════════════════════╡
| │ 9223372036854775807 │
| └─────────────────────┘
|
| value_counts(self, *, sort: 'bool' = False, parallel: 'bool' = False, name: 'str_ | None' = None, normalize: 'bool' = False) -> 'Expr'
| Count the occurrence of unique values.
|
| Parameters
| ----------
| sort
| Sort the output by count, in descending order.
| If set to `False` (default), the order is non-deterministic.
| parallel
| Execute the computation in parallel.
|
| .. note::
| This option should likely *not* be enabled in a `group_by` context,
| as the computation will already be parallelized per group.
| name
| Give the resulting count column a specific name; if `normalize` is
| True this defaults to "proportion", otherwise defaults to "count".
| normalize
| If True, the count is returned as the relative frequency of unique
| values normalized to 1.0.
|
| Returns
| -------
| Expr
| Expression of type :class:`Struct`, mapping unique values to their
| count (or proportion).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"color": ["red", "blue", "red", "green", "blue", "blue"]}
| ... )
| >>> df_count = df.select(pl.col("color").value_counts())
| >>> df_count # doctest: +IGNORE_RESULT
| shape: (3, 1)
| ┌─────────────┐
| │ color │
| │ --- │
| │ struct[2] │
| ╞═════════════╡
| │ {"green",1} │
| │ {"blue",3} │
| │ {"red",2} │
| └─────────────┘
|
| >>> df_count.unnest("color") # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌───────┬───────┐
| │ color ┆ count │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═══════╡
| │ green ┆ 1 │
| │ blue ┆ 3 │
| │ red ┆ 2 │
| └───────┴───────┘
|
| Sort the output by (descending) count, customize the field name,
| and normalize the count to its relative proportion (of 1.0).
|
| >>> df_count = df.select(
| ... pl.col("color").value_counts(
| ... name="fraction",
| ... normalize=True,
| ... sort=True,
| ... )
| ... )
| >>> df_count
| shape: (3, 1)
| ┌────────────────────┐
| │ color │
| │ --- │
| │ struct[2] │
| ╞════════════════════╡
| │ {"blue",0.5} │
| │ {"red",0.333333} │
| │ {"green",0.166667} │
| └────────────────────┘
|
| >>> df_count.unnest("color")
| shape: (3, 2)
| ┌───────┬──────────┐
| │ color ┆ fraction │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪══════════╡
| │ blue ┆ 0.5 │
| │ red ┆ 0.333333 │
| │ green ┆ 0.166667 │
| └───────┴──────────┘
|
| Note that `group_by` can be used to generate counts.
|
| >>> df.group_by("color").len() # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌───────┬─────┐
| │ color ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═════╡
| │ red ┆ 2 │
| │ green ┆ 1 │
| │ blue ┆ 3 │
| └───────┴─────┘
|
| To add counts as a new column `pl.len()` can be used as a window function.
|
| >>> df.with_columns(pl.len().over("color"))
| shape: (6, 2)
| ┌───────┬─────┐
| │ color ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═════╡
| │ red ┆ 2 │
| │ blue ┆ 3 │
| │ red ┆ 2 │
| │ green ┆ 1 │
| │ blue ┆ 3 │
| │ blue ┆ 3 │
| └───────┴─────┘
|
| >>> df.with_columns((pl.len().over("color") / pl.len()).alias("fraction"))
| shape: (6, 2)
| ┌───────┬──────────┐
| │ color ┆ fraction │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪══════════╡
| │ red ┆ 0.333333 │
| │ blue ┆ 0.5 │
| │ red ┆ 0.333333 │
| │ green ┆ 0.166667 │
| │ blue ┆ 0.5 │
| │ blue ┆ 0.5 │
| └───────┴──────────┘
|
| var(self, ddof: 'int' = 1) -> 'Expr'
| Get variance.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").var())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
|
| where(self, predicate: 'Expr') -> 'Expr'
| Filter a single column.
|
| .. deprecated:: 0.20.4
| Use the :func:`filter` method instead.
|
| Alias for :func:`filter`.
|
| Parameters
| ----------
| predicate
| Boolean expression.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group_col": ["g1", "g1", "g2"],
| ... "b": [1, 2, 3],
| ... }
| ... )
| >>> df.group_by("group_col").agg( # doctest: +SKIP
| ... [
| ... pl.col("b").where(pl.col("b") < 2).sum().alias("lt"),
| ... pl.col("b").where(pl.col("b") >= 2).sum().alias("gte"),
| ... ]
| ... ).sort("group_col")
| shape: (2, 3)
| ┌───────────┬─────┬─────┐
| │ group_col ┆ lt ┆ gte │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════════╪═════╪═════╡
| │ g1 ┆ 1 ┆ 2 │
| │ g2 ┆ 0 ┆ 3 │
| └───────────┴─────┴─────┘
|
| xor(self, other: 'Any') -> 'Expr'
| Method equivalent of bitwise exclusive-or operator `expr ^ other`.
|
| Parameters
| ----------
| other
| Integer or boolean value; accepts expression input.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"x": [True, False, True, False], "y": [True, True, False, False]}
| ... )
| >>> df.with_columns(pl.col("x").xor(pl.col("y")).alias("x ^ y"))
| shape: (4, 3)
| ┌───────┬───────┬───────┐
| │ x ┆ y ┆ x ^ y │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞═══════╪═══════╪═══════╡
| │ true ┆ true ┆ false │
| │ false ┆ true ┆ true │
| │ true ┆ false ┆ true │
| │ false ┆ false ┆ false │
| └───────┴───────┴───────┘
|
| >>> def binary_string(n: int) -> str:
| ... return bin(n)[2:].zfill(8)
| >>>
| >>> df = pl.DataFrame(
| ... data={"x": [10, 8, 250, 66], "y": [1, 2, 3, 4]},
| ... schema={"x": pl.UInt8, "y": pl.UInt8},
| ... )
| >>> df.with_columns(
| ... pl.col("x")
| ... .map_elements(binary_string, return_dtype=pl.String)
| ... .alias("bin_x"),
| ... pl.col("y")
| ... .map_elements(binary_string, return_dtype=pl.String)
| ... .alias("bin_y"),
| ... pl.col("x").xor(pl.col("y")).alias("xor_xy"),
| ... pl.col("x")
| ... .xor(pl.col("y"))
| ... .map_elements(binary_string, return_dtype=pl.String)
| ... .alias("bin_xor_xy"),
| ... )
| shape: (4, 6)
| ┌─────┬─────┬──────────┬──────────┬────────┬────────────┐
| │ x ┆ y ┆ bin_x ┆ bin_y ┆ xor_xy ┆ bin_xor_xy │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ str ┆ str ┆ u8 ┆ str │
| ╞═════╪═════╪══════════╪══════════╪════════╪════════════╡
| │ 10 ┆ 1 ┆ 00001010 ┆ 00000001 ┆ 11 ┆ 00001011 │
| │ 8 ┆ 2 ┆ 00001000 ┆ 00000010 ┆ 10 ┆ 00001010 │
| │ 250 ┆ 3 ┆ 11111010 ┆ 00000011 ┆ 249 ┆ 11111001 │
| │ 66 ┆ 4 ┆ 01000010 ┆ 00000100 ┆ 70 ┆ 01000110 │
| └─────┴─────┴──────────┴──────────┴────────┴────────────┘
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| deserialize(source: 'str_ | Path | IOBase | bytes', *, format: 'SerializationFormat' = 'binary') -> 'Expr'
| Read a serialized expression from a file.
|
| Parameters
| ----------
| source
| Path to a file or a file-like object (by file-like object, we refer to
| objects that have a `read()` method, such as a file handler (e.g.
| via builtin `open` function) or `BytesIO`).
| format
| The format with which the Expr was serialized. Options:
|
| - `"binary"`: Deserialize from binary format (bytes). This is the default.
| - `"json"`: Deserialize from JSON format (string).
|
| Warnings
| --------
| This function uses :mod:`pickle` if the logical plan contains Python UDFs,
| and as such inherits the security implications. Deserializing can execute
| arbitrary code, so it should only be attempted on trusted data.
|
| See Also
| --------
| Expr.meta.serialize
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| >>> import io
| >>> expr = pl.col("foo").sum().over("bar")
| >>> bytes = expr.meta.serialize()
| >>> pl.Expr.deserialize(io.BytesIO(bytes))
| <Expr ['col("foo").sum().over([col("ba…'] at ...>
|
| from_json(value: 'str_') -> 'Expr'
| Read an expression from a JSON encoded string to construct an Expression.
|
| .. deprecated:: 0.20.11
| This method has been renamed to :meth:`deserialize`.
| Note that the new method operates on file-like inputs rather than strings.
| Enclose your input in `io.StringIO` to keep the same behavior.
|
| Parameters
| ----------
| value
| JSON encoded string value
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| arr
| Create an object namespace of all array related methods.
|
| See the individual method pages for full details.
|
| bin
| Create an object namespace of all binary related methods.
|
| See the individual method pages for full details
|
| cat
| Create an object namespace of all categorical related methods.
|
| See the individual method pages for full details
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": ["a", "b"]}).select(
| ... pl.col("values").cast(pl.Categorical)
| ... )
| >>> df.select(pl.col("values").cat.get_categories())
| shape: (2, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ str │
| ╞════════╡
| │ a │
| │ b │
| └────────┘
|
| dt
| Create an object namespace of all datetime related methods.
|
| ext
| Create an object namespace of all extension type related expressions.
|
| See the individual method pages for full details.
|
| list
| Create an object namespace of all list related methods.
|
| See the individual method pages for full details.
|
| meta
| Create an object namespace of all meta related expression methods.
|
| This can be used to modify and traverse existing expressions.
|
| name
| Create an object namespace of all expressions that modify expression names.
|
| See the individual method pages for full details.
|
| str
| Create an object namespace of all string related methods.
|
| See the individual method pages for full details.
|
| Examples
| --------
| >>> df = pl.DataFrame({"letters": ["a", "b"]})
| >>> df.select(pl.col("letters").str.to_uppercase())
| shape: (2, 1)
| ┌─────────┐
| │ letters │
| │ --- │
| │ str │
| ╞═════════╡
| │ A │
| │ B │
| └─────────┘
|
| struct
| Create an object namespace of all struct related methods.
|
| See the individual method pages for full details.
|
| Examples
| --------
| >>> df = (
| ... pl.DataFrame(
| ... {
| ... "int": [1, 2],
| ... "str": ["a", "b"],
| ... "bool": [True, None],
| ... "list": [[1, 2], [3]],
| ... }
| ... )
| ... .to_struct("my_struct")
| ... .to_frame()
| ... )
| >>> df.select(pl.col("my_struct").struct.field("str"))
| shape: (2, 1)
| ┌─────┐
| │ str │
| │ --- │
| │ str │
| ╞═════╡
| │ a │
| │ b │
| └─────┘
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set[str_]]', '_pyexpr': 'Py...
polars.Extension
Python Library Documentation: class Extension in module polars.datatypes.classes
class Extension(BaseExtension)
| Extension(name: 'str', storage: 'PolarsDataType', metadata: 'str | None' = None) -> 'None'
|
| Generic extension data type.
|
| When `UNKNOWN_EXTENSION_TYPE_BEHAVIOR` is set to `"load_as_extension"`, any
| non-registered extension type will be loaded as this type.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| See Also
| --------
| BaseExtension
| polars.register_extension_type
|
| Method resolution order:
| Extension
| BaseExtension
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from BaseExtension:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __getstate__(self) -> 'tuple[str, PolarsDataType, str | None]'
| Helper for pickle.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, name: 'str', storage: 'PolarsDataType', metadata: 'str | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, state: 'tuple[str, PolarsDataType, str | None]') -> 'None'
|
| ext_metadata(self) -> 'str | None'
| Returns the metadata for this extension type.
|
| ext_name(self) -> 'str'
| Returns the name of this extension type.
|
| ext_storage(self) -> 'PolarsDataType'
| Returns the storage type for this extension type.
|
| ----------------------------------------------------------------------
| Class methods inherited from BaseExtension:
|
| ext_from_params(name: 'str', storage: 'PolarsDataType', metadata: 'str | None') -> 'Any'
| Creates an Extension type instance from its parameters.
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Field
Python Library Documentation: class Field in module polars.datatypes.classes
class Field(builtins.object)
| Field(name: 'str', dtype: 'PolarsDataType') -> 'None'
|
| Definition of a single field within a `Struct` DataType.
|
| Parameters
| ----------
| name
| The name of the field within its parent `Struct`.
| dtype
| The `DataType` of the field's values.
|
| Methods defined here:
|
| __eq__(self, other: 'Field') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, name: 'str', dtype: 'PolarsDataType') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'dtype': 'PolarsDataType', 'name': 'str'}
polars.FileProviderArgs
Python Library Documentation: class FileProviderArgs in module polars.io.partition
class FileProviderArgs(builtins.object)
| FileProviderArgs(*, index_in_partition: 'int', partition_keys: 'DataFrame') -> None
|
| Holds information on the file being sinked to.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, *, index_in_partition: 'int', partition_keys: 'DataFrame') -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'index_in_partition': 'int', 'partition_keys': 'Dat...
|
| __dataclass_fields__ = {'index_in_partition': Field(name='index_in_par...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ()
polars.Float16
Python Library Documentation: class Float16 in module polars.datatypes.classes
class Float16(FloatType)
| 16-bit floating point type.
|
| .. warning::
| Regular computing platforms do not natively support `Float16` operations,
| and compute operations on `Float16` will be significantly slower as a result
| than operation on :class:`Float32` or :class:`Float64`.
| As such, it is recommended to cast to `Float32` before doing any compute
| operations, and cast back to `Float16` afterward if needed.
|
| Method resolution order:
| Float16
| FloatType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Float32
Python Library Documentation: class Float32 in module polars.datatypes.classes
class Float32(FloatType)
| 32-bit floating point type.
|
| Method resolution order:
| Float32
| FloatType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Float64
Python Library Documentation: class Float64 in module polars.datatypes.classes
class Float64(FloatType)
| 64-bit floating point type.
|
| Method resolution order:
| Float64
| FloatType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.GPUEngine
Python Library Documentation: class GPUEngine in module polars.lazyframe.engine_config
class GPUEngine(builtins.object)
| GPUEngine(*, device: 'int | None' = None, memory_resource: 'Any | None' = None, raise_on_fail: 'bool' = False, **kwargs: 'Any') -> 'None'
|
| Configuration options for the GPU execution engine.
|
| Use this if you want control over details of the execution.
|
| Parameters
| ----------
| device : int, default None
| Select the GPU used to run the query. If not provided, the
| query uses the current CUDA device.
| memory_resource : rmm.mr.DeviceMemoryResource, default None
| Provide a memory resource for GPU memory allocations.
|
| .. warning::
| If passing a `memory_resource`, you must ensure that it is valid
| for the selected `device`. See the `RMM documentation
| <https://github.com/rapidsai/rmm?tab=readme-ov-file#multiple-devices>`_
| for more details.
|
| raise_on_fail : bool, default False
| If True, do not fall back to the Polars CPU engine if the GPU
| engine cannot execute the query, but instead raise an error.
|
| Methods defined here:
|
| __init__(self, *, device: 'int | None' = None, memory_resource: 'Any | None' = None, raise_on_fail: 'bool' = False, **kwargs: 'Any') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'config': 'Mapping[str, Any]', 'device': 'int | Non...
polars.Int128
Python Library Documentation: class Int128 in module polars.datatypes.classes
class Int128(SignedIntegerType)
| 128-bit signed integer type.
|
| .. warning::
| This functionality is considered **unstable**.
| It is a work-in-progress feature and may not always work as expected.
| It may be changed at any point without it being considered a breaking change.
|
| Method resolution order:
| Int128
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Int16
Python Library Documentation: class Int16 in module polars.datatypes.classes
class Int16(SignedIntegerType)
| 16-bit signed integer type.
|
| Method resolution order:
| Int16
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Int32
Python Library Documentation: class Int32 in module polars.datatypes.classes
class Int32(SignedIntegerType)
| 32-bit signed integer type.
|
| Method resolution order:
| Int32
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Int64
Python Library Documentation: class Int64 in module polars.datatypes.classes
class Int64(SignedIntegerType)
| 64-bit signed integer type.
|
| Method resolution order:
| Int64
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Int8
Python Library Documentation: class Int8 in module polars.datatypes.classes
class Int8(SignedIntegerType)
| 8-bit signed integer type.
|
| Method resolution order:
| Int8
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.LazyFrame
Python Library Documentation: class LazyFrame in module polars.lazyframe.frame
class LazyFrame(builtins.object)
| LazyFrame(data: 'FrameInitTypes | None' = None, schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100, nan_to_null: 'bool' = False, height: 'int | None' = None) -> 'None'
|
| Representation of a Lazy computation graph/query against a DataFrame.
|
| This allows for whole-query optimisation in addition to parallelism, and
| is the preferred (and highest-performance) mode of operation for polars.
|
| Parameters
| ----------
| data : dict, Sequence, ndarray, Series, or pandas.DataFrame
| Two-dimensional data in various forms; dict input must contain Sequences,
| Generators, or a `range`. Sequence may contain Series or other Sequences.
| schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
| The LazyFrame schema may be declared in several ways:
|
| * As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
| * As a list of column names; in this case types are automatically inferred.
| * As a list of (name,type) pairs; this is equivalent to the dictionary form.
|
| The order of the schema determines the column order of the frame.
| When passing a dict, its insertion order is respected. To override specific
| column data types by name without changing column order, use
| ``schema_overrides`` instead.
|
| If you supply a list of column names that does not match the names in the
| underlying data, the names given here will overwrite them. The number
| of names given in the schema should match the underlying data dimensions.
| schema_overrides : dict, default None
| Support type specification or override of one or more columns; note that
| any dtypes inferred from the schema param will be overridden.
|
| The number of entries in the schema should match the underlying data
| dimensions, unless a sequence of dictionaries is being passed, in which case
| a *partial* schema can be declared to prevent specific fields from being loaded.
| strict : bool, default True
| Throw an error if any `data` value does not exactly match the given or inferred
| data type for that column. If set to `False`, values that do not match the data
| type are cast to that data type or, if casting is not possible, set to null
| instead.
| orient : {'col', 'row'}, default None
| Whether to interpret two-dimensional data as columns or as rows. If None,
| the orientation is inferred by matching the columns and data dimensions. If
| this does not yield conclusive results, column orientation is used.
| infer_schema_length : int or None
| The maximum number of rows to scan for schema inference. If set to `None`, the
| full data may be scanned *(this can be slow)*. This parameter only applies if
| the input data is a sequence or generator of rows; other input is read as-is.
| nan_to_null : bool, default False
| If the data comes from one or more numpy arrays, can optionally convert input
| data np.nan values to null instead. This is a no-op for all other input data.
| height : int or None, default None
| Allows constructing DataFrames with 0 width and a specified height. If
| passed with data, ensures the resulting DataFrame has this height.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Notes
| -----
| Initialising `LazyFrame(...)` directly is equivalent to `DataFrame(...).lazy()`.
|
| Examples
| --------
| Constructing a LazyFrame directly from a dictionary:
|
| >>> data = {"a": [1, 2], "b": [3, 4]}
| >>> lf = pl.LazyFrame(data)
| >>> lf.collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
|
| Notice that the dtypes are automatically inferred as Polars Int64:
|
| >>> lf.collect_schema().dtypes()
| [Int64, Int64]
|
| To specify a more detailed/specific frame schema you can supply the `schema`
| parameter with a dictionary of (name,dtype) pairs...
|
| >>> data = {"col1": [0, 2], "col2": [3, 7]}
| >>> lf2 = pl.LazyFrame(data, schema={"col1": pl.Float32, "col2": pl.Int64})
| >>> lf2.collect()
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 0.0 ┆ 3 │
| │ 2.0 ┆ 7 │
| └──────┴──────┘
|
| ...a sequence of (name,dtype) pairs...
|
| >>> data = {"col1": [1, 2], "col2": [3, 4]}
| >>> lf3 = pl.LazyFrame(data, schema=[("col1", pl.Float32), ("col2", pl.Int64)])
| >>> lf3.collect()
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 1.0 ┆ 3 │
| │ 2.0 ┆ 4 │
| └──────┴──────┘
|
| ...or a list of typed Series.
|
| >>> data = [
| ... pl.Series("col1", [1, 2], dtype=pl.Float32),
| ... pl.Series("col2", [3, 4], dtype=pl.Int64),
| ... ]
| >>> lf4 = pl.LazyFrame(data)
| >>> lf4.collect()
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 1.0 ┆ 3 │
| │ 2.0 ┆ 4 │
| └──────┴──────┘
|
| Constructing a LazyFrame from a numpy ndarray, specifying column names:
|
| >>> import numpy as np
| >>> data = np.array([(1, 2), (3, 4)], dtype=np.int64)
| >>> lf5 = pl.LazyFrame(data, schema=["a", "b"], orient="col")
| >>> lf5.collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
|
| Constructing a LazyFrame from a list of lists, row orientation specified:
|
| >>> data = [[1, 2, 3], [4, 5, 6]]
| >>> lf6 = pl.LazyFrame(data, schema=["a", "b", "c"], orient="row")
| >>> lf6.collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| Methods defined here:
|
| __bool__(self) -> 'NoReturn'
|
| __contains__(self, key: 'str') -> 'bool'
|
| __copy__(self) -> 'LazyFrame'
|
| __deepcopy__(self, memo: 'None' = None) -> 'LazyFrame'
|
| __eq__(self, other: 'object') -> 'NoReturn'
| Return self==value.
|
| __ge__(self, other: 'Any') -> 'NoReturn'
| Return self>=value.
|
| __getitem__(self, item: 'slice_') -> 'LazyFrame'
| Support slice syntax, returning a new LazyFrame.
|
| All other forms of subscripting are currently unsupported here; use `select`,
| `filter`, or other standard methods instead.
|
| Notes
| -----
| LazyFrame is designed primarily for efficient computation and does not know
| its own length so, unlike DataFrame, certain slice patterns (such as those
| requiring negative stop/step) may not be supported.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
| >>> lf[:2].collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| └─────┴─────┘
| >>> lf[::2].collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 3 ┆ 6 │
| └─────┴─────┘
|
| __getstate__(self) -> 'bytes'
| Helper for pickle.
|
| __gt__(self, other: 'Any') -> 'NoReturn'
| Return self>value.
|
| __init__(self, data: 'FrameInitTypes | None' = None, schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100, nan_to_null: 'bool' = False, height: 'int | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __le__(self, other: 'Any') -> 'NoReturn'
| Return self<=value.
|
| __lt__(self, other: 'Any') -> 'NoReturn'
| Return self<value.
|
| __ne__(self, other: 'object') -> 'NoReturn'
| Return self!=value.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, state: 'bytes') -> 'None'
|
| __str__(self) -> 'str'
| Return str(self).
|
| approx_n_unique(self) -> 'LazyFrame'
| Approximate count of unique values.
|
| .. deprecated:: 0.20.11
| Use `select(pl.all().approx_n_unique())` instead.
|
| This is done using the HyperLogLog++ algorithm for cardinality estimation.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.approx_n_unique().collect() # doctest: +SKIP
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| bottom_k(self, k: 'int', *, by: 'IntoExpr | Iterable[IntoExpr]', reverse: 'bool | Sequence[bool]' = False) -> 'LazyFrame'
| Return the `k` smallest rows.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| k
| Number of rows to return.
| by
| Column(s) used to determine the bottom rows.
| Accepts expression input. Strings are parsed as column names.
| reverse
| Consider the `k` largest elements of the `by` column(s) (instead of the `k`
| smallest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [2, 1, 1, 3, 2, 1],
| ... }
| ... )
|
| Get the rows which contain the 4 smallest values in column b.
|
| >>> lf.bottom_k(4, by="b").collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 1 │
| │ a ┆ 1 │
| │ c ┆ 1 │
| │ a ┆ 2 │
| └─────┴─────┘
|
| Get the rows which contain the 4 smallest values when sorting on column a and b.
|
| >>> lf.bottom_k(4, by=["a", "b"]).collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ a ┆ 2 │
| │ b ┆ 1 │
| │ b ┆ 2 │
| └─────┴─────┘
|
| cache(self) -> 'LazyFrame'
| Cache the result once the execution of the physical plan hits this node.
|
| It is not recommended using this as the optimizer likely can do a better job.
|
| cast(self, dtypes: 'Mapping[ColumnNameOrSelector | PolarsDataType, PolarsDataType | PythonDataType] | PolarsDataType | pl.DataTypeExpr | Schema', *, strict: 'bool' = True) -> 'LazyFrame'
| Cast LazyFrame column(s) to the specified dtype(s).
|
| Parameters
| ----------
| dtypes
| Mapping of column names (or selector) to dtypes, or a single dtype
| to which all columns will be cast.
| strict
| Throw an error if a cast could not be done (for instance, due to an
| overflow).
|
| Examples
| --------
| >>> from datetime import date
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": [date(2020, 1, 2), date(2021, 3, 4), date(2022, 5, 6)],
| ... }
| ... )
|
| Cast specific frame columns to the specified dtypes:
|
| >>> lf.cast({"foo": pl.Float32, "bar": pl.UInt8}).collect()
| shape: (3, 3)
| ┌─────┬─────┬────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f32 ┆ u8 ┆ date │
| ╞═════╪═════╪════════════╡
| │ 1.0 ┆ 6 ┆ 2020-01-02 │
| │ 2.0 ┆ 7 ┆ 2021-03-04 │
| │ 3.0 ┆ 8 ┆ 2022-05-06 │
| └─────┴─────┴────────────┘
|
| Cast all frame columns matching one dtype (or dtype group) to another dtype:
|
| >>> lf.cast({pl.Date: pl.Datetime}).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────────────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ datetime[μs] │
| ╞═════╪═════╪═════════════════════╡
| │ 1 ┆ 6.0 ┆ 2020-01-02 00:00:00 │
| │ 2 ┆ 7.0 ┆ 2021-03-04 00:00:00 │
| │ 3 ┆ 8.0 ┆ 2022-05-06 00:00:00 │
| └─────┴─────┴─────────────────────┘
|
| Use selectors to define the columns being cast:
|
| >>> import polars.selectors as cs
| >>> lf.cast({cs.numeric(): pl.UInt32, cs.temporal(): pl.String}).collect()
| shape: (3, 3)
| ┌─────┬─────┬────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ str │
| ╞═════╪═════╪════════════╡
| │ 1 ┆ 6 ┆ 2020-01-02 │
| │ 2 ┆ 7 ┆ 2021-03-04 │
| │ 3 ┆ 8 ┆ 2022-05-06 │
| └─────┴─────┴────────────┘
|
| Cast all frame columns to the specified dtype:
|
| >>> lf.cast(pl.String).collect().to_dict(as_series=False)
| {'foo': ['1', '2', '3'],
| 'bar': ['6.0', '7.0', '8.0'],
| 'ham': ['2020-01-02', '2021-03-04', '2022-05-06']}
|
| clear(self, n: 'int' = 0) -> 'LazyFrame'
| Create an empty copy of the current LazyFrame, with zero to 'n' rows.
|
| Returns a copy with an identical schema but no data.
|
| Parameters
| ----------
| n
| Number of (empty) rows to return in the cleared frame.
|
| See Also
| --------
| clone : Cheap deepcopy/clone.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> lf.clear().collect()
| shape: (0, 3)
| ┌─────┬─────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞═════╪═════╪══════╡
| └─────┴─────┴──────┘
|
| >>> lf.clear(2).collect()
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ null ┆ null ┆ null │
| └──────┴──────┴──────┘
|
| clone(self) -> 'LazyFrame'
| Create a copy of this LazyFrame.
|
| This is a cheap operation that does not copy data.
|
| See Also
| --------
| clear : Create an empty copy of the current LazyFrame, with identical
| schema but no data.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> lf.clone()
| <LazyFrame at ...>
|
| collect(self, *, type_coercion: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, no_optimization: 'bool' = False, engine: 'EngineType' = 'auto', background: 'bool' = False, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, **_kwargs: 'Any') -> 'DataFrame | InProcessQuery'
| Materialize this `LazyFrame` into a `DataFrame`.
|
| By default, all query optimizations are enabled. Individual optimizations may
| be disabled by setting the corresponding parameter to `False`.
|
| Parameters
| ----------
| type_coercion
| Do type coercion optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| predicate_pushdown
| Do predicate pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| projection_pushdown
| Do projection pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| simplify_expression
| Run simplify expressions optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| slice_pushdown
| Slice pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subplan_elim
| Will try to cache branching subplans that occur on self-joins or unions.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subexpr_elim
| Common subexpressions will be cached and reused.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| cluster_with_columns
| Combine sequential independent calls to with_columns
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| collapse_joins
| Collapse a join and filters into a faster join
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| no_optimization
| Turn off (certain) optimizations.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| background
| Run the query in the background and get a handle to the query.
| This handle can be used to fetch the result or cancel the query.
|
| .. warning::
| Background mode is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| See Also
| --------
| explain : Print the query plan that is evaluated with collect.
| profile : Collect the LazyFrame and time each node in the computation graph.
| polars.collect_all : Collect multiple LazyFrames at the same time.
| polars.Config.set_streaming_chunk_size : Set the size of streaming batches.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a").agg(pl.all().sum()).collect() # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| Collect in streaming mode
|
| >>> lf.group_by("a").agg(pl.all().sum()).collect(
| ... engine="streaming"
| ... ) # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| Collect in GPU mode
|
| >>> lf.group_by("a").agg(pl.all().sum()).collect(engine="gpu") # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 11 ┆ 10 │
| │ a ┆ 4 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| With control over the device used
|
| >>> lf.group_by("a").agg(pl.all().sum()).collect(
| ... engine=pl.GPUEngine(device=1)
| ... ) # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 11 ┆ 10 │
| │ a ┆ 4 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| collect_async(self, *, gevent: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'Awaitable[DataFrame] | _GeventDataFrameResult[DataFrame]'
| Collect DataFrame asynchronously in thread pool.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Collects into a DataFrame (like :func:`collect`) but, instead of returning
| a DataFrame directly, it is scheduled to be collected inside a thread pool,
| while this method returns almost instantly.
|
| This can be useful if you use `gevent` or `asyncio` and want to release
| control to other greenlets/tasks while LazyFrames are being collected.
|
| Parameters
| ----------
| gevent
| Return wrapper to `gevent.event.AsyncResult` instead of Awaitable
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU
| systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| The GPU engine does not support async, or running in the
| background. If either are enabled, then GPU execution is switched off.
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| If `gevent=False` (default) then returns an awaitable.
|
| If `gevent=True` then returns wrapper that has a
| `.get(block=True, timeout=None)` method.
|
| See Also
| --------
| polars.collect_all : Collect multiple LazyFrames at the same time.
| polars.collect_all_async : Collect multiple LazyFrames at the same time lazily.
|
| Notes
| -----
| In case of error `set_exception` is used on
| `asyncio.Future`/`gevent.event.AsyncResult` and will be reraised by them.
|
| Examples
| --------
| >>> import asyncio
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> async def main():
| ... return await (
| ... lf.group_by("a", maintain_order=True)
| ... .agg(pl.all().sum())
| ... .collect_async()
| ... )
| >>> asyncio.run(main())
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| collect_batches(self, *, chunk_size: 'int | None' = None, maintain_order: 'bool' = True, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'Iterator[DataFrame]'
| Evaluate the query in streaming mode and get a generator that returns chunks.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| The query will always be fully executed unless `stop` is called, so you should
| call next until all chunks have been seen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. warning::
| This method is much slower than native sinks. Only use it if you cannot
| implement your logic otherwise.
|
| Parameters
| ----------
| chunk_size
| The number of rows that are buffered before a chunk is given.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
| lazy
| Start the query when first requesting a batch.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> for df in lf.collect_batches():
| ... print(df) # doctest: +SKIP
|
| collect_schema(self) -> 'Schema'
| Resolve the schema of this LazyFrame.
|
| .. caution::
| Computing the schema of a LazyFrame is a potentially expensive operation,
| as it may involve reading metadata from (slow) disk storage, or performing
| network requests if the data is remote.
|
| Examples
| --------
| Determine the schema.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.collect_schema()
| Schema({'foo': Int64, 'bar': Float64, 'ham': String})
|
| Access various properties of the schema.
|
| >>> schema = lf.collect_schema()
| >>> schema["bar"]
| Float64
| >>> schema.names()
| ['foo', 'bar', 'ham']
| >>> schema.dtypes()
| [Int64, Float64, String]
| >>> schema.len()
| 3
|
| count(self) -> 'LazyFrame'
| Return the number of non-null elements for each column.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {"a": [1, 2, 3, 4], "b": [1, 2, 1, None], "c": [None, None, None, None]}
| ... )
| >>> lf.count().collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 4 ┆ 3 ┆ 0 │
| └─────┴─────┴─────┘
|
| describe(self, percentiles: 'Sequence[float] | float | None' = (0.25, 0.5, 0.75), *, interpolation: 'QuantileMethod' = 'nearest') -> 'DataFrame'
| Creates a summary of statistics for a LazyFrame, returning a DataFrame.
|
| Parameters
| ----------
| percentiles
| One or more percentiles to include in the summary statistics.
| All values must be in the range `[0, 1]`.
|
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method used when calculating percentiles.
|
| Returns
| -------
| DataFrame
|
| Notes
| -----
| The median is included by default as the 50% percentile.
|
| Warnings
| --------
| * This method does *not* maintain the laziness of the frame, and will `collect`
| the final result. This could potentially be an expensive operation.
| * We do not guarantee the output of `describe` to be stable. It will show
| statistics that we deem informative, and may be updated in the future.
| Using `describe` programmatically (versus interactive exploration) is
| not recommended for this reason.
|
| Examples
| --------
| >>> from datetime import date, time
| >>> lf = pl.LazyFrame(
| ... {
| ... "float": [1.0, 2.8, 3.0],
| ... "int": [40, 50, None],
| ... "bool": [True, False, True],
| ... "str": ["zz", "xx", "yy"],
| ... "date": [date(2020, 1, 1), date(2021, 7, 5), date(2022, 12, 31)],
| ... "time": [time(10, 20, 30), time(14, 45, 50), time(23, 15, 10)],
| ... }
| ... )
|
| Show default frame statistics:
|
| >>> lf.describe()
| shape: (9, 7)
| ┌────────────┬──────────┬──────────┬──────────┬──────┬─────────────────────┬──────────┐
| │ statistic ┆ float ┆ int ┆ bool ┆ str ┆ date ┆ time │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞════════════╪══════════╪══════════╪══════════╪══════╪═════════════════════╪══════════╡
| │ count ┆ 3.0 ┆ 2.0 ┆ 3.0 ┆ 3 ┆ 3 ┆ 3 │
| │ null_count ┆ 0.0 ┆ 1.0 ┆ 0.0 ┆ 0 ┆ 0 ┆ 0 │
| │ mean ┆ 2.266667 ┆ 45.0 ┆ 0.666667 ┆ null ┆ 2021-07-02 16:00:00 ┆ 16:07:10 │
| │ std ┆ 1.101514 ┆ 7.071068 ┆ null ┆ null ┆ null ┆ null │
| │ min ┆ 1.0 ┆ 40.0 ┆ 0.0 ┆ xx ┆ 2020-01-01 ┆ 10:20:30 │
| │ 25% ┆ 2.8 ┆ 40.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 50% ┆ 2.8 ┆ 50.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 75% ┆ 3.0 ┆ 50.0 ┆ null ┆ null ┆ 2022-12-31 ┆ 23:15:10 │
| │ max ┆ 3.0 ┆ 50.0 ┆ 1.0 ┆ zz ┆ 2022-12-31 ┆ 23:15:10 │
| └────────────┴──────────┴──────────┴──────────┴──────┴─────────────────────┴──────────┘
|
| Customize which percentiles are displayed, applying linear interpolation:
|
| >>> with pl.Config(tbl_rows=12):
| ... lf.describe(
| ... percentiles=[0.1, 0.3, 0.5, 0.7, 0.9],
| ... interpolation="linear",
| ... )
| shape: (11, 7)
| ┌────────────┬──────────┬──────────┬──────────┬──────┬─────────────────────┬──────────┐
| │ statistic ┆ float ┆ int ┆ bool ┆ str ┆ date ┆ time │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞════════════╪══════════╪══════════╪══════════╪══════╪═════════════════════╪══════════╡
| │ count ┆ 3.0 ┆ 2.0 ┆ 3.0 ┆ 3 ┆ 3 ┆ 3 │
| │ null_count ┆ 0.0 ┆ 1.0 ┆ 0.0 ┆ 0 ┆ 0 ┆ 0 │
| │ mean ┆ 2.266667 ┆ 45.0 ┆ 0.666667 ┆ null ┆ 2021-07-02 16:00:00 ┆ 16:07:10 │
| │ std ┆ 1.101514 ┆ 7.071068 ┆ null ┆ null ┆ null ┆ null │
| │ min ┆ 1.0 ┆ 40.0 ┆ 0.0 ┆ xx ┆ 2020-01-01 ┆ 10:20:30 │
| │ 10% ┆ 1.36 ┆ 41.0 ┆ null ┆ null ┆ 2020-04-20 ┆ 11:13:34 │
| │ 30% ┆ 2.08 ┆ 43.0 ┆ null ┆ null ┆ 2020-11-26 ┆ 12:59:42 │
| │ 50% ┆ 2.8 ┆ 45.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 70% ┆ 2.88 ┆ 47.0 ┆ null ┆ null ┆ 2022-02-07 ┆ 18:09:34 │
| │ 90% ┆ 2.96 ┆ 49.0 ┆ null ┆ null ┆ 2022-09-13 ┆ 21:33:18 │
| │ max ┆ 3.0 ┆ 50.0 ┆ 1.0 ┆ zz ┆ 2022-12-31 ┆ 23:15:10 │
| └────────────┴──────────┴──────────┴──────────┴──────┴─────────────────────┴──────────┘
|
| drop(self, *columns: 'ColumnNameOrSelector | Iterable[ColumnNameOrSelector]', strict: 'bool' = True) -> 'LazyFrame'
| Remove columns from the DataFrame.
|
| Parameters
| ----------
| *columns
| Names of the columns that should be removed from the dataframe.
| Accepts column selector input.
| strict
| Validate that all column names exist in the current schema,
| and throw an exception if any do not.
|
| Examples
| --------
| Drop a single column by passing the name of that column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.drop("ham").collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 6.0 │
| │ 2 ┆ 7.0 │
| │ 3 ┆ 8.0 │
| └─────┴─────┘
|
| Drop multiple columns by passing a selector.
|
| >>> import polars.selectors as cs
| >>> lf.drop(cs.numeric()).collect()
| shape: (3, 1)
| ┌─────┐
| │ ham │
| │ --- │
| │ str │
| ╞═════╡
| │ a │
| │ b │
| │ c │
| └─────┘
|
| Use positional arguments to drop multiple columns.
|
| >>> lf.drop("foo", "ham").collect()
| shape: (3, 1)
| ┌─────┐
| │ bar │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 6.0 │
| │ 7.0 │
| │ 8.0 │
| └─────┘
|
| drop_nans(self, subset: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None) -> 'LazyFrame'
| Drop all rows that contain one or more NaN values.
|
| The original order of the remaining rows is preserved.
|
| Parameters
| ----------
| subset
| Column name(s) for which NaN values are considered; if set to `None`
| (default), use all columns (note that only floating-point columns
| can contain NaNs).
|
| See Also
| --------
| drop_nulls
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To drop null values, use :func:`drop_nulls`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [-20.5, float("nan"), 80.0],
| ... "bar": [float("nan"), 110.0, 25.5],
| ... "ham": ["xxx", "yyy", None],
| ... }
| ... )
|
| The default behavior of this method is to drop rows where any single
| value in the row is NaN:
|
| >>> lf.drop_nans().collect()
| shape: (1, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════╪══════╪══════╡
| │ 80.0 ┆ 25.5 ┆ null │
| └──────┴──────┴──────┘
|
| This behaviour can be constrained to consider only a subset of columns, as
| defined by name, or with a selector. For example, dropping rows only if
| there is a NaN in the "bar" column:
|
| >>> lf.drop_nans(subset=["bar"]).collect()
| shape: (2, 3)
| ┌──────┬───────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════╪═══════╪══════╡
| │ NaN ┆ 110.0 ┆ yyy │
| │ 80.0 ┆ 25.5 ┆ null │
| └──────┴───────┴──────┘
|
| Dropping a row only if *all* values are NaN requires a different formulation:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [float("nan"), float("nan"), float("nan"), float("nan")],
| ... "b": [10.0, 2.5, float("nan"), 5.25],
| ... "c": [65.75, float("nan"), float("nan"), 10.5],
| ... }
| ... )
| >>> lf.filter(~pl.all_horizontal(pl.all().is_nan())).collect()
| shape: (3, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞═════╪══════╪═══════╡
| │ NaN ┆ 10.0 ┆ 65.75 │
| │ NaN ┆ 2.5 ┆ NaN │
| │ NaN ┆ 5.25 ┆ 10.5 │
| └─────┴──────┴───────┘
|
| drop_nulls(self, subset: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None) -> 'LazyFrame'
| Drop all rows that contain one or more null values.
|
| The original order of the remaining rows is preserved.
|
| See Also
| --------
| drop_nans
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To drop NaN values, use :func:`drop_nans`.
|
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, None, 8],
| ... "ham": ["a", "b", None],
| ... }
| ... )
|
| The default behavior of this method is to drop rows where any single
| value in the row is null:
|
| >>> lf.drop_nulls().collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| This behaviour can be constrained to consider only a subset of columns, as
| defined by name or with a selector. For example, dropping rows if there is
| a null in any of the integer columns:
|
| >>> import polars.selectors as cs
| >>> lf.drop_nulls(subset=cs.integer()).collect()
| shape: (2, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 6 ┆ a │
| │ 3 ┆ 8 ┆ null │
| └─────┴─────┴──────┘
|
| Dropping a row only if *all* values are null requires a different formulation:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [None, None, None, None],
| ... "b": [1, 2, None, 1],
| ... "c": [1, None, None, 1],
| ... }
| ... )
| >>> lf.filter(~pl.all_horizontal(pl.all().is_null())).collect()
| shape: (3, 3)
| ┌──────┬─────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ null ┆ i64 ┆ i64 │
| ╞══════╪═════╪══════╡
| │ null ┆ 1 ┆ 1 │
| │ null ┆ 2 ┆ null │
| │ null ┆ 1 ┆ 1 │
| └──────┴─────┴──────┘
|
| execute(self, *, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, engine: 'EngineType' = 'auto', **_kwargs: 'Any') -> 'QueryResult'
| Execute the query into a `QueryResult`.
|
| This method of materializing a `LazyFrame` makes no guarantees as to where
| the result is materialized. This can be on the GPU for the GPU-engine,
| on the cluster or remote storage for the distributed engine and the streaming
| engine could spill the result if it needed to.
|
| The `QueryResult` can always be consumed as a new `LazyFrame` by calling `.lazy`
|
| Parameters
| ----------
| engine
| Select the engine used to process the query, optional.
| At the moment, if set to `"auto"` (default), the query
| is run using the polars in-memory engine. Polars will also
| attempt to use the engine set by the `POLARS_ENGINE_AFFINITY`
| environment variable. If it cannot run the query using the
| selected engine, the query is run using the polars in-memory
| engine. If set to `"gpu"`, the GPU engine is used. Fine-grained
| control over the GPU engine, for example which device to use
| on a system with multiple devices, is possible by providing a
| :class:`~.GPUEngine` object with configuration options.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| See Also
| --------
| explain : Print the query plan that is evaluated with collect.
| profile : Collect the LazyFrame and time each node in the computation graph.
| polars.collect_all : Collect multiple LazyFrames at the same time.
| polars.Config.set_streaming_chunk_size : Set the size of streaming batches.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a").agg(pl.all().sum()).collect() # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| Collect in streaming mode
|
| >>> lf.group_by("a").agg(pl.all().sum()).collect(
| ... engine="streaming"
| ... ) # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| Collect in GPU mode
|
| explain(self, *, format: 'ExplainFormat' = 'plain', optimized: 'bool' = True, type_coercion: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, streaming: 'bool' = False, engine: 'EngineType' = 'auto', tree_format: 'bool | None' = None, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'str'
| Create a string representation of the query plan.
|
| Different optimizations can be turned on or off.
|
| Parameters
| ----------
| format : {'plain', 'tree'}
| The format to use for displaying the logical plan.
| optimized
| Return an optimized query plan. Defaults to `True`.
| If this is set to `True` the subsequent
| optimization flags control which optimizations
| run.
| type_coercion
| Do type coercion optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| predicate_pushdown
| Do predicate pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| projection_pushdown
| Do projection pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| simplify_expression
| Run simplify expressions optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| slice_pushdown
| Slice pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subplan_elim
| Will try to cache branching subplans that occur on self-joins or unions.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subexpr_elim
| Common subexpressions will be cached and reused.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| cluster_with_columns
| Combine sequential independent calls to with_columns
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| collapse_joins
| Collapse a join and filters into a faster join
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| streaming
| Unused parameter, kept for backward compatibility.
|
| .. deprecated:: 1.30.0
| Use the `engine` parameter instead.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| tree_format
| Format the output as a tree.
|
| .. deprecated:: 0.20.30
| Use `format="tree"` instead.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort(
| ... "a"
| ... ).explain() # doctest: +SKIP
|
| explode(self, columns: 'ColumnNameOrSelector | Iterable[ColumnNameOrSelector]', *more_columns: 'ColumnNameOrSelector', empty_as_null: 'bool' = <object object at 0x00000280D5528F20>, keep_nulls: 'bool' = True) -> 'LazyFrame'
| Explode the DataFrame to long format by exploding the given columns.
|
| Parameters
| ----------
| columns
| Column names, expressions, or a selector defining them. The underlying
| columns being exploded must be of the `List` or `Array` data type.
| *more_columns
| Additional names of columns to explode, specified as positional arguments.
| empty_as_null
| Explode an empty list/array into a `null`.
| keep_nulls
| Explode a `null` list/array into a `null`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "letters": ["a", "a", "b", "c"],
| ... "numbers": [[1], [2, 3], [4, 5], [6, 7, 8]],
| ... }
| ... )
| >>> lf.explode("numbers", empty_as_null=False).collect()
| shape: (8, 2)
| ┌─────────┬─────────┐
| │ letters ┆ numbers │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════════╡
| │ a ┆ 1 │
| │ a ┆ 2 │
| │ a ┆ 3 │
| │ b ┆ 4 │
| │ b ┆ 5 │
| │ c ┆ 6 │
| │ c ┆ 7 │
| │ c ┆ 8 │
| └─────────┴─────────┘
|
| fetch(self, n_rows: 'int' = 500, **kwargs: 'Any') -> 'DataFrame'
| Collect a small number of rows for debugging purposes.
|
| .. deprecated:: 1.0
| Use :meth:`collect` instead, in conjunction with a call to :meth:`head`.`
|
| Notes
| -----
| This is similar to a :func:`collect` operation, but it overwrites the number of
| rows read by *every* scan operation. Be aware that `fetch` does not guarantee
| the final number of rows in the DataFrame. Filters, join operations and fewer
| rows being available in the scanned data will all influence the final number
| of rows (joins are especially susceptible to this, and may return no data
| at all if `n_rows` is too small as the join keys may not be present).
|
| Warnings
| --------
| This is strictly a utility function that can help to debug queries using a
| smaller number of rows, and should *not* be used in production code.
|
| fill_nan(self, value: 'int | float | Expr | None') -> 'LazyFrame'
| Fill floating point NaN values.
|
| Parameters
| ----------
| value
| Value used to fill NaN values.
|
| See Also
| --------
| fill_null
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To fill null values, use :func:`fill_null`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1.5, 2, float("nan"), 4],
| ... "b": [0.5, 4, float("nan"), 13],
| ... }
| ... )
| >>> lf.fill_nan(99).collect()
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪══════╡
| │ 1.5 ┆ 0.5 │
| │ 2.0 ┆ 4.0 │
| │ 99.0 ┆ 99.0 │
| │ 4.0 ┆ 13.0 │
| └──────┴──────┘
|
| fill_null(self, value: 'Any | Expr | None' = None, strategy: 'FillNullStrategy | None' = None, limit: 'int | None' = None, *, matches_supertype: 'bool' = True) -> 'LazyFrame'
| Fill null values using the specified value or strategy.
|
| Parameters
| ----------
| value
| Value used to fill null values.
| strategy : {None, 'forward', 'backward', 'min', 'max', 'mean', 'zero', 'one'}
| Strategy used to fill null values.
| limit
| Number of consecutive null values to fill when using the 'forward' or
| 'backward' strategy.
| matches_supertype
| Fill all matching supertypes of the fill `value` literal.
|
| See Also
| --------
| fill_nan
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To fill NaN values, use :func:`fill_nan`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, None, 4],
| ... "b": [0.5, 4, None, 13],
| ... }
| ... )
| >>> lf.fill_null(99).collect()
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 99 ┆ 99.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
| >>> lf.fill_null(strategy="forward").collect()
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 2 ┆ 4.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| >>> lf.fill_null(strategy="max").collect()
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 4 ┆ 13.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| >>> lf.fill_null(strategy="zero").collect()
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 0 ┆ 0.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| filter(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]', **constraints: 'Any') -> 'LazyFrame'
| Filter rows in the LazyFrame based on a predicate expression.
|
| The original order of the remaining rows is preserved.
|
| Rows where the filter predicate does not evaluate to True are discarded
| (this includes rows where the predicate evaluates as `null`).
|
| Parameters
| ----------
| predicates
| Expression that evaluates to a boolean Series.
| constraints
| Column filters; use `name = value` to filter columns using the supplied
| value. Each constraint behaves the same as `pl.col(name).eq(value)`,
| and is implicitly joined with the other filter conditions using `&`.
|
| Notes
| -----
| If you are transitioning from Pandas, and performing filter operations based on
| the comparison of two or more columns, please note that in Polars any comparison
| involving `null` values will result in a `null` result, *not* boolean True or
| False. As a result, these rows will not be retained. Ensure that null values
| are handled appropriately to avoid unexpected behaviour (see examples below).
|
| See Also
| --------
| remove
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3, None, 4, None, 0],
| ... "bar": [6, 7, 8, None, None, 9, 0],
| ... "ham": ["a", "b", "c", None, "d", "e", "f"],
| ... }
| ... )
|
| Filter on one condition:
|
| >>> lf.filter(pl.col("foo") > 1).collect()
| shape: (3, 3)
| ┌─────┬──────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪══════╪═════╡
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ null ┆ d │
| └─────┴──────┴─────┘
|
| Filter on multiple conditions:
|
| >>> lf.filter((pl.col("foo") < 3) & (pl.col("ham") == "a")).collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| Provide multiple filters using `*args` syntax:
|
| >>> lf.filter(
| ... pl.col("foo") == 1,
| ... pl.col("ham") == "a",
| ... ).collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| Provide multiple filters using `**kwargs` syntax:
|
| >>> lf.filter(foo=1, ham="a").collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| Filter on an OR condition:
|
| >>> lf.filter(
| ... (pl.col("foo") == 1) | (pl.col("ham") == "c"),
| ... ).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Filter by comparing two columns against each other
|
| >>> lf.filter(
| ... pl.col("foo") == pl.col("bar"),
| ... ).collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 0 ┆ 0 ┆ f │
| └─────┴─────┴─────┘
|
| >>> lf.filter(
| ... pl.col("foo") != pl.col("bar"),
| ... ).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Notice how the row with `None` values is filtered out; using `ne_missing`
| ensures that null values compare equal, and we get similar behaviour to Pandas:
|
| >>> lf.filter(
| ... pl.col("foo").ne_missing(pl.col("bar")),
| ... ).collect()
| shape: (5, 3)
| ┌──────┬──────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ null ┆ d │
| │ null ┆ 9 ┆ e │
| └──────┴──────┴─────┘
|
| first(self) -> 'LazyFrame'
| Get the first row of the DataFrame.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> lf.first().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 2 │
| └─────┴─────┘
|
| gather(self, indices: 'int | Sequence[int] | IntoExpr | pl.Series | np.ndarray[Any, Any] | pl.LazyFrame', *, null_on_oob: 'bool' = False) -> 'LazyFrame'
| Selects rows from this LazyFrame at the given indices.
|
| .. warning::
| This functionality is experimental. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| indices
| The indices of the rows to select.
|
| Due to the lack of a ``LazySeries`` it's permitted to pass a single-width
| ``LazyFrame`` as indices as well.
|
| null_on_oob
| If true when an index is out-of-bounds a null row will be generated
| instead of raising an error.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"x": [2, 1, 0], "s": ["foo", "bar", "baz"]})
| >>> lf.gather([2, 0, 0]).collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 0 ┆ baz │
| │ 2 ┆ foo │
| │ 2 ┆ foo │
| └─────┴─────┘
|
| >>> lf.gather([0, 10, 1], null_on_oob=True).collect()
| shape: (3, 2)
| ┌──────┬──────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞══════╪══════╡
| │ 2 ┆ foo │
| │ null ┆ null │
| │ 1 ┆ bar │
| └──────┴──────┘
|
| >>> idxs = pl.LazyFrame({"i": [1, 10, 0], "b": [True, False, True]})
| >>> lf.gather(idxs.filter(pl.col.b).select(pl.col.i)).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ bar │
| │ 2 ┆ foo │
| └─────┴─────┘
|
| gather_every(self, n: 'int', offset: 'int' = 0) -> 'LazyFrame'
| Take every nth row in the LazyFrame and return as a new LazyFrame.
|
| Parameters
| ----------
| n
| Gather every *n*-th row.
| offset
| Starting index.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [5, 6, 7, 8],
| ... }
| ... )
| >>> lf.gather_every(2).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 5 │
| │ 3 ┆ 7 │
| └─────┴─────┘
| >>> lf.gather_every(2, offset=1).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 6 │
| │ 4 ┆ 8 │
| └─────┴─────┘
|
| group_by(self, *by: 'IntoExpr | Iterable[IntoExpr]', maintain_order: 'bool' = False, **named_by: 'IntoExpr') -> 'LazyGroupBy'
| Start a group by operation.
|
| Parameters
| ----------
| *by
| Column(s) to group by. Accepts expression input. Strings are parsed as
| column names.
| maintain_order
| Ensure that the order of the groups is consistent with the input data.
| This is slower than a default group by.
| Setting this to `True` blocks the possibility
| to run on the streaming engine.
| **named_by
| Additional columns to group by, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Examples
| --------
| Group by one column and call `agg` to compute the grouped sum of another
| column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "c"],
| ... "b": [1, 2, 1, 3, 3],
| ... "c": [5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a").agg(pl.col("b").sum()).collect() # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 2 │
| │ b ┆ 5 │
| │ c ┆ 3 │
| └─────┴─────┘
|
| Set `maintain_order=True` to ensure the order of the groups is consistent with
| the input.
|
| >>> lf.group_by("a", maintain_order=True).agg(pl.col("c")).collect()
| shape: (3, 2)
| ┌─────┬───────────┐
| │ a ┆ c │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═════╪═══════════╡
| │ a ┆ [5, 3] │
| │ b ┆ [4, 2] │
| │ c ┆ [1] │
| └─────┴───────────┘
|
| Group by multiple columns by passing a list of column names.
|
| >>> lf.group_by(["a", "b"]).agg(pl.max("c")).collect() # doctest: +SKIP
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘
|
| Or use positional arguments to group by multiple columns in the same way.
| Expressions are also accepted.
|
| >>> lf.group_by("a", pl.col("b") // 2).agg(
| ... pl.col("c").mean()
| ... ).collect() # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ f64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 0 ┆ 4.0 │
| │ b ┆ 1 ┆ 3.0 │
| │ c ┆ 1 ┆ 1.0 │
| └─────┴─────┴─────┘
|
| group_by_dynamic(self, index_column: 'IntoExpr', *, every: 'str | timedelta', period: 'str | timedelta | None' = None, offset: 'str | timedelta | None' = None, include_boundaries: 'bool' = False, closed: 'ClosedInterval' = 'left', label: 'Label' = 'left', group_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, start_by: 'StartBy' = 'window') -> 'LazyGroupBy'
| Group based on a time value (or index value of type Int32, Int64).
|
| Time windows are calculated and rows are assigned to windows. Different from a
| normal group by is that a row can be member of multiple groups.
| By default, the windows look like:
|
| - [start, start + period)
| - [start + every, start + every + period)
| - [start + 2*every, start + 2*every + period)
| - ...
|
| where `start` is determined by `start_by`, `offset`, `every`, and the earliest
| datapoint. See the `start_by` argument description for details.
|
| .. warning::
| The index column must be sorted in ascending order. If `group_by` is passed, then
| the index column must be sorted in ascending order within each group.
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order (or, if `group_by` is specified,
| then it must be sorted in ascending order within each group).
|
| In case of a dynamic group by on indices, dtype needs to be one of
| {Int32, Int64}. Note that Int32 gets temporarily cast to Int64, so if
| performance matters use an Int64 column.
| every
| interval of the window
| period
| length of the window, if None it will equal 'every'
| offset
| offset of the window, does not take effect if `start_by` is 'datapoint'.
| Defaults to zero.
| include_boundaries
| Add the lower and upper bound of the window to the "_lower_boundary" and
| "_upper_boundary" columns. This will impact performance because it's harder to
| parallelize
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
| label : {'left', 'right', 'datapoint'}
| Define which label to use for the window:
|
| - 'left': lower boundary of the window
| - 'right': upper boundary of the window
| - 'datapoint': the first value of the index column in the given window.
| If you don't need the label to be at one of the boundaries, choose this
| option for maximum performance
| group_by
| Also group by this column/these columns
| start_by : {'window', 'datapoint', 'monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday', 'sunday'}
| The strategy to determine the start of the first window by.
|
| * 'window': Start by taking the earliest timestamp, truncating it with
| `every`, and then adding `offset`.
| Note that weekly windows start on Monday.
| * 'datapoint': Start from the first encountered data point.
| * a day of the week (only takes effect if `every` contains `'w'`):
|
| * 'monday': Start the window on the Monday before the first data point.
| * 'tuesday': Start the window on the Tuesday before the first data point.
| * ...
| * 'sunday': Start the window on the Sunday before the first data point.
|
| The resulting window is then shifted back until the earliest datapoint
| is in or in front of it.
|
| Returns
| -------
| LazyGroupBy
| Object you can call `.agg` on to aggregate by groups, the result
| of which will be sorted by `index_column` (but note that if `group_by` columns are
| passed, it will only be sorted within each group).
|
| See Also
| --------
| rolling
|
| Notes
| -----
| 1) If you're coming from pandas, then
|
| .. code-block:: python
|
| # polars
| df.group_by_dynamic("ts", every="1d").agg(pl.col("value").sum())
|
| is equivalent to
|
| .. code-block:: python
|
| # pandas
| df.set_index("ts").resample("D")["value"].sum().reset_index()
|
| though note that, unlike pandas, polars doesn't add extra rows for empty
| windows. If you need `index_column` to be evenly spaced, then please combine
| with :func:`DataFrame.upsample`.
|
| 2) The `every`, `period` and `offset` arguments are created with
| the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them (except in `every`):
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| In case of a group_by_dynamic on an integer column, the windows are defined by:
|
| - "1i" # length 1
| - "10i" # length 10
|
| Examples
| --------
| >>> from datetime import datetime
| >>> lf = pl.LazyFrame(
| ... {
| ... "time": pl.datetime_range(
| ... start=datetime(2021, 12, 16),
| ... end=datetime(2021, 12, 16, 3),
| ... interval="30m",
| ... eager=True,
| ... ),
| ... "n": range(7),
| ... }
| ... )
| >>> lf.collect()
| shape: (7, 2)
| ┌─────────────────────┬─────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ i64 │
| ╞═════════════════════╪═════╡
| │ 2021-12-16 00:00:00 ┆ 0 │
| │ 2021-12-16 00:30:00 ┆ 1 │
| │ 2021-12-16 01:00:00 ┆ 2 │
| │ 2021-12-16 01:30:00 ┆ 3 │
| │ 2021-12-16 02:00:00 ┆ 4 │
| │ 2021-12-16 02:30:00 ┆ 5 │
| │ 2021-12-16 03:00:00 ┆ 6 │
| └─────────────────────┴─────┘
|
| Group by windows of 1 hour.
|
| >>> lf.group_by_dynamic("time", every="1h", closed="right").agg(
| ... pl.col("n")
| ... ).collect()
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-15 23:00:00 ┆ [0] │
| │ 2021-12-16 00:00:00 ┆ [1, 2] │
| │ 2021-12-16 01:00:00 ┆ [3, 4] │
| │ 2021-12-16 02:00:00 ┆ [5, 6] │
| └─────────────────────┴───────────┘
|
| The window boundaries can also be added to the aggregation result
|
| >>> lf.group_by_dynamic(
| ... "time", every="1h", include_boundaries=True, closed="right"
| ... ).agg(pl.col("n").mean()).collect()
| shape: (4, 4)
| ┌─────────────────────┬─────────────────────┬─────────────────────┬─────┐
| │ _lower_boundary ┆ _upper_boundary ┆ time ┆ n │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ f64 │
| ╞═════════════════════╪═════════════════════╪═════════════════════╪═════╡
| │ 2021-12-15 23:00:00 ┆ 2021-12-16 00:00:00 ┆ 2021-12-15 23:00:00 ┆ 0.0 │
| │ 2021-12-16 00:00:00 ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 00:00:00 ┆ 1.5 │
| │ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ 3.5 │
| │ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ 5.5 │
| └─────────────────────┴─────────────────────┴─────────────────────┴─────┘
|
| When closed="left", the window excludes the right end of interval:
| [lower_bound, upper_bound)
|
| >>> lf.group_by_dynamic("time", every="1h", closed="left").agg(
| ... pl.col("n")
| ... ).collect()
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-16 00:00:00 ┆ [0, 1] │
| │ 2021-12-16 01:00:00 ┆ [2, 3] │
| │ 2021-12-16 02:00:00 ┆ [4, 5] │
| │ 2021-12-16 03:00:00 ┆ [6] │
| └─────────────────────┴───────────┘
|
| When closed="both" the time values at the window boundaries belong to 2 groups.
|
| >>> lf.group_by_dynamic("time", every="1h", closed="both").agg(
| ... pl.col("n")
| ... ).collect()
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-16 00:00:00 ┆ [0, 1, 2] │
| │ 2021-12-16 01:00:00 ┆ [2, 3, 4] │
| │ 2021-12-16 02:00:00 ┆ [4, 5, 6] │
| │ 2021-12-16 03:00:00 ┆ [6] │
| └─────────────────────┴───────────┘
|
| Dynamic group bys can also be combined with grouping on normal keys
|
| >>> lf = lf.with_columns(groups=pl.Series(["a", "a", "a", "b", "b", "a", "a"]))
| >>> lf.collect()
| shape: (7, 3)
| ┌─────────────────────┬─────┬────────┐
| │ time ┆ n ┆ groups │
| │ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ str │
| ╞═════════════════════╪═════╪════════╡
| │ 2021-12-16 00:00:00 ┆ 0 ┆ a │
| │ 2021-12-16 00:30:00 ┆ 1 ┆ a │
| │ 2021-12-16 01:00:00 ┆ 2 ┆ a │
| │ 2021-12-16 01:30:00 ┆ 3 ┆ b │
| │ 2021-12-16 02:00:00 ┆ 4 ┆ b │
| │ 2021-12-16 02:30:00 ┆ 5 ┆ a │
| │ 2021-12-16 03:00:00 ┆ 6 ┆ a │
| └─────────────────────┴─────┴────────┘
| >>> lf.group_by_dynamic(
| ... "time",
| ... every="1h",
| ... closed="both",
| ... group_by="groups",
| ... include_boundaries=True,
| ... ).agg(pl.col("n")).collect()
| shape: (6, 5)
| ┌────────┬─────────────────────┬─────────────────────┬─────────────────────┬───────────┐
| │ groups ┆ _lower_boundary ┆ _upper_boundary ┆ time ┆ n │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ list[i64] │
| ╞════════╪═════════════════════╪═════════════════════╪═════════════════════╪═══════════╡
| │ a ┆ 2021-12-16 00:00:00 ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 00:00:00 ┆ [0, 1, 2] │
| │ a ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ [2] │
| │ a ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ [5, 6] │
| │ a ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 04:00:00 ┆ 2021-12-16 03:00:00 ┆ [6] │
| │ b ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ [3, 4] │
| │ b ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ [4] │
| └────────┴─────────────────────┴─────────────────────┴─────────────────────┴───────────┘
|
| Dynamic group by on an index column
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "idx": pl.int_range(0, 6, eager=True),
| ... "A": ["A", "A", "B", "B", "B", "C"],
| ... }
| ... )
| >>> lf.group_by_dynamic(
| ... "idx",
| ... every="2i",
| ... period="3i",
| ... include_boundaries=True,
| ... closed="right",
| ... ).agg(pl.col("A").alias("A_agg_list")).collect()
| shape: (4, 4)
| ┌─────────────────┬─────────────────┬─────┬─────────────────┐
| │ _lower_boundary ┆ _upper_boundary ┆ idx ┆ A_agg_list │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ list[str] │
| ╞═════════════════╪═════════════════╪═════╪═════════════════╡
| │ -2 ┆ 1 ┆ -2 ┆ ["A", "A"] │
| │ 0 ┆ 3 ┆ 0 ┆ ["A", "B", "B"] │
| │ 2 ┆ 5 ┆ 2 ┆ ["B", "B", "C"] │
| │ 4 ┆ 7 ┆ 4 ┆ ["C"] │
| └─────────────────┴─────────────────┴─────┴─────────────────┘
|
| head(self, n: 'int' = 5) -> 'LazyFrame'
| Get the first `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [7, 8, 9, 10, 11, 12],
| ... }
| ... )
| >>> lf.head().collect()
| shape: (5, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| │ 4 ┆ 10 │
| │ 5 ┆ 11 │
| └─────┴─────┘
| >>> lf.head(2).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| └─────┴─────┘
|
| inspect(self, fmt: 'str' = '{}') -> 'LazyFrame'
| Inspect a node in the computation graph.
|
| Print the value that this node in the computation graph evaluates to and pass on
| the value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"foo": [1, 1, -2, 3]})
| >>> (
| ... lf.with_columns(pl.col("foo").cum_sum().alias("bar"))
| ... .inspect() # print the node before the filter
| ... .filter(pl.col("bar") == pl.col("foo"))
| ... )
| <LazyFrame at ...>
|
| interpolate(self) -> 'LazyFrame'
| Interpolate intermediate values. The interpolation method is linear.
|
| Nulls at the beginning and end of the series remain null.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, None, 9, 10],
| ... "bar": [6, 7, 9, None],
| ... "baz": [1, None, None, 9],
| ... }
| ... )
| >>> lf.interpolate().collect()
| shape: (4, 3)
| ┌──────┬──────┬──────────┐
| │ foo ┆ bar ┆ baz │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪══════════╡
| │ 1.0 ┆ 6.0 ┆ 1.0 │
| │ 5.0 ┆ 7.0 ┆ 3.666667 │
| │ 9.0 ┆ 9.0 ┆ 6.333333 │
| │ 10.0 ┆ null ┆ 9.0 │
| └──────┴──────┴──────────┘
|
| join(self, other: 'LazyFrame', on: 'str | Expr | Sequence[str | Expr] | None' = None, how: 'JoinStrategy' = 'inner', *, left_on: 'str | Expr | Sequence[str | Expr] | None' = None, right_on: 'str | Expr | Sequence[str | Expr] | None' = None, suffix: 'str' = '_right', validate: 'JoinValidation' = 'm:m', nulls_equal: 'bool' = False, coalesce: 'bool | None' = None, maintain_order: 'MaintainOrderJoin | None' = None, build_side: 'JoinBuildSide' = 'auto', allow_parallel: 'bool' = True, force_parallel: 'bool' = False) -> 'LazyFrame'
| Add a join operation to the Logical Plan.
|
| .. versionchanged:: 1.24
| The `join_nulls` parameter was renamed `nulls_equal`.
|
| Parameters
| ----------
| other
| Lazy DataFrame to join with.
| on
| Name(s) of the join columns in both DataFrames. If set, `left_on` and
| `right_on` should be None. This should not be specified if `how='cross'`.
| how : {'inner','left', 'right', 'full', 'semi', 'anti', 'cross'}
| Join strategy.
|
| .. list-table ::
| :header-rows: 0
|
| * - **inner**
| - *(Default)* Returns rows that have matching values in both tables.
| * - **left**
| - Returns all rows from the left table, and the matched rows from
| the right table.
| * - **right**
| - Returns all rows from the right table, and the matched rows from
| the left table.
| * - **full**
| - Returns all rows from both tables, joining matching rows and
| filling non-matches with null values.
| * - **cross**
| - Returns the Cartesian product of rows from both tables
| * - **semi**
| - Returns rows from the left table that have a match in the right
| table. Does not return columns from the right table.
| * - **anti**
| - Returns rows from the left table that have no match in the right
| table. Does not return columns from the right table.
|
| left_on
| Join column of the left DataFrame.
| right_on
| Join column of the right DataFrame.
| suffix
| Suffix to append to columns with a duplicate name.
| validate: {'m:m', 'm:1', '1:m', '1:1'}
| Checks if join is of specified type.
|
| .. list-table ::
| :header-rows: 0
|
| * - **m:m**
| - *(Default)* Many-to-many. Does not result in checks.
| * - **1:1**
| - One-to-one. Checks if join keys are unique in both left and
| right datasets.
| * - **1:m**
| - One-to-many. Checks if join keys are unique in left dataset.
| * - **m:1**
| - Many-to-one. Check if join keys are unique in right dataset.
| nulls_equal
| Join on null values. By default null values will never produce matches.
| coalesce
| Coalescing behavior (merging of join columns).
|
| .. list-table ::
| :header-rows: 0
|
| * - **None**
| - *(Default)* Coalesce unless `how='full'` is specified.
| * - **True**
| - Always coalesce join columns.
| * - **False**
| - Never coalesce join columns.
|
| .. note::
| Joining on any other expressions than `col`
| will turn off coalescing.
| maintain_order : {'none', 'left', 'right', 'left_right', 'right_left'}
| Which DataFrame row order to preserve, if any.
| Do not rely on any observed ordering without explicitly setting this
| parameter, as your code may break in a future release.
| Not specifying any ordering can improve performance.
|
| .. list-table ::
| :header-rows: 0
|
| * - **none**
| - *(Default)* No specific ordering is desired. The ordering might
| differ across Polars versions or even between different runs.
| * - **left**
| - Preserves the order of the left DataFrame.
| * - **right**
| - Preserves the order of the right DataFrame.
| * - **left_right**
| - First preserves the order of the left DataFrame, then the right.
| * - **right_left**
| - First preserves the order of the right DataFrame, then the left.
| build_side: {'auto', 'prefer_left', 'prefer_right', 'force_left', 'force_right'}
| Which side of the join will be used as the build side. This side will be
| likely be held in memory as a hash table. Note that unless a `force_`
| variant is chosen, the chosen side might differ across Polars versions or
| even between different runs.
|
| .. list-table ::
| :header-rows: 0
|
| * - **auto**
| - *(Default)* Let Polars figure out the build side.
| * - **prefer_left**
| - Unless there's a very good reason to believe that the right side is
| smaller, use the left side.
| * - **prefer_right**
| - Unless there's a very good reason to believe that the left side is
| smaller, use the right side.
| * - **force_left**
| - Always use the left side.
| * - **force_right**
| - Always use the right side.
|
| .. warning::
| This functionality is considered **experimental**. It may be removed or
| changed at any point without it being considered a breaking change.
|
| allow_parallel
| Allow the physical plan to optionally evaluate the computation of both
| DataFrames up to the join in parallel.
| force_parallel
| Force the physical plan to evaluate the computation of both DataFrames up to
| the join in parallel.
|
| See Also
| --------
| join_asof
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> other_lf = pl.LazyFrame(
| ... {
| ... "apple": ["x", "y", "z"],
| ... "ham": ["a", "b", "d"],
| ... }
| ... )
| >>> lf.join(other_lf, on="ham").collect()
| shape: (2, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| └─────┴─────┴─────┴───────┘
| >>> lf.join(other_lf, on="ham", how="full").collect()
| shape: (4, 5)
| ┌──────┬──────┬──────┬───────┬───────────┐
| │ foo ┆ bar ┆ ham ┆ apple ┆ ham_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞══════╪══════╪══════╪═══════╪═══════════╡
| │ 1 ┆ 6.0 ┆ a ┆ x ┆ a │
| │ 2 ┆ 7.0 ┆ b ┆ y ┆ b │
| │ null ┆ null ┆ null ┆ z ┆ d │
| │ 3 ┆ 8.0 ┆ c ┆ null ┆ null │
| └──────┴──────┴──────┴───────┴───────────┘
| >>> lf.join(other_lf, on="ham", how="left", coalesce=True).collect()
| shape: (3, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| │ 3 ┆ 8.0 ┆ c ┆ null │
| └─────┴─────┴─────┴───────┘
| >>> lf.join(other_lf, on="ham", how="semi").collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 7.0 ┆ b │
| └─────┴─────┴─────┘
| >>> lf.join(other_lf, on="ham", how="anti").collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| >>> lf.join(other_lf, how="cross").collect()
| shape: (9, 5)
| ┌─────┬─────┬─────┬───────┬───────────┐
| │ foo ┆ bar ┆ ham ┆ apple ┆ ham_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╪═══════════╡
| │ 1 ┆ 6.0 ┆ a ┆ x ┆ a │
| │ 1 ┆ 6.0 ┆ a ┆ y ┆ b │
| │ 1 ┆ 6.0 ┆ a ┆ z ┆ d │
| │ 2 ┆ 7.0 ┆ b ┆ x ┆ a │
| │ 2 ┆ 7.0 ┆ b ┆ y ┆ b │
| │ 2 ┆ 7.0 ┆ b ┆ z ┆ d │
| │ 3 ┆ 8.0 ┆ c ┆ x ┆ a │
| │ 3 ┆ 8.0 ┆ c ┆ y ┆ b │
| │ 3 ┆ 8.0 ┆ c ┆ z ┆ d │
| └─────┴─────┴─────┴───────┴───────────┘
|
| join_asof(self, other: 'LazyFrame', *, left_on: 'str | None | Expr' = None, right_on: 'str | None | Expr' = None, on: 'str | None | Expr' = None, by_left: 'str | Sequence[str] | None' = None, by_right: 'str | Sequence[str] | None' = None, by: 'str | Sequence[str] | None' = None, strategy: 'AsofJoinStrategy' = 'backward', suffix: 'str' = '_right', tolerance: 'str | int | float | timedelta | None' = None, allow_parallel: 'bool' = True, force_parallel: 'bool' = False, coalesce: 'bool' = True, allow_exact_matches: 'bool' = True, check_sortedness: 'bool' = True) -> 'LazyFrame'
| Perform an asof join.
|
| This is similar to a left-join except that we match on nearest key rather than
| equal keys.
|
| Both DataFrames must be sorted by the `on` key (within each `by` group, if
| specified).
|
| For each row in the left DataFrame:
|
| - A "backward" search selects the last row in the right DataFrame whose
| 'on' key is less than or equal to the left's key.
|
| - A "forward" search selects the first row in the right DataFrame whose
| 'on' key is greater than or equal to the left's key.
|
| A "nearest" search selects the last row in the right DataFrame whose value
| is nearest to the left's key. String keys are not currently supported for a
| nearest search.
|
| The default is "backward".
|
| Parameters
| ----------
| other
| Lazy DataFrame to join with.
| left_on
| Join column of the left DataFrame.
| right_on
| Join column of the right DataFrame.
| on
| Join column of both DataFrames. If set, `left_on` and `right_on` should be
| None.
| by_left
| Join on these columns before doing asof join.
| by_right
| Join on these columns before doing asof join.
| by
| Join on these columns before doing asof join.
| strategy : {'backward', 'forward', 'nearest'}
| Join strategy.
| suffix
| Suffix to append to columns with a duplicate name.
| tolerance
| Numeric tolerance. By setting this the join will only be done if the near
| keys are within this distance. If an asof join is done on columns of dtype
| "Date", "Datetime", "Duration" or "Time", use either a datetime.timedelta
| object or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of
| ambiguity, we follow RFC-5545 and preserve the DST fold of the original
| datetime). Similarly for "calendar week", "calendar month",
| "calendar quarter", and "calendar year".
|
| allow_parallel
| Allow the physical plan to optionally evaluate the computation of both
| DataFrames up to the join in parallel.
| force_parallel
| Force the physical plan to evaluate the computation of both DataFrames up to
| the join in parallel.
| coalesce
| Coalescing behavior (merging of `on` / `left_on` / `right_on` columns):
|
| - True: -> Always coalesce join columns.
| - False: -> Never coalesce join columns.
|
| Note that joining on any other expressions than `col`
| will turn off coalescing.
| allow_exact_matches
| Whether exact matches are valid join predicates.
|
| - If True, allow matching with the same ``on`` value
| (i.e. less-than-or-equal-to / greater-than-or-equal-to)
| - If False, don't match the same ``on`` value
| (i.e., strictly less-than / strictly greater-than).
| check_sortedness
| Check the sortedness of the asof keys. If the keys are not sorted Polars
| will error. Currently, the `in-memory` engine cannot check the sortedness
| if 'by' groups are provided. The `streaming` engine will only check the
| sortedness of the rows it processes.
|
| Notes
| -----
| If 'by' is set, the implementation will compute the asof join over all of the
| groups concurrently. This can potentially lead to high memory usage if there
| are many groups.
|
| This can be mitigated by sorting (via :meth:`.sort() <polars.LazyFrame.sort>`)
| both of the input LazyFrames by the 'by' keys (or using :meth:`.set_sorted()
| <polars.LazyFrame.set_sorted>` if the columns are already sorted)
| before computing the join operation; and using the streaming engine to collect
| the results. For example:
|
| >>> # Compute streaming asof join with 'by' groups
| >>> result = (
| ... left.sort("by", "on").join_asof( # Sort left manually
| ... right.set_sorted("by", "on"), # Set right as already sorted
| ... )
| ... ).collect(streaming=True) # doctest: +SKIP
|
| Examples
| --------
| >>> from datetime import date
| >>> gdp = pl.LazyFrame(
| ... {
| ... "date": pl.date_range(
| ... date(2016, 1, 1),
| ... date(2020, 1, 1),
| ... "1y",
| ... eager=True,
| ... ),
| ... "gdp": [4164, 4411, 4566, 4696, 4827],
| ... }
| ... )
| >>> gdp.collect()
| shape: (5, 2)
| ┌────────────┬──────┐
| │ date ┆ gdp │
| │ --- ┆ --- │
| │ date ┆ i64 │
| ╞════════════╪══════╡
| │ 2016-01-01 ┆ 4164 │
| │ 2017-01-01 ┆ 4411 │
| │ 2018-01-01 ┆ 4566 │
| │ 2019-01-01 ┆ 4696 │
| │ 2020-01-01 ┆ 4827 │
| └────────────┴──────┘
|
| >>> population = pl.LazyFrame(
| ... {
| ... "date": [date(2016, 3, 1), date(2018, 8, 1), date(2019, 1, 1)],
| ... "population": [82.19, 82.66, 83.12],
| ... }
| ... ).sort("date")
| >>> population.collect()
| shape: (3, 2)
| ┌────────────┬────────────┐
| │ date ┆ population │
| │ --- ┆ --- │
| │ date ┆ f64 │
| ╞════════════╪════════════╡
| │ 2016-03-01 ┆ 82.19 │
| │ 2018-08-01 ┆ 82.66 │
| │ 2019-01-01 ┆ 83.12 │
| └────────────┴────────────┘
|
| Note how the dates don't quite match. If we join them using `join_asof` and
| `strategy='backward'`, then each date from `population` which doesn't have an
| exact match is matched with the closest earlier date from `gdp`:
|
| >>> population.join_asof(gdp, on="date", strategy="backward").collect()
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 4566 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2016-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2018-01-01` from `gdp`.
|
| You can verify this by passing `coalesce=False`:
|
| >>> population.join_asof(
| ... gdp, on="date", strategy="backward", coalesce=False
| ... ).collect()
| shape: (3, 4)
| ┌────────────┬────────────┬────────────┬──────┐
| │ date ┆ population ┆ date_right ┆ gdp │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ date ┆ i64 │
| ╞════════════╪════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 2016-01-01 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 2018-01-01 ┆ 4566 │
| │ 2019-01-01 ┆ 83.12 ┆ 2019-01-01 ┆ 4696 │
| └────────────┴────────────┴────────────┴──────┘
|
| If we instead use `strategy='forward'`, then each date from `population` which
| doesn't have an exact match is matched with the closest later date from `gdp`:
|
| >>> population.join_asof(gdp, on="date", strategy="forward").collect()
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4411 │
| │ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2017-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2019-01-01` from `gdp`.
|
| Finally, `strategy='nearest'` gives us a mix of the two results above, as each
| date from `population` which doesn't have an exact match is matched with the
| closest date from `gdp`, regardless of whether it's earlier or later:
|
| >>> population.join_asof(gdp, on="date", strategy="nearest").collect()
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2016-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2019-01-01` from `gdp`.
|
| They `by` argument allows joining on another column first, before the asof join.
| In this example we join by `country` first, then asof join by date, as above.
|
| >>> gdp_dates = pl.date_range( # fmt: skip
| ... date(2016, 1, 1), date(2020, 1, 1), "1y", eager=True
| ... )
| >>> gdp2 = pl.LazyFrame(
| ... {
| ... "country": ["Germany"] * 5 + ["Netherlands"] * 5,
| ... "date": pl.concat([gdp_dates, gdp_dates]),
| ... "gdp": [4164, 4411, 4566, 4696, 4827, 784, 833, 914, 910, 909],
| ... }
| ... ).sort("country", "date")
| >>>
| >>> gdp2.collect()
| shape: (10, 3)
| ┌─────────────┬────────────┬──────┐
| │ country ┆ date ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 │
| ╞═════════════╪════════════╪══════╡
| │ Germany ┆ 2016-01-01 ┆ 4164 │
| │ Germany ┆ 2017-01-01 ┆ 4411 │
| │ Germany ┆ 2018-01-01 ┆ 4566 │
| │ Germany ┆ 2019-01-01 ┆ 4696 │
| │ Germany ┆ 2020-01-01 ┆ 4827 │
| │ Netherlands ┆ 2016-01-01 ┆ 784 │
| │ Netherlands ┆ 2017-01-01 ┆ 833 │
| │ Netherlands ┆ 2018-01-01 ┆ 914 │
| │ Netherlands ┆ 2019-01-01 ┆ 910 │
| │ Netherlands ┆ 2020-01-01 ┆ 909 │
| └─────────────┴────────────┴──────┘
| >>> pop2 = pl.LazyFrame(
| ... {
| ... "country": ["Germany"] * 3 + ["Netherlands"] * 3,
| ... "date": [
| ... date(2016, 3, 1),
| ... date(2018, 8, 1),
| ... date(2019, 1, 1),
| ... date(2016, 3, 1),
| ... date(2018, 8, 1),
| ... date(2019, 1, 1),
| ... ],
| ... "population": [82.19, 82.66, 83.12, 17.11, 17.32, 17.40],
| ... }
| ... ).sort("country", "date")
| >>>
| >>> pop2.collect()
| shape: (6, 3)
| ┌─────────────┬────────────┬────────────┐
| │ country ┆ date ┆ population │
| │ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ f64 │
| ╞═════════════╪════════════╪════════════╡
| │ Germany ┆ 2016-03-01 ┆ 82.19 │
| │ Germany ┆ 2018-08-01 ┆ 82.66 │
| │ Germany ┆ 2019-01-01 ┆ 83.12 │
| │ Netherlands ┆ 2016-03-01 ┆ 17.11 │
| │ Netherlands ┆ 2018-08-01 ┆ 17.32 │
| │ Netherlands ┆ 2019-01-01 ┆ 17.4 │
| └─────────────┴────────────┴────────────┘
| >>> pop2.join_asof(gdp2, by="country", on="date", strategy="nearest").collect()
| shape: (6, 4)
| ┌─────────────┬────────────┬────────────┬──────┐
| │ country ┆ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ f64 ┆ i64 │
| ╞═════════════╪════════════╪════════════╪══════╡
| │ Germany ┆ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ Germany ┆ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ Germany ┆ 2019-01-01 ┆ 83.12 ┆ 4696 │
| │ Netherlands ┆ 2016-03-01 ┆ 17.11 ┆ 784 │
| │ Netherlands ┆ 2018-08-01 ┆ 17.32 ┆ 910 │
| │ Netherlands ┆ 2019-01-01 ┆ 17.4 ┆ 910 │
| └─────────────┴────────────┴────────────┴──────┘
|
| join_where(self, other: 'LazyFrame', *predicates: 'Expr | Iterable[Expr]', suffix: 'str' = '_right') -> 'LazyFrame'
| Perform a join based on one or multiple (in)equality predicates.
|
| This performs an inner join, so only rows where all predicates are true
| are included in the result, and a row from either DataFrame may be included
| multiple times in the result.
|
| .. note::
| The row order of the input DataFrames is not preserved.
|
| .. warning::
| This functionality is experimental. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| other
| DataFrame to join with.
| *predicates
| (In)Equality condition to join the two tables on.
| When a column name occurs in both tables, the proper suffix must
| be applied in the predicate.
| suffix
| Suffix to append to columns with a duplicate name.
|
| Examples
| --------
| Join two lazyframes together based on two predicates which get AND-ed together.
|
| >>> east = pl.LazyFrame(
| ... {
| ... "id": [100, 101, 102],
| ... "dur": [120, 140, 160],
| ... "rev": [12, 14, 16],
| ... "cores": [2, 8, 4],
| ... }
| ... )
| >>> west = pl.LazyFrame(
| ... {
| ... "t_id": [404, 498, 676, 742],
| ... "time": [90, 130, 150, 170],
| ... "cost": [9, 13, 15, 16],
| ... "cores": [4, 2, 1, 4],
| ... }
| ... )
| >>> east.join_where(
| ... west,
| ... pl.col("dur") < pl.col("time"),
| ... pl.col("rev") < pl.col("cost"),
| ... ).collect()
| shape: (5, 8)
| ┌─────┬─────┬─────┬───────┬──────┬──────┬──────┬─────────────┐
| │ id ┆ dur ┆ rev ┆ cores ┆ t_id ┆ time ┆ cost ┆ cores_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╪══════╪══════╪══════╪═════════════╡
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 498 ┆ 130 ┆ 13 ┆ 2 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| └─────┴─────┴─────┴───────┴──────┴──────┴──────┴─────────────┘
|
| To OR them together, use a single expression and the `|` operator.
|
| >>> east.join_where(
| ... west,
| ... (pl.col("dur") < pl.col("time")) | (pl.col("rev") < pl.col("cost")),
| ... ).collect()
| shape: (6, 8)
| ┌─────┬─────┬─────┬───────┬──────┬──────┬──────┬─────────────┐
| │ id ┆ dur ┆ rev ┆ cores ┆ t_id ┆ time ┆ cost ┆ cores_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╪══════╪══════╪══════╪═════════════╡
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 498 ┆ 130 ┆ 13 ┆ 2 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 102 ┆ 160 ┆ 16 ┆ 4 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| └─────┴─────┴─────┴───────┴──────┴──────┴──────┴─────────────┘
|
| last(self) -> 'LazyFrame'
| Get the last row of the DataFrame.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 5, 3],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> lf.last().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 6 │
| └─────┴─────┘
|
| lazy(self) -> 'LazyFrame'
| Return lazy representation, i.e. itself.
|
| Useful for writing code that expects either a :class:`DataFrame` or
| :class:`LazyFrame`. On LazyFrame this is a no-op, and returns the same object.
|
| Returns
| -------
| LazyFrame
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> lf.lazy()
| <LazyFrame at ...>
|
| limit(self, n: 'int' = 5) -> 'LazyFrame'
| Get the first `n` rows.
|
| Alias for :func:`LazyFrame.head`.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [7, 8, 9, 10, 11, 12],
| ... }
| ... )
| >>> lf.limit().collect()
| shape: (5, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| │ 4 ┆ 10 │
| │ 5 ┆ 11 │
| └─────┴─────┘
| >>> lf.limit(2).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| └─────┴─────┘
|
| map_batches(self, function: 'Callable[[DataFrame], DataFrame]', *, predicate_pushdown: 'bool' = False, projection_pushdown: 'bool' = False, slice_pushdown: 'bool' = False, no_optimizations: 'bool | None' = None, schema: 'None | SchemaDict' = None, validate_output_schema: 'bool' = True, streamable: 'bool' = False) -> 'LazyFrame'
| Apply a custom function.
|
| It is important that the function returns a Polars DataFrame.
|
| Parameters
| ----------
| function
| Lambda/ function to apply.
| predicate_pushdown
| Allow predicate pushdown optimization to pass this node.
| projection_pushdown
| Allow projection pushdown optimization to pass this node.
| slice_pushdown
| Allow slice pushdown optimization to pass this node.
| no_optimizations
| .. deprecated:: 1.30.0
| This parameter is deprecated and will be removed in a future version.
| The `_pushdown` parameters now default to `False`, so this parameter
| is no longer needed.
| schema
| Output schema of the function, if set to `None` we assume that the schema
| will remain unchanged by the applied function.
| validate_output_schema
| It is paramount that polars' schema is correct. This flag will ensure that
| the output schema of this function will be checked with the expected schema.
| Setting this to `False` will not do this check, but may lead to hard to
| debug bugs.
| streamable
| Whether the function that is given is eligible to be running with the
| streaming engine. That means that the function must produce the same result
| when it is executed in batches or when it is be executed on the full
| dataset.
|
| Warnings
| --------
| The `schema` of a `LazyFrame` must always be correct. It is up to the caller
| of this function to ensure that this invariant is upheld.
|
| It is important that the optimization flags are correct. If the custom function
| for instance does an aggregation of a column, `predicate_pushdown` should not
| be allowed, as this prunes rows and will influence your aggregation results.
|
| Notes
| -----
| A UDF passed to `map_batches` must be pure, meaning that it cannot modify or
| depend on state other than its arguments.
|
| Examples
| --------
| >>> lf = ( # doctest: +SKIP
| ... pl.LazyFrame(
| ... {
| ... "a": pl.int_range(-100_000, 0, eager=True),
| ... "b": pl.int_range(0, 100_000, eager=True),
| ... }
| ... )
| ... .map_batches(lambda x: 2 * x, streamable=True)
| ... .collect(engine="streaming")
| ... )
| shape: (100_000, 2)
| ┌─────────┬────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════════╪════════╡
| │ -200000 ┆ 0 │
| │ -199998 ┆ 2 │
| │ -199996 ┆ 4 │
| │ -199994 ┆ 6 │
| │ … ┆ … │
| │ -8 ┆ 199992 │
| │ -6 ┆ 199994 │
| │ -4 ┆ 199996 │
| │ -2 ┆ 199998 │
| └─────────┴────────┘
|
| match_to_schema(self, schema: 'SchemaDict | Schema', *, missing_columns: "Literal['insert', 'raise'] | Mapping[str, Literal['insert', 'raise'] | Expr] | Expr" = 'raise', missing_struct_fields: "Literal['insert', 'raise'] | Mapping[str, Literal['insert', 'raise']]" = 'raise', extra_columns: "Literal['ignore', 'raise']" = 'raise', extra_struct_fields: "Literal['ignore', 'raise'] | Mapping[str, Literal['ignore', 'raise']]" = 'raise', integer_cast: "Literal['upcast', 'forbid'] | Mapping[str, Literal['upcast', 'forbid']]" = 'forbid', float_cast: "Literal['upcast', 'forbid'] | Mapping[str, Literal['upcast', 'forbid']]" = 'forbid') -> 'LazyFrame'
| Match or evolve the schema of a LazyFrame into a specific schema.
|
| By default, match_to_schema returns an error if the input schema does not
| exactly match the target schema. It also allows columns to be freely reordered,
| with additional coercion rules available through optional parameters.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| schema
| Target schema to match or evolve to.
| missing_columns
| Raise of insert missing columns from the input with respect to the `schema`.
|
| This can also be an expression per column with what to insert if it is
| missing.
| missing_struct_fields
| Raise of insert missing struct fields from the input with respect to the
| `schema`.
| extra_columns
| Raise of ignore extra columns from the input with respect to the `schema`.
| extra_struct_fields
| Raise of ignore extra struct fields from the input with respect to the
| `schema`.
| integer_cast
| Forbid of upcast for integer columns from the input to the respective column
| in `schema`.
| float_cast
| Forbid of upcast for float columns from the input to the respective column
| in `schema`.
|
| Examples
| --------
| Ensuring the schema matches
|
| >>> lf = pl.LazyFrame({"a": [1, 2, 3], "b": ["A", "B", "C"]})
| >>> lf.match_to_schema({"a": pl.Int64, "b": pl.String}).collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ A │
| │ 2 ┆ B │
| │ 3 ┆ C │
| └─────┴─────┘
| >>> (lf.match_to_schema({"a": pl.Int64}).collect()) # doctest: +SKIP
| polars.exceptions.SchemaError: extra columns in `match_to_schema`: "b"
|
| Adding missing columns
|
| >>> (
| ... pl.LazyFrame({"a": [1, 2, 3]})
| ... .match_to_schema(
| ... {"a": pl.Int64, "b": pl.String},
| ... missing_columns="insert",
| ... )
| ... .collect()
| ... )
| shape: (3, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪══════╡
| │ 1 ┆ null │
| │ 2 ┆ null │
| │ 3 ┆ null │
| └─────┴──────┘
| >>> (
| ... pl.LazyFrame({"a": [1, 2, 3]})
| ... .match_to_schema(
| ... {"a": pl.Int64, "b": pl.String},
| ... missing_columns={"b": pl.col.a.cast(pl.String)},
| ... )
| ... .collect()
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| └─────┴─────┘
|
| Removing extra columns
|
| >>> (
| ... pl.LazyFrame({"a": [1, 2, 3], "b": ["A", "B", "C"]})
| ... .match_to_schema(
| ... {"a": pl.Int64},
| ... extra_columns="ignore",
| ... )
| ... .collect()
| ... )
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Upcasting integers and floats
|
| >>> (
| ... pl.LazyFrame(
| ... {"a": [1, 2, 3], "b": [1.0, 2.0, 3.0]},
| ... schema={"a": pl.Int32, "b": pl.Float32},
| ... )
| ... .match_to_schema(
| ... {"a": pl.Int64, "b": pl.Float64},
| ... integer_cast="upcast",
| ... float_cast="upcast",
| ... )
| ... .collect()
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 1.0 │
| │ 2 ┆ 2.0 │
| │ 3 ┆ 3.0 │
| └─────┴─────┘
|
| max(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their maximum value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.max().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| mean(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their mean value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.mean().collect()
| shape: (1, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════╡
| │ 2.5 ┆ 1.25 │
| └─────┴──────┘
|
| median(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their median value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.median().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 2.5 ┆ 1.0 │
| └─────┴─────┘
|
| melt(self, id_vars: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, value_vars: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, variable_name: 'str | None' = None, value_name: 'str | None' = None, *, streamable: 'bool' = True) -> 'LazyFrame'
| Unpivot a DataFrame from wide to long format.
|
| Optionally leaves identifiers set.
|
| This function is useful to massage a DataFrame into a format where one or more
| columns are identifier variables (id_vars) while all other columns, considered
| measured variables (value_vars), are "unpivoted" to the row axis leaving just
| two non-identifier columns, 'variable' and 'value'.
|
| .. deprecated:: 1.0.0
| Use the :meth:`.unpivot` method instead.
|
| Parameters
| ----------
| id_vars
| Column(s) or selector(s) to use as identifier variables.
| value_vars
| Column(s) or selector(s) to use as values variables; if `value_vars`
| is empty all columns that are not in `id_vars` will be used.
| variable_name
| Name to give to the `variable` column. Defaults to "variable"
| value_name
| Name to give to the `value` column. Defaults to "value"
| streamable
| Allow this node to run in the streaming engine.
| If this runs in streaming, the output of the unpivot operation
| will not have a stable ordering.
|
| merge_sorted(self, other: 'LazyFrame', key: 'str | Sequence[str]', *, maintain_order: 'bool' = False) -> 'LazyFrame'
| Take two sorted DataFrames and merge them by the sorted key.
|
| The output of this operation will also be sorted.
| It is the callers responsibility that the frames
| are sorted in ascending order by the key(s), with null
| keys at the end, otherwise the order of the output
| will not make sense.
|
| The schemas of both LazyFrames must be equal.
|
| Parameters
| ----------
| other
| Other DataFrame that must be merged
| key
| Key column(s) that the frames are sorted by. A single column name or a
| sequence of column names can be passed. When multiple keys are given the
| frames are merged as if sorted by those keys in order.
| maintain_order
| If ``True``, the output is guaranteed to have left-biased ordering
| for equal keys: rows from the left frame appear before rows from
| the right frame when their keys are equal.
|
| Examples
| --------
| >>> df0 = pl.LazyFrame(
| ... {"name": ["steve", "elise", "bob"], "age": [42, 44, 18]}
| ... ).sort("age")
| >>> df0.collect()
| shape: (3, 2)
| ┌───────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═════╡
| │ bob ┆ 18 │
| │ steve ┆ 42 │
| │ elise ┆ 44 │
| └───────┴─────┘
| >>> df1 = pl.LazyFrame(
| ... {"name": ["anna", "megan", "steve", "thomas"], "age": [21, 33, 42, 20]}
| ... ).sort("age")
| >>> df1.collect()
| shape: (4, 2)
| ┌────────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞════════╪═════╡
| │ thomas ┆ 20 │
| │ anna ┆ 21 │
| │ megan ┆ 33 │
| │ steve ┆ 42 │
| └────────┴─────┘
| >>> df0.merge_sorted(df1, key="age").collect()
| shape: (7, 2)
| ┌────────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞════════╪═════╡
| │ bob ┆ 18 │
| │ thomas ┆ 20 │
| │ anna ┆ 21 │
| │ megan ┆ 33 │
| │ steve ┆ 42 │
| │ steve ┆ 42 │
| │ elise ┆ 44 │
| └────────┴─────┘
|
| Multiple keys can be passed to merge frames sorted by a composite key. The
| frames are merged as if sorted by ``key_1``, then ``key_2``.
|
| >>> df0 = pl.LazyFrame({"key_1": [1, 1, 3], "key_2": [1, 4, 2]})
| >>> df1 = pl.LazyFrame({"key_1": [1, 2, 3], "key_2": [2, 1, 1]})
| >>> df0.merge_sorted(df1, key=["key_1", "key_2"]).collect()
| shape: (6, 2)
| ┌───────┬───────┐
| │ key_1 ┆ key_2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪═══════╡
| │ 1 ┆ 1 │
| │ 1 ┆ 2 │
| │ 1 ┆ 4 │
| │ 2 ┆ 1 │
| │ 3 ┆ 1 │
| │ 3 ┆ 2 │
| └───────┴───────┘
|
| Notes
| -----
| Unless ``maintain_order=True``, no guarantee is given over the output
| row order when the key is equal between the both dataframes.
|
| The key(s) must be sorted in ascending order.
|
| min(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their minimum value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.min().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| └─────┴─────┘
|
| null_count(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame as the sum of their null value count.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, None, 3],
| ... "bar": [6, 7, None],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.null_count().collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 1 ┆ 0 │
| └─────┴─────┴─────┘
|
| pipe(self, function: 'Callable[Concatenate[LazyFrame, P], T]', *args: 'P.args', **kwargs: 'P.kwargs') -> 'T'
| Offers a structured way to apply a sequence of user-defined functions (UDFs).
|
| Parameters
| ----------
| function
| Callable; will receive the frame as the first parameter,
| followed by any given args/kwargs.
| *args
| Arguments to pass to the UDF.
| **kwargs
| Keyword arguments to pass to the UDF.
|
| See Also
| --------
| pipe_with_schema
|
| Examples
| --------
| >>> def cast_str_to_int(lf: pl.LazyFrame, col_name: str) -> pl.LazyFrame:
| ... return lf.with_columns(pl.col(col_name).cast(pl.Int64))
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": ["10", "20", "30", "40"],
| ... }
| ... )
| >>> lf.pipe(cast_str_to_int, col_name="b").collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 10 │
| │ 2 ┆ 20 │
| │ 3 ┆ 30 │
| │ 4 ┆ 40 │
| └─────┴─────┘
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "b": [1, 2],
| ... "a": [3, 4],
| ... }
| ... )
| >>> lf.collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ b ┆ a │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
| >>> lf.pipe(lambda lf: lf.select(sorted(lf.collect_schema()))).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 1 │
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| pipe_with_schema(self, function: 'Callable[[LazyFrame, Schema], LazyFrame]') -> 'LazyFrame'
| Allows to alter the lazy frame during the plan stage with the resolved schema.
|
| In contrast to `pipe`, this method does not execute `function` immediately but
| only during the plan stage. This allows using the resolved schema of the input
| to dynamically alter the lazy frame. This also means that any exceptions raised
| by `function` will only be emitted during the plan stage.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| Parameters
| ----------
| function
| Callable; will receive the frame as the first parameter and the resolved
| schema as the second parameter.
|
| See Also
| --------
| pipe
|
| Examples
| --------
| >>> def cast_to_float_if_necessary(
| ... lf: pl.LazyFrame, schema: pl.Schema
| ... ) -> pl.LazyFrame:
| ... required_casts = [
| ... pl.col(name).cast(pl.Float64)
| ... for name, dtype in schema.items()
| ... if not dtype.is_float()
| ... ]
| ... return lf.with_columns(required_casts)
| >>> lf = pl.LazyFrame(
| ... {"a": [1.0, 2.0], "b": ["1.0", "2.5"], "c": [2.0, 3.0]},
| ... schema={"a": pl.Float64, "b": pl.String, "c": pl.Float32},
| ... )
| >>> lf.pipe_with_schema(cast_to_float_if_necessary).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f32 │
| ╞═════╪═════╪═════╡
| │ 1.0 ┆ 1.0 ┆ 2.0 │
| │ 2.0 ┆ 2.5 ┆ 3.0 │
| └─────┴─────┴─────┘
|
| pivot(self, on: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector]', on_columns: 'Sequence[Any] | pl.Series | pl.DataFrame', *, index: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, values: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, aggregate_function: 'PivotAgg | Expr | None' = None, maintain_order: 'bool' = False, separator: 'str' = '_', column_naming: "Literal['auto', 'combine']" = 'auto') -> 'LazyFrame'
| Create a spreadsheet-style pivot table as a DataFrame.
|
| Parameters
| ----------
| on
| The column(s) whose values will be used as the new columns of the output
| DataFrame.
| on_columns
| What value combinations will be considered for the output table.
| index
| The column(s) that remain from the input to the output. The output DataFrame will have one row
| for each unique combination of the `index`'s values.
| If None, all remaining columns not specified on `on` and `values` will be used. At least one
| of `index` and `values` must be specified.
| values
| The existing column(s) of values which will be moved under the new columns from index. If an
| aggregation is specified, these are the values on which the aggregation will be computed.
| If None, all remaining columns not specified on `on` and `index` will be used.
| At least one of `index` and `values` must be specified.
| aggregate_function
| Choose from:
|
| - None: no aggregation takes place, will raise error if multiple values are in group.
| - A predefined aggregate function string, one of
| {'min', 'max', 'first', 'last', 'sum', 'mean', 'median', 'len', 'item'}
| - An expression to do the aggregation. The expression can only access data from the respective
| 'values' columns as generated by pivot, through `pl.element()`.
| maintain_order
| Ensure the values of `index` are sorted by discovery order.
| separator
| Used as separator/delimiter in generated column names in case of multiple
| `values` columns.
| column_naming : {'auto', 'combine'}
| How resulting column names will be constructed.
|
| * 'auto': The default; combine with separator if there are multiple
| `values` columns, otherwise just use the `on_columns` names.
| * 'combine': Always combine the `values` columns' names with
| the `on_columns` names.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Notes
| -----
| In some other frameworks, you might know this operation as `pivot_wider`.
|
| Examples
| --------
| You can use `pivot` to reshape a dataframe from "long" to "wide" format.
|
| For example, suppose we have a dataframe of test scores achieved by some
| students, where each row represents a distinct test.
|
| >>> df = pl.DataFrame(
| ... {
| ... "name": ["Cady", "Cady", "Karen", "Karen"],
| ... "subject": ["maths", "physics", "maths", "physics"],
| ... "test_1": [98, 99, 61, 58],
| ... "test_2": [100, 100, 60, 60],
| ... }
| ... )
| >>> df
| shape: (4, 4)
| ┌───────┬─────────┬────────┬────────┐
| │ name ┆ subject ┆ test_1 ┆ test_2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 ┆ i64 │
| ╞═══════╪═════════╪════════╪════════╡
| │ Cady ┆ maths ┆ 98 ┆ 100 │
| │ Cady ┆ physics ┆ 99 ┆ 100 │
| │ Karen ┆ maths ┆ 61 ┆ 60 │
| │ Karen ┆ physics ┆ 58 ┆ 60 │
| └───────┴─────────┴────────┴────────┘
|
| Using `pivot`, we can reshape so we have one row per student, with different
| subjects as columns, and their `test_1` scores as values:
|
| >>> df.lazy().pivot(
| ... "subject",
| ... on_columns=["maths", "physics"],
| ... index="name",
| ... values="test_1",
| ... ).collect() # doctest: +IGNORE_RESULT
| shape: (2, 3)
| ┌───────┬───────┬─────────┐
| │ name ┆ maths ┆ physics │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════╪═══════╪═════════╡
| │ Cady ┆ 98 ┆ 99 │
| │ Karen ┆ 61 ┆ 58 │
| └───────┴───────┴─────────┘
|
| You can use selectors too - here we include all test scores in the pivoted table:
|
| >>> import polars.selectors as cs
| >>> df.lazy().pivot(
| ... "subject",
| ... on_columns=["maths", "physics"],
| ... values=cs.starts_with("test"),
| ... ).collect() # doctest: +IGNORE_RESULT
| shape: (2, 5)
| ┌───────┬──────────────┬────────────────┬──────────────┬────────────────┐
| │ name ┆ test_1_maths ┆ test_1_physics ┆ test_2_maths ┆ test_2_physics │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═══════╪══════════════╪════════════════╪══════════════╪════════════════╡
| │ Cady ┆ 98 ┆ 99 ┆ 100 ┆ 100 │
| │ Karen ┆ 61 ┆ 58 ┆ 60 ┆ 60 │
| └───────┴──────────────┴────────────────┴──────────────┴────────────────┘
|
| If you end up with multiple values per cell, you can specify how to aggregate
| them with `aggregate_function`:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "ix": [1, 1, 2, 2, 1, 2],
| ... "col": ["a", "a", "a", "a", "b", "b"],
| ... "foo": [0, 1, 2, 2, 7, 1],
| ... "bar": [0, 2, 0, 0, 9, 4],
| ... }
| ... )
| >>> lf.pivot(
| ... "col", on_columns=["a", "b"], index="ix", aggregate_function="sum"
| ... ).collect() # doctest: +IGNORE_RESULT
| shape: (2, 5)
| ┌─────┬───────┬───────┬───────┬───────┐
| │ ix ┆ foo_a ┆ foo_b ┆ bar_a ┆ bar_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═══════╪═══════╪═══════╪═══════╡
| │ 1 ┆ 1 ┆ 7 ┆ 2 ┆ 9 │
| │ 2 ┆ 4 ┆ 1 ┆ 0 ┆ 4 │
| └─────┴───────┴───────┴───────┴───────┘
|
| You can also pass a custom aggregation function using
| :meth:`polars.element`:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "col1": ["a", "a", "a", "b", "b", "b"],
| ... "col2": ["x", "x", "x", "x", "y", "y"],
| ... "col3": [6, 7, 3, 2, 5, 7],
| ... }
| ... )
| >>> lf.pivot(
| ... "col2",
| ... on_columns=["x", "y"],
| ... index="col1",
| ... values="col3",
| ... aggregate_function=pl.element().tanh().mean(),
| ... ).collect() # doctest: +IGNORE_RESULT
| shape: (2, 3)
| ┌──────┬──────────┬──────────┐
| │ col1 ┆ x ┆ y │
| │ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 │
| ╞══════╪══════════╪══════════╡
| │ a ┆ 0.998347 ┆ null │
| │ b ┆ 0.964028 ┆ 0.999954 │
| └──────┴──────────┴──────────┘
|
| profile(self, *, type_coercion: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, no_optimization: 'bool' = False, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, show_plot: 'bool' = False, truncate_nodes: 'int' = 0, figsize: 'tuple[int, int]' = (18, 8), engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, **_kwargs: 'Any') -> 'tuple[DataFrame, DataFrame]'
| Profile a LazyFrame.
|
| .. deprecated:: 1.43.0
| It was made for the older in-memory engine, but from version 2.0,
| Polars uses a streaming engine by default. Due to the concurrent
| nature of the streaming engine, the profiling information from this
| function would be misleading.
|
| This will run the query and return a tuple
| containing the materialized DataFrame and a DataFrame that
| contains profiling information of each node that is executed.
|
| The units of the timings are microseconds.
|
| Parameters
| ----------
| type_coercion
| Do type coercion optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| predicate_pushdown
| Do predicate pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| projection_pushdown
| Do projection pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| simplify_expression
| Run simplify expressions optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| no_optimization
| Turn off (certain) optimizations.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| slice_pushdown
| Slice pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subplan_elim
| Will try to cache branching subplans that occur on self-joins or unions.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subexpr_elim
| Common subexpressions will be cached and reused.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| cluster_with_columns
| Combine sequential independent calls to with_columns
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| collapse_joins
| Collapse a join and filters into a faster join
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| show_plot
| Show a gantt chart of the profiling result
| truncate_nodes
| Truncate the label lengths in the gantt chart to this number of
| characters.
| figsize
| matplotlib figsize of the profiling plot
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort(
| ... "a"
| ... ).profile() # doctest: +SKIP
| (shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘,
| shape: (3, 3)
| ┌─────────────────────────┬───────┬──────┐
| │ node ┆ start ┆ end │
| │ --- ┆ --- ┆ --- │
| │ str ┆ u64 ┆ u64 │
| ╞═════════════════════════╪═══════╪══════╡
| │ optimization ┆ 0 ┆ 5 │
| │ group_by_partitioned(a) ┆ 5 ┆ 470 │
| │ sort(a) ┆ 475 ┆ 1964 │
| └─────────────────────────┴───────┴──────┘)
|
| quantile(self, quantile: 'float | Expr', interpolation: 'QuantileMethod' = 'nearest') -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their quantile value.
|
| Parameters
| ----------
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.quantile(0.7).collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 3.0 ┆ 1.0 │
| └─────┴─────┘
|
| remote(self, context: 'pc.ClientContext | None' = None, *, plan_type: 'pc._typing.PlanTypePreference' = 'dot', n_retries: 'int' = 0, engine: 'pc._typing.Engine' = 'auto', scaling_mode: 'pc._typing.ScalingMode' = 'auto') -> 'pc.LazyFrameRemote'
| Run a query remotely on Polars Cloud.
|
| This allows you to run Polars remotely on
| one or more workers via several strategies
| for distributed compute.
|
| Read more in the `Announcement post <https://pola.rs/posts/polars-cloud-what-we-are-building/>`_
|
| Parameters
| ----------
| context
| Compute context in which queries are executed.
| If none given, it will take the default context.
| plan_type: {'plain', 'dot'}
| Whether to give a dot diagram of a plain text
| version of logical plan.
| n_retries:
| How often a stage should be retried on failure.
| engine: {'auto', 'streaming', 'in-memory'}
| This will serve as a hint that tells Polars which engine
| to prefer. It doesn't have to be respected.
| scaling_mode: {'auto', 'single-node', 'distributed'}
| If set to auto, a query that doesn't explicitly specify
| a scaling mode via `remote().distributed()` or
| `remote().single_node()` will run in distributed mode
| if the cluster has more than 1 node.
|
|
| Examples
| --------
| Run a query on a cloud instance.
|
| >>> lf = pl.LazyFrame([1, 2, 3]).sum()
| >>> in_progress = lf.remote().execute(blocking=False) # doctest: +SKIP
| >>> # do some other work
| >>> in_progress.await_result() # doctest: +SKIP
| shape: (1, 1)
| ┌──────────┐
| │ column_0 │
| │ --- │
| │ i64 │
| ╞══════════╡
| │ 6 │
| └──────────┘
|
| Explicitly run a query distributed.
|
| >>> lf = (
| ... pl.scan_parquet("s3://my_bucket/").group_by("key").agg(pl.sum("values"))
| ... )
| >>> result = lf.remote().distributed().execute() # doctest: +SKIP
| shape: (1, 1)
| ┌──────────┐
| │ column_0 │
| │ --- │
| │ i64 │
| ╞══════════╡
| │ 6 │
| └──────────┘
|
| remove(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]', **constraints: 'Any') -> 'LazyFrame'
| Remove rows, dropping those that match the given predicate expression(s).
|
| The original order of the remaining rows is preserved.
|
| Rows where the filter predicate does not evaluate to True are retained
| (this includes rows where the predicate evaluates as `null`).
|
| Parameters
| ----------
| predicates
| Expression(s) that evaluate to a boolean Series. When multiple predicates
| are provided they are combined using `&` (logical AND), so a row is only
| removed when *every* predicate evaluates to True for that row.
| constraints
| Column filters; use `name = value` to filter columns using the supplied
| value. Each constraint behaves the same as `pl.col(name).eq(value)`,
| and is implicitly joined with the other filter conditions using `&`.
|
| Notes
| -----
| If you are transitioning from Pandas, and performing filter operations based on
| the comparison of two or more columns, please note that in Polars any comparison
| involving `null` values will result in a `null` result, *not* boolean True or
| False. As a result, these rows will not be removed. Ensure that null values
| are handled appropriately to avoid unexpected behaviour (see examples below).
|
| See Also
| --------
| filter
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [2, 3, None, 4, 0],
| ... "bar": [5, 6, None, None, 0],
| ... "ham": ["a", "b", None, "c", "d"],
| ... }
| ... )
|
| Remove rows matching a condition:
|
| >>> lf.remove(
| ... pl.col("bar") >= 5,
| ... ).collect()
| shape: (3, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| │ 0 ┆ 0 ┆ d │
| └──────┴──────┴──────┘
|
| Discard rows based on multiple conditions, combined with and/or operators:
|
| >>> lf.remove(
| ... (pl.col("foo") >= 0) & (pl.col("bar") >= 0),
| ... ).collect()
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| >>> lf.remove(
| ... (pl.col("foo") >= 0) | (pl.col("bar") >= 0),
| ... ).collect()
| shape: (1, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| └──────┴──────┴──────┘
|
| Provide multiple constraints using `*args` syntax:
|
| >>> lf.remove(
| ... pl.col("ham").is_not_null(),
| ... pl.col("bar") >= 0,
| ... ).collect()
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| Provide constraints(s) using `**kwargs` syntax:
|
| >>> lf.remove(foo=0, bar=0).collect()
| shape: (4, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ 2 ┆ 5 ┆ a │
| │ 3 ┆ 6 ┆ b │
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| Remove rows by comparing two columns against each other; in this case, we
| remove rows where the two columns are not equal (using `ne_missing` to
| ensure that null values compare equal):
|
| >>> lf.remove(
| ... pl.col("foo").ne_missing(pl.col("bar")),
| ... ).collect()
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 0 ┆ 0 ┆ d │
| └──────┴──────┴──────┘
|
| rename(self, mapping: 'Mapping[str, str] | Callable[[str], str]', *, strict: 'bool' = True) -> 'LazyFrame'
| Rename column names.
|
| Parameters
| ----------
| mapping
| Key value pairs that map from old name to new name, or a function
| that takes the old name as input and returns the new name.
| strict
| Validate that all column names exist in the current schema,
| and throw an exception if any do not. (Note that this parameter
| is a no-op when passing a function to `mapping`).
|
| See Also
| --------
| Expr.name.replace
|
| Notes
| -----
| If existing names are swapped (e.g. 'A' points to 'B' and 'B' points to 'A'),
| polars will block projection and predicate pushdowns at this node.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.rename({"foo": "apple"}).collect()
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ apple ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═══════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └───────┴─────┴─────┘
| >>> lf.rename(lambda column_name: "c" + column_name[1:]).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ coo ┆ car ┆ cam │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| reverse(self) -> 'LazyFrame'
| Reverse the DataFrame.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "key": ["a", "b", "c"],
| ... "val": [1, 2, 3],
| ... }
| ... )
| >>> lf.reverse().collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ key ┆ val │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ c ┆ 3 │
| │ b ┆ 2 │
| │ a ┆ 1 │
| └─────┴─────┘
|
| rolling(self, index_column: 'IntoExpr', *, period: 'str | timedelta', offset: 'str | timedelta | None' = None, closed: 'ClosedInterval' = 'right', group_by: 'IntoExpr | Iterable[IntoExpr] | None' = None) -> 'LazyGroupBy'
| Create rolling groups based on a temporal or integer column.
|
| Different from a `group_by_dynamic` the windows are now determined by the
| individual values and are not of constant intervals. For constant intervals
| use :func:`LazyFrame.group_by_dynamic`.
|
| If you have a time series `<t_0, t_1, ..., t_n>`, then by default the
| windows created will be
|
| * (t_0 - period, t_0]
| * (t_1 - period, t_1]
| * ...
| * (t_n - period, t_n]
|
| whereas if you pass a non-default `offset`, then the windows will be
|
| * (t_0 + offset, t_0 + offset + period]
| * (t_1 + offset, t_1 + offset + period]
| * ...
| * (t_n + offset, t_n + offset + period]
|
| The `period` and `offset` arguments are created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order (or, if `group_by` is
| specified, then it must be sorted in ascending order within each group).
|
| In case of a rolling group by on indices, dtype needs to be one of
| {UInt32, UInt64, Int32, Int64}. Note that the first three get temporarily
| cast to Int64, so if performance matters use an Int64 column.
| period
| Length of the window - must be non-negative.
| offset
| Offset of the window. Default is `-period`.
| closed : {'right', 'left', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
| group_by
| Also group by this column/these columns
|
| Returns
| -------
| LazyGroupBy
| Object you can call `.agg` on to aggregate by groups, the result
| of which will be sorted by `index_column` (but note that if `group_by`
| columns are passed, it will only be sorted within each group).
|
| See Also
| --------
| group_by_dynamic
|
| Examples
| --------
| >>> dates = [
| ... "2020-01-01 13:45:48",
| ... "2020-01-01 16:42:13",
| ... "2020-01-01 16:45:09",
| ... "2020-01-02 18:12:48",
| ... "2020-01-03 19:45:32",
| ... "2020-01-08 23:16:43",
| ... ]
| >>> df = pl.LazyFrame({"dt": dates, "a": [3, 7, 5, 9, 2, 1]}).with_columns(
| ... pl.col("dt").str.strptime(pl.Datetime).set_sorted()
| ... )
| >>> out = (
| ... df.rolling(index_column="dt", period="2d")
| ... .agg(
| ... pl.sum("a").alias("sum_a"),
| ... pl.min("a").alias("min_a"),
| ... pl.max("a").alias("max_a"),
| ... )
| ... .collect()
| ... )
| >>> out
| shape: (6, 4)
| ┌─────────────────────┬───────┬───────┬───────┐
| │ dt ┆ sum_a ┆ min_a ┆ max_a │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ i64 ┆ i64 │
| ╞═════════════════════╪═══════╪═══════╪═══════╡
| │ 2020-01-01 13:45:48 ┆ 3 ┆ 3 ┆ 3 │
| │ 2020-01-01 16:42:13 ┆ 10 ┆ 3 ┆ 7 │
| │ 2020-01-01 16:45:09 ┆ 15 ┆ 3 ┆ 7 │
| │ 2020-01-02 18:12:48 ┆ 24 ┆ 3 ┆ 9 │
| │ 2020-01-03 19:45:32 ┆ 11 ┆ 2 ┆ 9 │
| │ 2020-01-08 23:16:43 ┆ 1 ┆ 1 ┆ 1 │
| └─────────────────────┴───────┴───────┴───────┘
|
| select(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'LazyFrame'
| Select columns from this LazyFrame.
|
| Parameters
| ----------
| *exprs
| Column(s) to select, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names,
| other non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to select, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Examples
| --------
| Pass the name of a column to select that column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.select("foo").collect()
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Multiple columns can be selected by passing a list of column names.
|
| >>> lf.select(["foo", "bar"]).collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 6 │
| │ 2 ┆ 7 │
| │ 3 ┆ 8 │
| └─────┴─────┘
|
| Multiple columns can also be selected using positional arguments instead of a
| list. Expressions are also accepted.
|
| >>> lf.select(pl.col("foo"), pl.col("bar") + 1).collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| └─────┴─────┘
|
| Use keyword arguments to easily name your expression inputs.
|
| >>> lf.select(
| ... threshold=pl.when(pl.col("foo") > 2).then(10).otherwise(0)
| ... ).collect()
| shape: (3, 1)
| ┌───────────┐
| │ threshold │
| │ --- │
| │ i32 │
| ╞═══════════╡
| │ 0 │
| │ 0 │
| │ 10 │
| └───────────┘
|
| select_seq(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'LazyFrame'
| Select columns from this LazyFrame.
|
| This will run all expression sequentially instead of in parallel.
| Use this when the work per expression is cheap.
|
| Parameters
| ----------
| *exprs
| Column(s) to select, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names,
| other non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to select, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| See Also
| --------
| select
|
| serialize(self, file: 'IOBase | str | Path | None' = None, *, format: 'SerializationFormat' = 'binary') -> 'bytes | str | None'
| Serialize the logical plan of this LazyFrame to a file or string in JSON format.
|
| Parameters
| ----------
| file
| File path to which the result should be written. If set to `None`
| (default), the output is returned as a string instead.
| format
| The format in which to serialize. Options:
|
| - `"binary"`: Serialize to binary format (bytes). This is the default.
| - `"json"`: Serialize to JSON format (string) (deprecated).
|
| See Also
| --------
| LazyFrame.deserialize
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| Serialize the logical plan into a binary representation.
|
| >>> lf = pl.LazyFrame({"a": [1, 2, 3]}).sum()
| >>> bytes = lf.serialize()
|
| The bytes can later be deserialized back into a LazyFrame.
|
| >>> import io
| >>> pl.LazyFrame.deserialize(io.BytesIO(bytes)).collect()
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 6 │
| └─────┘
|
| set_sorted(self, column: 'str | list[str]', *more_columns: 'str', descending: 'bool | list[bool]' = False, nulls_last: 'bool | list[bool]' = False) -> 'LazyFrame'
| Flag a column as sorted.
|
| This can speed up future operations.
|
| Parameters
| ----------
| column
| Column(s) that is sorted
| more_columns
| Columns that are sorted over after `column`.
| descending
| Whether the column is sorted in descending order.
| nulls_last
| Whether the nulls are at the end.
|
| Warnings
| --------
| This can lead to incorrect results if the data is NOT sorted!!
| Use with care!
|
| shift(self, n: 'int | IntoExprColumn' = 1, *, fill_value: 'IntoExpr | None' = None) -> 'LazyFrame'
| Shift values by the given number of indices.
|
| Parameters
| ----------
| n
| Number of indices to shift forward. If a negative value is passed, values
| are shifted in the opposite direction instead.
| fill_value
| Fill the resulting null values with this value. Accepts scalar expression
| input. Non-expression inputs are parsed as literals.
|
| Notes
| -----
| This method is similar to the `LAG` operation in SQL when the value for `n`
| is positive. With a negative value for `n`, it is similar to `LEAD`.
|
| Examples
| --------
| By default, values are shifted forward by one index.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [5, 6, 7, 8],
| ... }
| ... )
| >>> lf.shift().collect()
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ null ┆ null │
| │ 1 ┆ 5 │
| │ 2 ┆ 6 │
| │ 3 ┆ 7 │
| └──────┴──────┘
|
| Pass a negative value to shift in the opposite direction instead.
|
| >>> lf.shift(-2).collect()
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ 3 ┆ 7 │
| │ 4 ┆ 8 │
| │ null ┆ null │
| │ null ┆ null │
| └──────┴──────┘
|
| Specify `fill_value` to fill the resulting null values.
|
| >>> lf.shift(-2, fill_value=100).collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 7 │
| │ 4 ┆ 8 │
| │ 100 ┆ 100 │
| │ 100 ┆ 100 │
| └─────┴─────┘
|
| show(self, limit: 'int | None' = 5, *, ascii_tables: 'bool | None' = None, decimal_separator: 'str | None' = None, thousands_separator: 'str | bool | None' = None, float_precision: 'int | None' = None, fmt_float: 'FloatFmt | None' = None, fmt_str_lengths: 'int | None' = None, fmt_table_cell_list_len: 'int | None' = None, tbl_cell_alignment: 'Alignment | None' = None, tbl_cell_numeric_alignment: 'Alignment | None' = None, tbl_cols: 'int | None' = None, tbl_column_data_type_inline: 'bool | None' = None, tbl_dataframe_shape_below: 'bool | None' = None, tbl_formatting: 'TableFormatNames | None' = None, tbl_hide_column_data_types: 'bool | None' = None, tbl_hide_column_names: 'bool | None' = None, tbl_hide_dtype_separator: 'bool | None' = None, tbl_hide_dataframe_shape: 'bool | None' = None, tbl_width_chars: 'int | None' = None, trim_decimal_zeros: 'bool | None' = True) -> 'None'
| Show the first `n` rows.
|
| Parameters
| ----------
| limit : int
| Number of rows to show. If None is passed, raises a ValueError. This is done
| to match the signature of :func:`DataFrame.show`.
| ascii_tables : bool
| Use ASCII characters to display table outlines. Set False to revert to the
| default UTF8_FULL_CONDENSED formatting style. See
| :func:`Config.set_ascii_tables` for more information.
| decimal_separator : str
| Set the decimal separator character. See
| :func:`Config.set_decimal_separator` for more information.
| thousands_separator : str, bool
| Set the thousands grouping separator character. See
| :func:`Config.set_thousands_separator` for more information.
| float_precision : int
| Number of decimal places to display for floating point values. See
| :func:`Config.set_float_precision` for more information.
| fmt_float : {"mixed", "full"}
| Control how floating point values are displayed. See
| :func:`Config.set_fmt_float` for more information. Supported options are:
|
| * "mixed": Limit the number of decimal places and use scientific notation
| for large/small values.
| * "full": Print the full precision of the floating point number.
|
| fmt_str_lengths : int
| Number of characters to display for string values. See
| :func:`Config.set_fmt_str_lengths` for more information.
| fmt_table_cell_list_len : int
| Number of elements to display for List values. See
| :func:`Config.set_fmt_table_cell_list_len` for more information.
| tbl_cell_alignment : str
| Set table cell alignment. See :func:`Config.set_tbl_cell_alignment` for more
| information. Supported options are:
|
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| tbl_cell_numeric_alignment : str
| Set table cell alignment for numeric columns. See
| :func:`Config.set_tbl_cell_numeric_alignment` for more information.
| Supported options are:
|
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| tbl_cols : int
| Number of columns to display. See :func:`Config.set_tbl_cols` for more
| information.
| tbl_column_data_type_inline : bool
| Moves the data type inline with the column name (to the right, in
| parentheses). See :func:`Config.set_tbl_column_data_type_inline` for more
| information.
| tbl_dataframe_shape_below : bool
| Print the DataFrame shape information below the data when displaying tables.
| See :func:`Config.set_tbl_dataframe_shape_below` for more information.
| tbl_formatting : str
| Set table formatting style. See :func:`Config.set_tbl_formatting` for more
| information. Supported options are:
|
| * "ASCII_FULL": ASCII, with all borders and lines, including row dividers.
| * "ASCII_FULL_CONDENSED": Same as ASCII_FULL, but with dense row spacing.
| * "ASCII_NO_BORDERS": ASCII, no borders.
| * "ASCII_BORDERS_ONLY": ASCII, borders only.
| * "ASCII_BORDERS_ONLY_CONDENSED": ASCII, borders only, dense row spacing.
| * "ASCII_HORIZONTAL_ONLY": ASCII, horizontal lines only.
| * "ASCII_MARKDOWN": Markdown format (ascii ellipses for truncated values).
| * "MARKDOWN": Markdown format (utf8 ellipses for truncated values).
| * "UTF8_FULL": UTF8, with all borders and lines, including row dividers.
| * "UTF8_FULL_CONDENSED": Same as UTF8_FULL, but with dense row spacing.
| * "UTF8_NO_BORDERS": UTF8, no borders.
| * "UTF8_BORDERS_ONLY": UTF8, borders only.
| * "UTF8_HORIZONTAL_ONLY": UTF8, horizontal lines only.
| * "NOTHING": No borders or other lines.
|
| tbl_hide_column_data_types : bool
| Hide table column data types (i64, f64, str etc.). See
| :func:`Config.set_tbl_hide_column_data_types` for more information.
| tbl_hide_column_names : bool
| Hide table column names. See :func:`Config.set_tbl_hide_column_names` for
| more information.
| tbl_hide_dtype_separator : bool
| Hide the '---' separator between the column names and column types. See
| :func:`Config.set_tbl_hide_dtype_separator` for more information.
| tbl_hide_dataframe_shape : bool
| Hide the DataFrame shape information when displaying tables. See
| :func:`Config.set_tbl_hide_dataframe_shape` for more information.
| tbl_width_chars : int
| Set the maximum width of a table in characters. See
| :func:`Config.set_tbl_width_chars` for more information.
| trim_decimal_zeros : bool
| Strip trailing zeros from Decimal data type values. See
| :func:`Config.set_trim_decimal_zeros` for more information.
|
| Warnings
| --------
| * This method does *not* maintain the laziness of the frame, and will `collect`
| the final result. This could potentially be an expensive operation.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [7, 8, 9, 10, 11, 12],
| ... }
| ... )
| >>> lf.show()
| shape: (5, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| │ 4 ┆ 10 │
| │ 5 ┆ 11 │
| └─────┴─────┘
| >>> lf.show(2)
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| └─────┴─────┘
|
| show_graph(self, *, optimized: 'bool' = True, show: 'bool' = True, output_path: 'str | Path | None' = None, raw_output: 'bool' = False, figsize: 'tuple[float, float]' = (16.0, 12.0), type_coercion: 'bool' = True, _type_check: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, engine: 'EngineType' = 'auto', plan_stage: 'PlanStage' = 'ir', _check_order: 'bool' = True, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'str | None'
| Show a plot of the query plan.
|
| Note that Graphviz must be installed to render the visualization (if not
| already present, you can download it here: `<https://graphviz.org/download>`_).
|
| Parameters
| ----------
| optimized
| Optimize the query plan.
| show
| Show the figure.
| output_path
| Write the figure to disk.
| raw_output
| Return dot syntax. This cannot be combined with `show` and/or `output_path`.
| figsize
| Passed to matplotlib if `show == True`.
| type_coercion
| Do type coercion optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| predicate_pushdown
| Do predicate pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| projection_pushdown
| Do projection pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| simplify_expression
| Run simplify expressions optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| slice_pushdown
| Slice pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subplan_elim
| Will try to cache branching subplans that occur on self-joins or unions.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subexpr_elim
| Common subexpressions will be cached and reused.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| cluster_with_columns
| Combine sequential independent calls to with_columns.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| collapse_joins
| Collapse a join and filters into a faster join.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| plan_stage : {'ir', 'physical'}
| Select the stage to display. Currently only the streaming engine has a
| separate physical stage, for the other engines both IR and physical are the
| same.
| optimizations
| The set of the optimizations considered during query optimization.
|
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort(
| ... "a"
| ... ).show_graph() # doctest: +SKIP
|
| sink_batches(self, function: 'Callable[[DataFrame], bool | None]', *, chunk_size: 'int | None' = None, maintain_order: 'bool' = True, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'pl.LazyFrame | None'
| Evaluate the query and call a user-defined function for every ready batch.
|
| This allows streaming results that are larger than RAM in certain cases.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. warning::
| This method is much slower than native sinks. Only use it if you cannot
| implement your logic otherwise.
|
| Parameters
| ----------
| function
| Function to run with a batch that is ready. If the function returns
| `True`, this signals that no more results are needed, allowing for
| early stopping.
| chunk_size
| The number of rows that are buffered before the callback is called.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
| lazy: bool
| Wait to start execution until `collect` is called.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_batches(lambda df: print(df)) # doctest: +SKIP
|
| sink_csv(self, path: 'str | Path | IO[bytes] | IO[str] | PartitionBy', *, include_bom: 'bool' = False, compression: "Literal['uncompressed', 'gzip', 'zstd']" = 'uncompressed', compression_level: 'int | None' = None, check_extension: 'bool' = True, include_header: 'bool' = True, separator: 'str' = ',', line_terminator: 'str' = '\n', quote_char: 'str' = '"', batch_size: 'int' = 1024, datetime_format: 'str | None' = None, date_format: 'str | None' = None, time_format: 'str | None' = None, float_scientific: 'bool | None' = None, float_precision: 'int | None' = None, decimal_comma: 'bool' = False, null_value: 'str | None' = None, quote_style: 'CsvQuoteStyle | None' = None, maintain_order: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, sync_on_close: 'SyncOnCloseMethod | None' = None, mkdir: 'bool' = False, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'LazyFrame | None'
| Evaluate the query in streaming mode and write to a CSV file.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| Parameters
| ----------
| path
| File path to which the file should be written.
| include_bom
| Whether to include UTF-8 BOM in the CSV output.
| compression
| What compression format to use.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| compression_level
| The compression level to use, typically 0-9 or `None` to let the
| engine choose.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| check_extension
| Whether to check if the filename matches the compression settings.
| Will raise an error if compression is set to 'uncompressed' and the
| filename ends in one of (".gz", ".zst", ".zstd") or if
| compression != 'uncompressed' and the file uses an mismatched
| extension. Only applies if file is a path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| include_header
| Whether to include header in the CSV output.
| separator
| Separate CSV fields with this symbol.
| line_terminator
| String used to end each row.
| quote_char
| Byte to use as quoting character.
| batch_size
| Number of rows that will be processed per thread.
| datetime_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate. If no format specified, the default fractional-second
| precision is inferred from the maximum timeunit found in the frame's
| Datetime cols (if any).
| date_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate.
| time_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate.
| float_scientific
| Whether to use scientific form always (true), never (false), or
| automatically (None) for floating-point datatypes.
| float_precision
| Number of decimal places to write, applied to both floating-point
| datatypes.
| decimal_comma
| Use a comma as the decimal separator instead of a point. Floats will be
| encapsulated in quotes if necessary; set the field separator to override.
| null_value
| A string representing null values (defaulting to the empty string).
| quote_style : {'necessary', 'always', 'non_numeric', 'never'}
| Determines the quoting strategy used.
|
| - necessary (default): This puts quotes around fields only when necessary.
| They are necessary when fields contain a quote,
| delimiter or record terminator.
| Quotes are also necessary when writing an empty record
| (which is indistinguishable from a record with one empty field).
| This is the default.
| - always: This puts quotes around every field. Always.
| - never: This never puts quotes around fields, even if that results in
| invalid CSV data (e.g.: by not quoting strings containing the
| separator).
| - non_numeric: This puts quotes around all fields that are non-numeric.
| Namely, when writing a field that does not parse as a valid float
| or integer, then quotes will be used even if they aren`t strictly
| necessary.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| sync_on_close: { None, 'data', 'all' }
| Sync to disk when before closing a file.
|
| * `None` does not sync.
| * `data` syncs the file contents.
| * `all` syncs the file contents and metadata.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| lazy: bool
| Wait to start execution until `collect` is called.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_csv("out.csv") # doctest: +SKIP
|
| Sink to a `BytesIO` object.
|
| >>> import io
| >>> buf = io.BytesIO() # doctest: +SKIP
| >>> pl.LazyFrame({"x": [1, 2, 1]}).sink_csv(buf) # doctest: +SKIP
|
| Split into a hive-partitioning style partition:
|
| >>> pl.LazyFrame({"x": [1, 2, 1], "y": [3, 4, 5]}).sink_csv(
| ... pl.PartitionBy("./out/", key="x"),
| ... mkdir=True
| ... ) # doctest: +SKIP
|
| See Also
| --------
| PartitionBy
|
| sink_delta(self, target: 'str | Path | deltalake.DeltaTable', *, mode: "Literal['error', 'append', 'overwrite', 'ignore', 'merge']" = 'error', storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', delta_write_options: 'dict[str, Any] | None' = None, delta_merge_options: 'dict[str, Any] | None' = None, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'deltalake.table.TableMerger | None'
| Sink DataFrame as delta table.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| target
| URI of a table or a DeltaTable object.
| mode : {'error', 'append', 'overwrite', 'ignore', 'merge'}
| How to handle existing data.
|
| - If 'error', throw an error if the table already exists (default).
| - If 'append', will add new data.
| - If 'overwrite', will replace table with new data.
| - If 'ignore', will not write anything if table already exists.
| - If 'merge', return a `TableMerger` object to merge data from the DataFrame
| with the existing data.
| storage_options
| Extra options for the storage backends supported by `deltalake`.
| For cloud storages, this may include configurations for authentication etc.
|
| - See a list of supported storage options for S3 `here <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html#variants>`__.
| - See a list of supported storage options for GCS `here <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html#variants>`__.
| - See a list of supported storage options for Azure `here <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants>`__.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| delta_write_options
| Additional keyword arguments while writing a Delta lake Table.
| See a list of supported write options `here <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake>`__.
| delta_merge_options
| Keyword arguments which are required to `MERGE` a Delta lake Table.
| See a list of supported merge options `here <https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.merge>`__.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Raises
| ------
| TypeError
| If the DataFrame contains unsupported data types.
| ArrowInvalidError
| If the DataFrame contains data types that could not be cast to their
| primitive type.
| TableNotFoundError
| If the delta table doesn't exist and MERGE action is triggered
|
| Notes
| -----
| The Polars data types :class:`Null` and :class:`Time` are not supported
| by the delta protocol specification and will raise a TypeError. Columns
| using The :class:`Categorical` data type will be converted to
| normal (non-categorical) strings when written.
|
| Polars columns are always nullable. To write data to a delta table with
| non-nullable columns, a custom pyarrow schema has to be passed to the
| `delta_write_options`. See the last example below.
|
| Examples
| --------
| Sink a large than fits into memory dataset to a Delta Lake table.
|
| >>> lf = pl.scan_parquet(
| ... "/path/to/my_larger_than_ram_file.parquet"
| ... ) # doctest: +SKIP
| >>> table_path = "/path/to/delta-table/"
| >>> lf.sink_delta(table_path) # doctest: +SKIP
|
|
| Sink a dataframe to the local filesystem as a Delta Lake table.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> table_path = "/path/to/delta-table/"
| >>> df.lazy().sink_delta(table_path) # doctest: +SKIP
|
| Append data to an existing Delta Lake table on the local filesystem.
| Note that this will fail if the schema of the new data does not match the
| schema of the existing table.
|
| >>> df.lazy().sink_delta(table_path, mode="append") # doctest: +SKIP
|
| Overwrite a Delta Lake table as a new version.
| If the schemas of the new and old data are the same, specifying the
| `schema_mode` is not required.
|
| >>> existing_table_path = "/path/to/delta-table/"
| >>> df.lazy().sink_delta(
| ... existing_table_path,
| ... mode="overwrite",
| ... delta_write_options={"schema_mode": "overwrite"},
| ... ) # doctest: +SKIP
|
| Sink a DataFrame as a Delta Lake table to a cloud object store like S3.
|
| >>> table_path = "s3://bucket/prefix/to/delta-table/"
| >>> df.lazy().sink_delta(
| ... table_path,
| ... storage_options={
| ... "AWS_REGION": "THE_AWS_REGION",
| ... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
| ... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
| ... },
| ... ) # doctest: +SKIP
|
| Sink DataFrame as a Delta Lake table with non-nullable columns.
|
| >>> import pyarrow as pa
| >>> existing_table_path = "/path/to/delta-table/"
| >>> df.lazy().sink_delta(
| ... existing_table_path,
| ... delta_write_options={
| ... "schema": pa.schema([pa.field("foo", pa.int64(), nullable=False)])
| ... },
| ... ) # doctest: +SKIP
|
| Sink DataFrame as a Delta Lake table with zstd compression.
| For all `delta_write_options` keyword arguments, check the deltalake docs
| `here
| <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake>`__,
| and for Writer Properties in particular `here
| <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.WriterProperties>`__.
|
| >>> import deltalake
| >>> df.lazy().sink_delta(
| ... table_path,
| ... delta_write_options={
| ... "writer_properties": deltalake.WriterProperties(compression="zstd"),
| ... },
| ... ) # doctest: +SKIP
|
| Merge the DataFrame with an existing Delta Lake table.
| For all `TableMerger` methods, check the deltalake docs
| `here <https://delta-io.github.io/delta-rs/api/delta_table/delta_table_merger/>`__.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> table_path = "/path/to/delta-table/"
| >>> (
| ... df.lazy()
| ... .sink_delta(
| ... "table_path",
| ... mode="merge",
| ... delta_merge_options={
| ... "predicate": "s.foo = t.foo",
| ... "source_alias": "s",
| ... "target_alias": "t",
| ... },
| ... )
| ... .when_matched_update_all()
| ... .when_not_matched_insert_all()
| ... .execute()
| ... ) # doctest: +SKIP
|
| sink_iceberg(self, target: 'str | pyiceberg.table.Table', *, mode: "Literal['append', 'overwrite']", catalog: 'pyiceberg.catalog.Catalog | polars.io.iceberg.IcebergCatalogConfig | None' = None, storage_options: 'StorageOptionsDict | None' = None) -> 'pl.DataFrame'
| Sink a LazyFrame to an Iceberg table.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| target
| A PyIceberg Table object, or a 'namespace.table_name' identifier string.
| mode : {'append', 'overwrite'}
| How to handle existing data.
|
| - If 'append', will add new data.
| - If 'overwrite', will replace table with new data.
| catalog
| PyIceberg catalog to load the table from if the provided `target`
| was a table identifier.
| storage_options
| Extra options for the storage backends supported by `pyiceberg`.
| For cloud storages, this may include configurations for authentication etc.
|
| More info is available `here <https://py.iceberg.apache.org/configuration/>`__.
|
| Returns
| -------
| DataFrame
| Contains the new metadata path.
|
| sink_ipc(self, path: 'str | Path | IO[bytes] | PartitionBy', *, compression: 'IpcCompression | None' = 'uncompressed', compat_level: 'CompatLevel | None' = None, record_batch_size: 'int | None' = None, maintain_order: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, sync_on_close: 'SyncOnCloseMethod | None' = None, mkdir: 'bool' = False, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, _record_batch_statistics: 'bool' = False) -> 'LazyFrame | None'
| Evaluate the query in streaming mode and write to an IPC file.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| Parameters
| ----------
| path
| File path to which the file should be written.
| compression : {'uncompressed', 'lz4', 'zstd'}
| Choose "zstd" for good compression performance.
| Choose "lz4" for fast compression/decompression.
| compat_level
| Compatibility level to use when exporting Polars data structures.
| The default compatibility level is recommended for most users.
| Use ``pl.CompatLevel.oldest()`` for the most compatible level.
| ``pl.CompatLevel.newest()`` uses the highest supported compatibility
| level, but is considered unstable and may change without it being
| considered a breaking change.
| record_batch_size
| Size of the record batches in number of rows.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| sync_on_close: { None, 'data', 'all' }
| Sync to disk when before closing a file.
|
| * `None` does not sync.
| * `data` syncs the file contents.
| * `all` syncs the file contents and metadata.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| lazy: bool
| Wait to start execution until `collect` is called.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: not currently supported for this sink.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
|
| .. note::
| The GPU engine is currently not supported.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_ipc("out.arrow") # doctest: +SKIP
|
| Sink to a `BytesIO` object.
|
| >>> import io
| >>> buf = io.BytesIO() # doctest: +SKIP
| >>> pl.LazyFrame({"x": [1, 2, 1]}).sink_ipc(buf) # doctest: +SKIP
|
| Split into a hive-partitioning style partition:
|
| >>> pl.LazyFrame({"x": [1, 2, 1], "y": [3, 4, 5]}).sink_ipc(
| ... pl.PartitionBy("./out/", key="x"),
| ... mkdir=True
| ... ) # doctest: +SKIP
|
| See Also
| --------
| PartitionBy
|
| sink_ndjson(self, path: 'str | Path | IO[bytes] | IO[str] | PartitionBy', *, compression: "Literal['uncompressed', 'gzip', 'zstd']" = 'uncompressed', compression_level: 'int | None' = None, check_extension: 'bool' = True, maintain_order: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, sync_on_close: 'SyncOnCloseMethod | None' = None, mkdir: 'bool' = False, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'LazyFrame | None'
| Evaluate the query in streaming mode and write to an NDJSON file.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| Parameters
| ----------
| path
| File path to which the file should be written.
| compression
| What compression format to use.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| compression_level
| The compression level to use, typically 0-9 or `None` to let the
| engine choose.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| check_extension
| Whether to check if the filename matches the compression settings.
| Will raise an error if compression is set to 'uncompressed' and the
| filename ends in one of (".gz", ".zst", ".zstd") or if
| compression != 'uncompressed' and the file uses an mismatched
| extension. Only applies if file is a path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| sync_on_close: { None, 'data', 'all' }
| Sync to disk when before closing a file.
|
| * `None` does not sync.
| * `data` syncs the file contents.
| * `all` syncs the file contents and metadata.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| lazy: bool
| Wait to start execution until `collect` is called.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_ndjson("out.ndjson") # doctest: +SKIP
|
| Sink to a `BytesIO` object.
|
| >>> import io
| >>> buf = io.BytesIO() # doctest: +SKIP
| >>> pl.LazyFrame({"x": [1, 2, 1]}).sink_ndjson(buf) # doctest: +SKIP
|
| Split into a hive-partitioning style partition:
|
| >>> pl.LazyFrame({"x": [1, 2, 1], "y": [3, 4, 5]}).sink_ndjson(
| ... pl.PartitionBy("./out/", key="x"),
| ... mkdir=True
| ... ) # doctest: +SKIP
|
| See Also
| --------
| PartitionBy
|
| sink_parquet(self, path: 'str | Path | IO[bytes] | PartitionBy', *, compression: 'str' = 'zstd', compression_level: 'int | None' = None, statistics: 'bool | str | dict[str, bool]' = True, row_group_size: 'int | None' = None, data_page_size: 'int | None' = None, maintain_order: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, sync_on_close: 'SyncOnCloseMethod | None' = None, metadata: 'ParquetMetadata | None' = None, arrow_schema: 'ArrowSchemaExportable | None' = None, mkdir: 'bool' = False, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, _sinked_paths_callback: 'SinkedPathsCallback | None' = None) -> 'LazyFrame | None'
| Evaluate the query in streaming mode and write to a Parquet file.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| Parameters
| ----------
| path
| File path to which the file should be written.
| compression : {'lz4', 'uncompressed', 'snappy', 'gzip', 'brotli', 'zstd'}
| Choose "zstd" for good compression performance.
| Choose "lz4" for fast compression/decompression.
| Choose "snappy" for more backwards compatibility guarantees
| when you deal with older parquet readers.
| compression_level
| The level of compression to use. Higher compression means smaller files on
| disk.
|
| - "gzip" : min-level: 0, max-level: 9, default: 6.
| - "brotli" : min-level: 0, max-level: 11, default: 1.
| - "zstd" : min-level: 1, max-level: 22, default: 3.
| statistics
| Write statistics to the parquet headers. This is the default behavior.
|
| Possible values:
|
| - `True`: enable default set of statistics (default). Some
| statistics may be disabled.
| - `False`: disable all statistics
| - "full": calculate and write all available statistics.
| - `{ "statistic-key": True / False, ... }`. Available keys:
|
| - "min": column minimum value (default: `True`)
| - "max": column maximum value (default: `True`)
| - "distinct_count": number of unique column values (default: `False`)
| - "null_count": number of null values in column (default: `True`)
| row_group_size
| Size of the row groups in number of rows.
| If None (default), the chunks of the `DataFrame` are
| used. Writing in smaller chunks may reduce memory pressure and improve
| writing speeds.
| data_page_size
| Size limit of individual data pages.
| If not set defaults to 1024 * 1024 bytes
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| sync_on_close: { None, 'data', 'all' }
| Sync to disk when before closing a file.
|
| * `None` does not sync.
| * `data` syncs the file contents.
| * `all` syncs the file contents and metadata.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| metadata
| A dictionary or callback to add key-values to the file-level Parquet
| metadata.
|
| .. warning::
| This functionality is considered **experimental**. It may be removed or
| changed at any point without it being considered a breaking change.
| arrow_schema
| Provide a custom arrow schema to write to the file. This allows
| setting custom schema and field-level metadata. Names and dtypes
| must match.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| lazy: bool
| Wait to start execution until `collect` is called.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_parquet("out.parquet") # doctest: +SKIP
|
| Sink to a `BytesIO` object.
|
| >>> import io
| >>> buf = io.BytesIO() # doctest: +SKIP
| >>> pl.LazyFrame({"x": [1, 2, 1]}).sink_parquet(buf) # doctest: +SKIP
|
| Split into a hive-partitioning style partition:
|
| >>> pl.LazyFrame({"x": [1, 2, 1], "y": [3, 4, 5]}).sink_parquet(
| ... pl.PartitionBy("./out/", key="x"),
| ... mkdir=True
| ... ) # doctest: +SKIP
|
| See Also
| --------
| PartitionBy
|
| slice(self, offset: 'int', length: 'int | None' = None) -> 'LazyFrame'
| Get a slice of this DataFrame.
|
| Parameters
| ----------
| offset
| Start index. Negative indexing is supported.
| length
| Length of the slice. If set to `None`, all rows starting at the offset
| will be selected.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["x", "y", "z"],
| ... "b": [1, 3, 5],
| ... "c": [2, 4, 6],
| ... }
| ... )
| >>> lf.slice(1, 2).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ y ┆ 3 ┆ 4 │
| │ z ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| sort(self, by: 'IntoExpr | Iterable[IntoExpr]', *more_by: 'IntoExpr', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False, maintain_order: 'bool' = False, multithreaded: 'bool' = True) -> 'LazyFrame'
| Sort the LazyFrame by the given columns.
|
| Parameters
| ----------
| by
| Column(s) to sort by. Accepts expression input, including selectors. Strings
| are parsed as column names.
| *more_by
| Additional columns to sort by, specified as positional arguments.
| descending
| Sort in descending order. When sorting by multiple columns, can be specified
| per column by passing a sequence of booleans.
| nulls_last
| Place null values last; can specify a single boolean applying to all columns
| or a sequence of booleans for per-column control.
| maintain_order
| Whether the order should be maintained if elements are equal.
| Note that if `true` streaming is not possible and performance might be
| worse since this requires a stable search.
| multithreaded
| Sort using multiple threads.
|
| Examples
| --------
| Pass a single column name to sort by that column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": [6.0, 5.0, 4.0],
| ... "c": ["a", "c", "b"],
| ... }
| ... )
| >>> lf.sort("a").collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ null ┆ 4.0 ┆ b │
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 5.0 ┆ c │
| └──────┴─────┴─────┘
|
| Sorting by expressions is also supported.
|
| >>> lf.sort(pl.col("a") + pl.col("b") * 2, nulls_last=True).collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 2 ┆ 5.0 ┆ c │
| │ 1 ┆ 6.0 ┆ a │
| │ null ┆ 4.0 ┆ b │
| └──────┴─────┴─────┘
|
| Sort by multiple columns by passing a list of columns.
|
| >>> lf.sort(["c", "a"], descending=True).collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 2 ┆ 5.0 ┆ c │
| │ null ┆ 4.0 ┆ b │
| │ 1 ┆ 6.0 ┆ a │
| └──────┴─────┴─────┘
|
| Or use positional arguments to sort by multiple columns in the same way.
|
| >>> lf.sort("c", "a", descending=[False, True]).collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ null ┆ 4.0 ┆ b │
| │ 2 ┆ 5.0 ┆ c │
| └──────┴─────┴─────┘
|
| sql(self, query: 'str', *, table_name: 'str' = 'self') -> 'LazyFrame'
| Execute a SQL query against the LazyFrame.
|
| .. versionadded:: 0.20.23
|
| .. warning::
| This functionality is considered **unstable**, although it is close to
| being considered stable. It may be changed at any point without it being
| considered a breaking change.
|
| Parameters
| ----------
| query
| SQL query to execute.
| table_name
| Optionally provide an explicit name for the table that represents the
| calling frame (defaults to "self").
|
| Notes
| -----
| * The calling LazyFrame is automatically registered as a table in the SQLContext
| under the name "self". If you want access to the DataFrames and LazyFrames
| found in the current globals, use the top-level :meth:`pl.sql <polars.sql>`.
| * More control over registration and execution behaviour is available by
| using the :class:`SQLContext` object.
|
| See Also
| --------
| SQLContext
|
| Examples
| --------
| >>> lf1 = pl.LazyFrame({"a": [1, 2, 3], "b": [6, 7, 8], "c": ["z", "y", "x"]})
| >>> lf2 = pl.LazyFrame({"a": [3, 2, 1], "d": [125, -654, 888]})
|
| Query the LazyFrame using SQL:
|
| >>> lf1.sql("SELECT c, b FROM self WHERE a > 1").collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ c ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ y ┆ 7 │
| │ x ┆ 8 │
| └─────┴─────┘
|
| Apply SQL transforms (aliasing "self" to "frame") then filter
| natively (you can freely mix SQL and native operations):
|
| >>> lf1.sql(
| ... query='''
| ... SELECT
| ... a,
| ... (a % 2 == 0) AS a_is_even,
| ... (b::float4 / 2) AS "b/2",
| ... CONCAT_WS(':', c, c, c) AS c_c_c
| ... FROM frame
| ... ORDER BY a
| ... ''',
| ... table_name="frame",
| ... ).filter(~pl.col("c_c_c").str.starts_with("x")).collect()
| shape: (2, 4)
| ┌─────┬───────────┬─────┬───────┐
| │ a ┆ a_is_even ┆ b/2 ┆ c_c_c │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ bool ┆ f32 ┆ str │
| ╞═════╪═══════════╪═════╪═══════╡
| │ 1 ┆ false ┆ 3.0 ┆ z:z:z │
| │ 2 ┆ true ┆ 3.5 ┆ y:y:y │
| └─────┴───────────┴─────┴───────┘
|
| std(self, ddof: 'int' = 1) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their standard deviation value.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.std().collect()
| shape: (1, 2)
| ┌──────────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════════╪═════╡
| │ 1.290994 ┆ 0.5 │
| └──────────┴─────┘
| >>> lf.std(ddof=0).collect()
| shape: (1, 2)
| ┌──────────┬──────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════════╪══════════╡
| │ 1.118034 ┆ 0.433013 │
| └──────────┴──────────┘
|
| sum(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their sum value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.sum().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 10 ┆ 5 │
| └─────┴─────┘
|
| tail(self, n: 'int' = 5) -> 'LazyFrame'
| Get the last `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [7, 8, 9, 10, 11, 12],
| ... }
| ... )
| >>> lf.tail().collect()
| shape: (5, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| │ 4 ┆ 10 │
| │ 5 ┆ 11 │
| │ 6 ┆ 12 │
| └─────┴─────┘
| >>> lf.tail(2).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 5 ┆ 11 │
| │ 6 ┆ 12 │
| └─────┴─────┘
|
| top_k(self, k: 'int', *, by: 'IntoExpr | Iterable[IntoExpr]', reverse: 'bool | Sequence[bool]' = False) -> 'LazyFrame'
| Return the `k` largest rows.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| k
| Number of rows to return.
| by
| Column(s) used to determine the top rows.
| Accepts expression input. Strings are parsed as column names.
| reverse
| Consider the `k` smallest elements of the `by` column(s) (instead of the `k`
| largest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| bottom_k
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [2, 1, 1, 3, 2, 1],
| ... }
| ... )
|
| Get the rows which contain the 4 largest values in column b.
|
| >>> lf.top_k(4, by="b").collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 3 │
| │ a ┆ 2 │
| │ b ┆ 2 │
| │ b ┆ 1 │
| └─────┴─────┘
|
| Get the rows which contain the 4 largest values when sorting on column b and a.
|
| >>> lf.top_k(4, by=["b", "a"]).collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 3 │
| │ b ┆ 2 │
| │ a ┆ 2 │
| │ c ┆ 1 │
| └─────┴─────┘
|
| unique(self, subset: 'IntoExpr | Collection[IntoExpr] | None' = None, *, keep: 'UniqueKeepStrategy' = 'any', maintain_order: 'bool' = False) -> 'LazyFrame'
| Drop duplicate rows from this LazyFrame.
|
| Parameters
| ----------
| subset
| Column name(s), selector(s), or expressions to consider when identifying
| duplicate rows. If set to `None` (default), all columns are considered.
| keep : {'first', 'last', 'any', 'none'}
| Which of the duplicate rows to keep.
|
| * 'any': Does not give any guarantee of which row is kept.
| This allows more optimizations.
| * 'none': Don't keep duplicate rows.
| * 'first': Keep the first unique row.
| * 'last': Keep the last unique row.
| maintain_order
| Keep the same order as the original DataFrame. This is more expensive
| to compute. Settings this to `True` blocks the possibility to run on
| the streaming engine.
|
| Returns
| -------
| LazyFrame
| LazyFrame with unique rows.
|
| Notes
| -----
| If you're coming from Pandas, this is similar to
| `pandas.DataFrame.drop_duplicates`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3, 1, 1],
| ... "bar": ["a", "a", "a", "x", "x"],
| ... "ham": ["b", "b", "b", "y", "y"],
| ... }
| ... )
|
| By default, all columns are considered when determining which rows are unique:
|
| >>> lf.unique(maintain_order=True).collect()
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
|
| We can also consider only a subset of columns when determining uniqueness,
| controlling which row we keep when duplicates are found:
|
| >>> lf.unique(subset="foo", keep="first", maintain_order=True).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| └─────┴─────┴─────┘
| >>> lf.unique(subset="foo", keep="last", maintain_order=True).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
| >>> lf.unique(subset="foo", keep="none", maintain_order=True).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| └─────┴─────┴─────┘
|
| Selectors can be used to define the "subset" parameter:
|
| >>> import polars.selectors as cs
| >>> lf.unique(subset=cs.string(), maintain_order=True).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
|
| We can also use an arbitrary expression in the "subset" parameter; in this
| example we use the part of the label in front of ":" to determine uniqueness:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "label": ["xx:1", "xx:2", "yy:3", "yy:4"],
| ... "value": [100, 200, 300, 400],
| ... }
| ... )
| >>> lf.unique(
| ... subset=pl.col("label").str.extract(r"^(\w+):"),
| ... maintain_order=True,
| ... keep="first",
| ... ).collect()
| shape: (2, 2)
| ┌───────┬───────┐
| │ label ┆ value │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═══════╡
| │ xx:1 ┆ 100 │
| │ yy:3 ┆ 300 │
| └───────┴───────┘
|
| unnest(self, columns: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None, *more_columns: 'ColumnNameOrSelector', separator: 'str | None' = None) -> 'LazyFrame'
| Decompose struct columns into separate columns for each of their fields.
|
| The new columns will be inserted into the DataFrame at the location of the
| struct column.
|
| If no columns are provided, all struct columns are unnested.
|
| Parameters
| ----------
| columns
| Name of the struct column(s) that should be unnested.
| *more_columns
| Additional columns to unnest, specified as positional arguments.
| separator
| Rename output column names as combination of the struct column name,
| name separator and field name.
|
| Examples
| --------
| >>> df = pl.LazyFrame(
| ... {
| ... "before": ["foo", "bar"],
| ... "t_a": [1, 2],
| ... "t_b": ["a", "b"],
| ... "t_c": [True, None],
| ... "t_d": [[1, 2], [3]],
| ... "after": ["baz", "womp"],
| ... }
| ... ).select("before", pl.struct(pl.col("^t_.$")).alias("t_struct"), "after")
| >>> df.collect()
| shape: (2, 3)
| ┌────────┬─────────────────────┬───────┐
| │ before ┆ t_struct ┆ after │
| │ --- ┆ --- ┆ --- │
| │ str ┆ struct[4] ┆ str │
| ╞════════╪═════════════════════╪═══════╡
| │ foo ┆ {1,"a",true,[1, 2]} ┆ baz │
| │ bar ┆ {2,"b",null,[3]} ┆ womp │
| └────────┴─────────────────────┴───────┘
| >>> df.unnest("t_struct").collect()
| shape: (2, 6)
| ┌────────┬─────┬─────┬──────┬───────────┬───────┐
| │ before ┆ t_a ┆ t_b ┆ t_c ┆ t_d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪═════╪═════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴─────┴─────┴──────┴───────────┴───────┘
|
| Unnest all struct columns by calling without arguments:
|
| >>> df.unnest().collect()
| shape: (2, 6)
| ┌────────┬─────┬─────┬──────┬───────────┬───────┐
| │ before ┆ t_a ┆ t_b ┆ t_c ┆ t_d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪═════╪═════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴─────┴─────┴──────┴───────────┴───────┘
|
| >>> df = pl.LazyFrame(
| ... {
| ... "before": ["foo", "bar"],
| ... "t_a": [1, 2],
| ... "t_b": ["a", "b"],
| ... "t_c": [True, None],
| ... "t_d": [[1, 2], [3]],
| ... "after": ["baz", "womp"],
| ... }
| ... ).select(
| ... "before",
| ... pl.struct(pl.col("^t_.$").name.map(lambda t: t[2:])).alias("t"),
| ... "after",
| ... )
| >>> df.unnest("t", separator="::").collect()
| shape: (2, 6)
| ┌────────┬──────┬──────┬──────┬───────────┬───────┐
| │ before ┆ t::a ┆ t::b ┆ t::c ┆ t::d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪══════╪══════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴──────┴──────┴──────┴───────────┴───────┘
|
| unpivot(self, on: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, *, index: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, variable_name: 'str | None' = None, value_name: 'str | None' = None, streamable: 'bool' = True) -> 'LazyFrame'
| Unpivot a DataFrame from wide to long format.
|
| Optionally leaves identifiers set.
|
| This function is useful to massage a DataFrame into a format where one or more
| columns are identifier variables (index) while all other columns, considered
| measured variables (on), are "unpivoted" to the row axis leaving just
| two non-identifier columns, 'variable' and 'value'.
|
| Parameters
| ----------
| on
| Column(s) or selector(s) to use as values variables; if `on`
| is empty no columns will be used. If set to `None` (default)
| all columns that are not in `index` will be used.
| index
| Column(s) or selector(s) to use as identifier variables.
| variable_name
| Name to give to the `variable` column. Defaults to "variable"
| value_name
| Name to give to the `value` column. Defaults to "value"
| streamable
| deprecated
|
| Notes
| -----
| If you're coming from pandas, this is similar to `pandas.DataFrame.melt`,
| but with `index` replacing `id_vars` and `on` replacing `value_vars`.
| In other frameworks, you might know this operation as `pivot_longer`.
|
| The resulting row order is unspecified.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["x", "y", "z"],
| ... "b": [1, 3, 5],
| ... "c": [2, 4, 6],
| ... }
| ... )
| >>> import polars.selectors as cs
| >>> lf.unpivot(cs.numeric(), index="a").collect()
| shape: (6, 3)
| ┌─────┬──────────┬───────┐
| │ a ┆ variable ┆ value │
| │ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 │
| ╞═════╪══════════╪═══════╡
| │ x ┆ b ┆ 1 │
| │ y ┆ b ┆ 3 │
| │ z ┆ b ┆ 5 │
| │ x ┆ c ┆ 2 │
| │ y ┆ c ┆ 4 │
| │ z ┆ c ┆ 6 │
| └─────┴──────────┴───────┘
|
| update(self, other: 'LazyFrame', on: 'str | Sequence[str] | None' = None, how: "Literal['left', 'inner', 'full']" = 'left', *, left_on: 'str | Sequence[str] | None' = None, right_on: 'str | Sequence[str] | None' = None, include_nulls: 'bool' = False, maintain_order: 'MaintainOrderJoin | None' = 'left') -> 'LazyFrame'
| Update the values in this `LazyFrame` with the values in `other`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| other
| LazyFrame that will be used to update the values
| on
| Column names that will be joined on. If set to `None` (default),
| the implicit row index of each frame is used as a join key.
| how : {'left', 'inner', 'full'}
| * 'left' will keep all rows from the left table; rows may be duplicated
| if multiple rows in the right frame match the left row's key.
| * 'inner' keeps only those rows where the key exists in both frames.
| * 'full' will update existing rows where the key matches while also
| adding any new rows contained in the given frame.
| left_on
| Join column(s) of the left DataFrame.
| right_on
| Join column(s) of the right DataFrame.
| include_nulls
| Overwrite values in the left frame with null values from the right frame.
| If set to `False` (default), null values in the right frame are ignored.
| maintain_order : {'none', 'left', 'right', 'left_right', 'right_left'}
| Which order of rows from the inputs to preserve. See :func:`~LazyFrame.join`
| for details. Unlike `join` this function preserves the left order by
| default.
|
| Notes
| -----
| This is syntactic sugar for a left/inner join that preserves the order
| of the left `DataFrame` by default, with an optional coalesce when
| `include_nulls = False`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "A": [1, 2, 3, 4],
| ... "B": [400, 500, 600, 700],
| ... }
| ... )
| >>> lf.collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 400 │
| │ 2 ┆ 500 │
| │ 3 ┆ 600 │
| │ 4 ┆ 700 │
| └─────┴─────┘
| >>> new_lf = pl.LazyFrame(
| ... {
| ... "B": [-66, None, -99],
| ... "C": [5, 3, 1],
| ... }
| ... )
|
| Update `df` values with the non-null values in `new_df`, by row index:
|
| >>> lf.update(new_lf).collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -66 │
| │ 2 ┆ 500 │
| │ 3 ┆ -99 │
| │ 4 ┆ 700 │
| └─────┴─────┘
|
| Update `df` values with the non-null values in `new_df`, by row index,
| but only keeping those rows that are common to both frames:
|
| >>> lf.update(new_lf, how="inner").collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -66 │
| │ 2 ┆ 500 │
| │ 3 ┆ -99 │
| └─────┴─────┘
|
| Update `df` values with the non-null values in `new_df`, using a full
| outer join strategy that defines explicit join columns in each frame:
|
| >>> lf.update(new_lf, left_on=["A"], right_on=["C"], how="full").collect()
| shape: (5, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -99 │
| │ 2 ┆ 500 │
| │ 3 ┆ 600 │
| │ 4 ┆ 700 │
| │ 5 ┆ -66 │
| └─────┴─────┘
|
| Update `df` values including null values in `new_df`, using a full
| outer join strategy that defines explicit join columns in each frame:
|
| >>> lf.update(
| ... new_lf, left_on="A", right_on="C", how="full", include_nulls=True
| ... ).collect()
| shape: (5, 2)
| ┌─────┬──────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════╡
| │ 1 ┆ -99 │
| │ 2 ┆ 500 │
| │ 3 ┆ null │
| │ 4 ┆ 700 │
| │ 5 ┆ -66 │
| └─────┴──────┘
|
| var(self, ddof: 'int' = 1) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their variance value.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.var().collect()
| shape: (1, 2)
| ┌──────────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════════╪══════╡
| │ 1.666667 ┆ 0.25 │
| └──────────┴──────┘
| >>> lf.var(ddof=0).collect()
| shape: (1, 2)
| ┌──────┬────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪════════╡
| │ 1.25 ┆ 0.1875 │
| └──────┴────────┘
|
| with_columns(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'LazyFrame'
| Add columns to this LazyFrame.
|
| Added columns will replace existing columns with the same name.
|
| Parameters
| ----------
| *exprs
| Column(s) to add, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names, other
| non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to add, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| LazyFrame
| A new LazyFrame with the columns added.
|
| Notes
| -----
| Creating a new LazyFrame using this method does not create a new copy of
| existing data.
|
| Examples
| --------
| Pass an expression to add it as a new column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> lf.with_columns((pl.col("a") ** 2).alias("a^2")).collect()
| shape: (4, 4)
| ┌─────┬──────┬───────┬─────┐
| │ a ┆ b ┆ c ┆ a^2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 │
| ╞═════╪══════╪═══════╪═════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 │
| │ 2 ┆ 4.0 ┆ true ┆ 4 │
| │ 3 ┆ 10.0 ┆ false ┆ 9 │
| │ 4 ┆ 13.0 ┆ true ┆ 16 │
| └─────┴──────┴───────┴─────┘
|
| Added columns will replace existing columns with the same name.
|
| >>> lf.with_columns(pl.col("a").cast(pl.Float64)).collect()
| shape: (4, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╡
| │ 1.0 ┆ 0.5 ┆ true │
| │ 2.0 ┆ 4.0 ┆ true │
| │ 3.0 ┆ 10.0 ┆ false │
| │ 4.0 ┆ 13.0 ┆ true │
| └─────┴──────┴───────┘
|
| Multiple columns can be added using positional arguments.
|
| >>> lf.with_columns(
| ... (pl.col("a") ** 2).alias("a^2"),
| ... (pl.col("b") / 2).alias("b/2"),
| ... (pl.col("c").not_()).alias("not c"),
| ... ).collect()
| shape: (4, 6)
| ┌─────┬──────┬───────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ a^2 ┆ b/2 ┆ not c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 ┆ 0.25 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 4 ┆ 2.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 9 ┆ 5.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 16 ┆ 6.5 ┆ false │
| └─────┴──────┴───────┴─────┴──────┴───────┘
|
| Multiple columns can also be added by passing a list of expressions.
|
| >>> lf.with_columns(
| ... [
| ... (pl.col("a") ** 2).alias("a^2"),
| ... (pl.col("b") / 2).alias("b/2"),
| ... (pl.col("c").not_()).alias("not c"),
| ... ]
| ... ).collect()
| shape: (4, 6)
| ┌─────┬──────┬───────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ a^2 ┆ b/2 ┆ not c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 ┆ 0.25 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 4 ┆ 2.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 9 ┆ 5.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 16 ┆ 6.5 ┆ false │
| └─────┴──────┴───────┴─────┴──────┴───────┘
|
| Use keyword arguments to easily name your expression inputs.
|
| >>> lf.with_columns(
| ... ab=pl.col("a") * pl.col("b"),
| ... not_c=pl.col("c").not_(),
| ... ).collect()
| shape: (4, 5)
| ┌─────┬──────┬───────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ ab ┆ not_c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 0.5 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 8.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 30.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 52.0 ┆ false │
| └─────┴──────┴───────┴──────┴───────┘
|
| with_columns_seq(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'LazyFrame'
| Add columns to this LazyFrame.
|
| Added columns will replace existing columns with the same name.
|
| This will run all expression sequentially instead of in parallel.
| Use this when the work per expression is cheap.
|
| Parameters
| ----------
| *exprs
| Column(s) to add, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names, other
| non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to add, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| LazyFrame
| A new LazyFrame with the columns added.
|
| See Also
| --------
| with_columns
|
| with_context(self, other: 'Self | list[Self]') -> 'LazyFrame'
| Add an external context to the computation graph.
|
| .. deprecated:: 1.0.0
| Use :func:`concat` instead, with `how='horizontal'`
|
| This allows expressions to also access columns from DataFrames
| that are not part of this one.
|
| Parameters
| ----------
| other
| Lazy DataFrame to join with.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"a": [1, 2, 3], "b": ["a", "c", None]})
| >>> lf_other = pl.LazyFrame({"c": ["foo", "ham"]})
| >>> lf.with_context(lf_other).select( # doctest: +SKIP
| ... pl.col("b") + pl.col("c").first()
| ... ).collect()
| shape: (3, 1)
| ┌──────┐
| │ b │
| │ --- │
| │ str │
| ╞══════╡
| │ afoo │
| │ cfoo │
| │ null │
| └──────┘
|
| Fill nulls with the median from another DataFrame:
|
| >>> train_lf = pl.LazyFrame(
| ... {"feature_0": [-1.0, 0, 1], "feature_1": [-1.0, 0, 1]}
| ... )
| >>> test_lf = pl.LazyFrame(
| ... {"feature_0": [-1.0, None, 1], "feature_1": [-1.0, 0, 1]}
| ... )
| >>> test_lf.with_context( # doctest: +SKIP
| ... train_lf.select(pl.all().name.suffix("_train"))
| ... ).select(
| ... pl.col("feature_0").fill_null(pl.col("feature_0_train").median())
| ... ).collect()
| shape: (3, 1)
| ┌───────────┐
| │ feature_0 │
| │ --- │
| │ f64 │
| ╞═══════════╡
| │ -1.0 │
| │ 0.0 │
| │ 1.0 │
| └───────────┘
|
| with_row_count(self, name: 'str' = 'row_nr', offset: 'int' = 0) -> 'LazyFrame'
| Add a column at index 0 that counts the rows.
|
| .. deprecated:: 0.20.4
| Use the :meth:`with_row_index` method instead.
| Note that the default column name has changed from 'row_nr' to 'index'.
|
| Parameters
| ----------
| name
| Name of the column to add.
| offset
| Start the row count at this offset.
|
| Warnings
| --------
| This can have a negative effect on query performance.
| This may, for instance, block predicate pushdown optimization.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> lf.with_row_count().collect() # doctest: +SKIP
| shape: (3, 3)
| ┌────────┬─────┬─────┐
| │ row_nr ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └────────┴─────┴─────┘
|
| with_row_index(self, name: 'str' = 'index', offset: 'int' = 0) -> 'LazyFrame'
| Add a row index as the first column in the LazyFrame.
|
| Parameters
| ----------
| name
| Name of the index column.
| offset
| Start the index at this offset. Cannot be negative.
|
| Warnings
| --------
| Using this function can have a negative effect on query performance.
| This may, for instance, block predicate pushdown optimization.
|
| Notes
| -----
| The resulting column does not have any special properties. It is a regular
| column of type `UInt32` (or `UInt64` in `polars[rt64]`).
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> lf.with_row_index().collect()
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ index ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞═══════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └───────┴─────┴─────┘
| >>> lf.with_row_index("id", offset=1000).collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ id ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞══════╪═════╪═════╡
| │ 1000 ┆ 1 ┆ 2 │
| │ 1001 ┆ 3 ┆ 4 │
| │ 1002 ┆ 5 ┆ 6 │
| └──────┴─────┴─────┘
|
| An index column can also be created using the expressions :func:`int_range`
| and :func:`len`.
|
| >>> lf.select(
| ... pl.int_range(pl.len(), dtype=pl.UInt32).alias("index"),
| ... pl.all(),
| ... ).collect()
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ index ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞═══════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └───────┴─────┴─────┘
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| deserialize(source: 'str | bytes | Path | IOBase', *, format: 'SerializationFormat' = 'binary') -> 'LazyFrame'
| Read a logical plan from a file to construct a LazyFrame.
|
| Parameters
| ----------
| source
| Path to a file or a file-like object (by file-like object, we refer to
| objects that have a `read()` method, such as a file handler (e.g.
| via builtin `open` function) or `BytesIO`).
| format
| The format with which the LazyFrame was serialized. Options:
|
| - `"binary"`: Deserialize from binary format (bytes). This is the default.
| - `"json"`: Deserialize from JSON format (string).
|
| Warnings
| --------
| This function uses :mod:`pickle` if the logical plan contains Python UDFs,
| and as such inherits the security implications. Deserializing can execute
| arbitrary code, so it should only be attempted on trusted data.
|
| See Also
| --------
| LazyFrame.serialize
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| >>> import io
| >>> lf = pl.LazyFrame({"a": [1, 2, 3]}).sum()
| >>> bytes = lf.serialize()
| >>> pl.LazyFrame.deserialize(io.BytesIO(bytes)).collect()
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 6 │
| └─────┘
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| columns
| Get the column names.
|
| Returns
| -------
| list of str
| A list containing the name of each column in order.
|
| Warnings
| --------
| Determining the column names of a LazyFrame requires resolving its schema,
| which is a potentially expensive operation.
| Using :meth:`collect_schema` is the idiomatic way of resolving the schema.
| This property exists only for symmetry with the DataFrame class.
|
| See Also
| --------
| collect_schema
| Schema.names
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... ).select("foo", "bar")
| >>> lf.columns # doctest: +SKIP
| ['foo', 'bar']
|
| dtypes
| Get the column data types.
|
| Returns
| -------
| list of DataType
| A list containing the data type of each column in order.
|
| Warnings
| --------
| Determining the data types of a LazyFrame requires resolving its schema,
| which is a potentially expensive operation.
| Using :meth:`collect_schema` is the idiomatic way to resolve the schema.
| This property exists only for symmetry with the DataFrame class.
|
| See Also
| --------
| collect_schema
| Schema.dtypes
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.dtypes # doctest: +SKIP
| [Int64, Float64, String]
|
| schema
| Get an ordered mapping of column names to their data type.
|
| Warnings
| --------
| Resolving the schema of a LazyFrame is a potentially expensive operation.
| Using :meth:`collect_schema` is the idiomatic way to resolve the schema.
| This property exists only for symmetry with the DataFrame class.
|
| See Also
| --------
| collect_schema
| Schema
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.schema # doctest: +SKIP
| Schema({'foo': Int64, 'bar': Float64, 'ham': String})
|
| width
| Get the number of columns.
|
| Returns
| -------
| int
|
| Warnings
| --------
| Determining the width of a LazyFrame requires resolving its schema,
| which is a potentially expensive operation.
| Using :meth:`collect_schema` is the idiomatic way to resolve the schema.
| This property exists only for symmetry with the DataFrame class.
|
| See Also
| --------
| collect_schema
| Schema.len
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4, 5, 6],
| ... }
| ... )
| >>> lf.width # doctest: +SKIP
| 2
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set[str]]', '_ldf': 'PyLazy...
|
| __hash__ = None
polars.List
Python Library Documentation: class List in module polars.datatypes.classes
class List(NestedType)
| List(inner: 'PolarsDataType | PythonDataType') -> 'None'
|
| Variable length list type.
|
| Parameters
| ----------
| inner
| The `DataType` of the values within each list.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "integer_lists": [[1, 2], [3, 4]],
| ... "float_lists": [[1.0, 2.0], [3.0, 4.0]],
| ... }
| ... )
| >>> df
| shape: (2, 2)
| ┌───────────────┬─────────────┐
| │ integer_lists ┆ float_lists │
| │ --- ┆ --- │
| │ list[i64] ┆ list[f64] │
| ╞═══════════════╪═════════════╡
| │ [1, 2] ┆ [1.0, 2.0] │
| │ [3, 4] ┆ [3.0, 4.0] │
| └───────────────┴─────────────┘
|
| Method resolution order:
| List
| NestedType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, inner: 'PolarsDataType | PythonDataType') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'inner': 'PolarsDataType'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Null
Python Library Documentation: class Null in module polars.datatypes.classes
class Null(DataType)
| Data type representing null values.
|
| Method resolution order:
| Null
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Object
Python Library Documentation: class Object in module polars.datatypes.classes
class Object(ObjectType)
| Data type for wrapping arbitrary Python objects.
|
| Method resolution order:
| Object
| ObjectType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.PartitionBy
Python Library Documentation: class PartitionBy in module polars.io.partition
class PartitionBy(builtins.object)
| PartitionBy(base_path: 'str | Path', *, file_path_provider: 'Callable[[FileProviderArgs], str | Path | IO[bytes] | IO[str]] | _InternalPlPathProviderConfig | None' = None, key: 'str | Expr | Sequence[str | Expr] | Mapping[str, Expr] | None' = None, include_key: 'bool | None' = None, max_rows_per_file: 'int | None' = None, approximate_bytes_per_file: "int | Literal['auto'] | None" = 'auto') -> 'None'
|
| Configuration for writing to multiple output files.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| base_path
| Base path to write to.
| file_path_provider
| Callable for custom file output paths.
| key
| Expressions to partition by.
| include_key
| Include the partition key expression outputs in the output files.
| max_rows_per_file
| Maximum number of rows to write for each file. Note that files may have
| less than this amount of rows.
| approximate_bytes_per_file
| Approximate number of bytes to write to each file. This is measured as
| the estimated size of the DataFrame in memory.
|
| Examples
| --------
| Split to multiple files partitioned by year:
|
| >>> pl.LazyFrame({"year": [2026, 2027, 1970], "month": [0, 0, 0]}).sink_parquet(
| ... pl.PartitionBy("data/", key="year")
| ... ) # doctest: +SKIP
|
| Split to multiple files based on size:
|
| >>> pl.LazyFrame({"year": [2026, 2027, 1970], "month": [0, 0, 0]}).sink_parquet(
| ... pl.PartitionBy(
| ... "data/", max_rows_per_file=1000, approximate_bytes_per_file=100_000_000
| ... )
| ... ) # doctest: +SKIP
|
| Split to multiple files partitioned by year, with limits on individual file sizes:
|
| >>> pl.LazyFrame({"year": [2026, 2027, 1970], "month": [0, 0, 0]}).sink_parquet(
| ... pl.PartitionBy(
| ... "data/",
| ... key="year",
| ... max_rows_per_file=1000,
| ... approximate_bytes_per_file=100_000_000,
| ... )
| ... ) # doctest: +SKIP
|
| Methods defined here:
|
| __init__(self, base_path: 'str | Path', *, file_path_provider: 'Callable[[FileProviderArgs], str | Path | IO[bytes] | IO[str]] | _InternalPlPathProviderConfig | None' = None, key: 'str | Expr | Sequence[str | Expr] | Mapping[str, Expr] | None' = None, include_key: 'bool | None' = None, max_rows_per_file: 'int | None' = None, approximate_bytes_per_file: "int | Literal['auto'] | None" = 'auto') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.QueryOptFlags
Python Library Documentation: class QueryOptFlags in module polars.lazyframe.opt_flags class QueryOptFlags(builtins.object) | QueryOptFlags(*, predicate_pushdown: 'None | bool' = None, projection_pushdown: 'None | bool' = None, simplify_expression: 'None | bool' = None, slice_pushdown: 'None | bool' = None, comm_subplan_elim: 'None | bool' = None, comm_subexpr_elim: 'None | bool' = None, cluster_with_columns: 'None | bool' = None, collapse_joins: 'None | bool' = None, check_order_observe: 'None | bool' = None, fast_projection: 'None | bool' = None, sort_collapse: 'None | bool' = None, pre_partition_hive: 'None | bool' = None) -> 'None' | | The set of the optimizations considered during query optimization. | | .. warning:: | This functionality is considered **unstable**. It may be changed | at any point without it being considered a breaking change. | | Methods defined here: | | __copy__(self) -> 'QueryOptFlags' | | __deepcopy__(self) -> 'QueryOptFlags' | | __init__(self, *, predicate_pushdown: 'None | bool' = None, projection_pushdown: 'None | bool' = None, simplify_expression: 'None | bool' = None, slice_pushdown: 'None | bool' = None, comm_subplan_elim: 'None | bool' = None, comm_subexpr_elim: 'None | bool' = None, cluster_with_columns: 'None | bool' = None, collapse_joins: 'None | bool' = None, check_order_observe: 'None | bool' = None, fast_projection: 'None | bool' = None, sort_collapse: 'None | bool' = None, pre_partition_hive: 'None | bool' = None) -> 'None' | Initialize self. See help(type(self)) for accurate signature. | | __str__(self) -> 'str' | Return str(self). | | no_optimizations(self) -> 'None' | Remove selected optimizations. | | update(self, *, predicate_pushdown: 'None | bool' = None, projection_pushdown: 'None | bool' = None, simplify_expression: 'None | bool' = None, slice_pushdown: 'None | bool' = None, comm_subplan_elim: 'None | bool' = None, comm_subexpr_elim: 'None | bool' = None, cluster_with_columns: 'None | bool' = None, collapse_joins: 'None | bool' = None, check_order_observe: 'None | bool' = None, fast_projection: 'None | bool' = None, sort_collapse: 'None | bool' = None, pre_partition_hive: 'None | bool' = None) -> 'QueryOptFlags' | Update the current optimization flags. | | ---------------------------------------------------------------------- | Static methods defined here: | | none(*, predicate_pushdown: 'None | bool' = None, projection_pushdown: 'None | bool' = None, simplify_expression: 'None | bool' = None, slice_pushdown: 'None | bool' = None, comm_subplan_elim: 'None | bool' = None, comm_subexpr_elim: 'None | bool' = None, cluster_with_columns: 'None | bool' = None, collapse_joins: 'None | bool' = None, check_order_observe: 'None | bool' = None, fast_projection: 'None | bool' = None, sort_collapse: 'None | bool' = None, pre_partition_hive: 'None | bool' = None) -> 'QueryOptFlags' | Create new empty set off optimizations. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object | | check_order_observe | Do not maintain order if the order would not be observed. | | cluster_with_columns | Cluster sequential `with_columns` calls to independent calls. | | comm_subexpr_elim | Elide duplicate expressions and caches their outputs. | | comm_subplan_elim | Elide duplicate plans and caches their outputs. | | fast_projection | Replace simple projections with a faster inlined projection that skips the expression engine. | | pre_partition_hive | Prepartition hive-partitioned joins on their partition key (requires `predicate_pushdown`). | | predicate_pushdown | Apply predicates/filters as early as possible. | | projection_pushdown | Only read columns that are used later in the query. | | simplify_expression | Run many expression optimization rules until fixed point. | | slice_pushdown | Pushdown slices/limits. | | sort_collapse | Collapse sequential sort nodes into a single sort node.
polars.SQLContext
Python Library Documentation: class SQLContext in module polars.sql.context
class SQLContext(typing.Generic)
| SQLContext(frames: 'Mapping[str, CompatibleFrameType | None] | None' = None, *, register_globals: 'bool | int' = False, eager: 'bool' = False, **named_frames: 'CompatibleFrameType | None') -> 'None'
|
| Run SQL queries against DataFrame, LazyFrame, and Series data.
|
| .. warning::
| This functionality is considered **unstable**, although it is close to being
| considered stable. It may be changed at any point without it being considered
| a breaking change.
|
| Method resolution order:
| SQLContext
| typing.Generic
| builtins.object
|
| Methods defined here:
|
| __enter__(self) -> 'SQLContext[FrameType]'
| Track currently registered tables on scope entry; supports nested scopes.
|
| __exit__(self, exc_type: 'type[BaseException] | None', exc_val: 'BaseException | None', exc_tb: 'TracebackType | None') -> 'None'
| Unregister any tables created within the given scope on context exit.
|
| See Also
| --------
| unregister
|
| __init__(self, frames: 'Mapping[str, CompatibleFrameType | None] | None' = None, *, register_globals: 'bool | int' = False, eager: 'bool' = False, **named_frames: 'CompatibleFrameType | None') -> 'None'
| Initialize a new `SQLContext`.
|
| .. versionchanged:: 0.20.31
| The `eager_execution` parameter was renamed `eager`.
|
| Parameters
| ----------
| frames
| A `{name:frame, ...}` mapping which can include Polars frames *and*
| pandas DataFrames, Series and pyarrow Table and RecordBatch objects.
| register_globals
| Register compatible objects (polars DataFrame, LazyFrame, and Series) found
| in the globals, automatically mapping their variable name to a table name.
| To register other objects (pandas/pyarrow data) pass them explicitly, or
| call the `execute_global` classmethod. If given an integer then only the
| most recent "n" objects found will be registered.
| eager
| If True, returns execution results as `DataFrame` instead of `LazyFrame`.
| (Note that the query itself is always executed in lazy-mode; this parameter
| impacts whether :meth:`execute` returns an eager or lazy result frame).
| **named_frames
| Named eager/lazy frames, provided as kwargs.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"a": [1, 2, 3], "b": ["x", None, "z"]})
| >>> res = pl.SQLContext(frame=lf).execute(
| ... "SELECT b, a*2 AS two_a FROM frame WHERE b IS NOT NULL"
| ... )
| >>> res.collect()
| shape: (2, 2)
| ┌─────┬───────┐
| │ b ┆ two_a │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═══════╡
| │ x ┆ 2 │
| │ z ┆ 6 │
| └─────┴───────┘
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| execute(self, query: 'str', *, eager: 'bool | None' = None) -> 'LazyFrame | DataFrame'
| Parse the given SQL query and execute it against the registered frame data.
|
| Parameters
| ----------
| query
| A valid string SQL query.
| eager
| Apply the query eagerly, returning `DataFrame` instead of `LazyFrame`.
| If unset, the value of the init-time "eager" parameter will be used.
| Note that the query itself is always executed in lazy-mode; this
| parameter only impacts the type of the returned frame.
|
| Examples
| --------
| Declare frame data and register with a SQLContext:
|
| >>> df = pl.DataFrame(
| ... data=[
| ... ("The Godfather", 1972, 6_000_000, 134_821_952, 9.2),
| ... ("The Dark Knight", 2008, 185_000_000, 533_316_061, 9.0),
| ... ("Schindler's List", 1993, 22_000_000, 96_067_179, 8.9),
| ... ("Pulp Fiction", 1994, 8_000_000, 107_930_000, 8.9),
| ... ("The Shawshank Redemption", 1994, 25_000_000, 28_341_469, 9.3),
| ... ],
| ... schema=["title", "release_year", "budget", "gross", "imdb_score"],
| ... orient="row",
| ... )
| >>> ctx = pl.SQLContext(films=df)
|
| Execute a SQL query against the registered frame data:
|
| >>> ctx.execute(
| ... '''
| ... SELECT title, release_year, imdb_score
| ... FROM films
| ... WHERE release_year > 1990
| ... ORDER BY imdb_score DESC
| ... ''',
| ... eager=True,
| ... )
| shape: (4, 3)
| ┌──────────────────────────┬──────────────┬────────────┐
| │ title ┆ release_year ┆ imdb_score │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ f64 │
| ╞══════════════════════════╪══════════════╪════════════╡
| │ The Shawshank Redemption ┆ 1994 ┆ 9.3 │
| │ The Dark Knight ┆ 2008 ┆ 9.0 │
| │ Schindler's List ┆ 1993 ┆ 8.9 │
| │ Pulp Fiction ┆ 1994 ┆ 8.9 │
| └──────────────────────────┴──────────────┴────────────┘
|
| Execute a GROUP BY query:
|
| >>> ctx.execute(
| ... '''
| ... SELECT
| ... MAX(release_year // 10) * 10 AS decade,
| ... SUM(gross) AS total_gross,
| ... COUNT(title) AS n_films,
| ... FROM films
| ... GROUP BY (release_year // 10) -- decade
| ... ORDER BY total_gross DESC
| ... ''',
| ... eager=True,
| ... )
| shape: (3, 3)
| ┌────────┬─────────────┬─────────┐
| │ decade ┆ total_gross ┆ n_films │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ u32 │
| ╞════════╪═════════════╪═════════╡
| │ 2000 ┆ 533316061 ┆ 1 │
| │ 1990 ┆ 232338648 ┆ 3 │
| │ 1970 ┆ 134821952 ┆ 1 │
| └────────┴─────────────┴─────────┘
|
| register(self, name: 'str', frame: 'CompatibleFrameType | None') -> 'Self'
| Register a single frame as a table, using the given name.
|
| Parameters
| ----------
| name
| Name of the table.
| frame
| eager/lazy frame to associate with this table name.
|
| See Also
| --------
| register_globals
| register_many
| unregister
|
| Examples
| --------
| >>> df = pl.DataFrame({"hello": ["world"]})
| >>> ctx = pl.SQLContext()
| >>> ctx.register("frame_data", df).execute("SELECT * FROM frame_data").collect()
| shape: (1, 1)
| ┌───────┐
| │ hello │
| │ --- │
| │ str │
| ╞═══════╡
| │ world │
| └───────┘
|
| register_globals(self, n: 'int | None' = None, *, all_compatible: 'bool' = True) -> 'Self'
| Register all frames (lazy or eager) found in the current globals scope.
|
| Automatically maps variable names to table names.
|
| See Also
| --------
| register
| register_many
| unregister
|
| Parameters
| ----------
| n
| Register only the most recent "n" frames.
| all_compatible
| Control whether we *also* register pandas DataFrame, Series, and
| pyarrow Table and RecordBatch objects. If False, only Polars
| classes are registered with the SQL engine.
|
| Examples
| --------
| >>> df1 = pl.DataFrame({"a": [1, 2, 3], "b": ["x", None, "z"]})
| >>> df2 = pl.DataFrame({"a": [2, 3, 4], "c": ["t", "w", "v"]})
|
| Register frames directly from variables found in the current globals scope:
|
| >>> ctx = pl.SQLContext(register_globals=True)
| >>> ctx.tables()
| ['df1', 'df2']
|
| Query using the register variable/frame names
|
| >>> ctx.execute(
| ... "SELECT a, b, c FROM df1 LEFT JOIN df2 USING (a) ORDER BY a DESC"
| ... ).collect()
| shape: (3, 3)
| ┌─────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪══════╪══════╡
| │ 3 ┆ z ┆ w │
| │ 2 ┆ null ┆ t │
| │ 1 ┆ x ┆ null │
| └─────┴──────┴──────┘
|
| register_many(self, frames: 'Mapping[str, CompatibleFrameType | None] | None' = None, **named_frames: 'CompatibleFrameType | None') -> 'Self'
| Register multiple eager/lazy frames as tables, using the associated names.
|
| Parameters
| ----------
| frames
| A `{name:frame, ...}` mapping.
| **named_frames
| Named eager/lazy frames, provided as kwargs.
|
| See Also
| --------
| register
| register_globals
| unregister
|
| Examples
| --------
| >>> lf1 = pl.LazyFrame({"a": [1, 2, 3], "b": ["m", "n", "o"]})
| >>> lf2 = pl.LazyFrame({"a": [2, 3, 4], "c": ["p", "q", "r"]})
| >>> lf3 = pl.LazyFrame({"a": [3, 4, 5], "b": ["s", "t", "u"]})
| >>> lf4 = pl.LazyFrame({"a": [4, 5, 6], "c": ["v", "w", "x"]})
|
| Register multiple frames at once, either by passing in as a dict...
|
| >>> ctx = pl.SQLContext().register_many({"tbl1": lf1, "tbl2": lf2})
| >>> ctx.tables()
| ['tbl1', 'tbl2']
|
| ...or using keyword args:
|
| >>> ctx.register_many(tbl3=lf3, tbl4=lf4).tables()
| ['tbl1', 'tbl2', 'tbl3', 'tbl4']
|
| tables(self) -> 'list[str]'
| Return a list of the registered table names.
|
| Notes
| -----
| The :meth:`tables` method will return the same values as the
| "SHOW TABLES" SQL statement, but as a list instead of a frame.
|
| Executing as SQL:
|
| >>> frame_data = pl.DataFrame({"hello": ["world"]})
| >>> ctx = pl.SQLContext(hello_world=frame_data)
| >>> ctx.execute("SHOW TABLES", eager=True)
| shape: (1, 1)
| ┌─────────────┐
| │ name │
| │ --- │
| │ str │
| ╞═════════════╡
| │ hello_world │
| └─────────────┘
|
| Calling the method:
|
| >>> ctx.tables()
| ['hello_world']
|
| Examples
| --------
| >>> df1 = pl.DataFrame({"hello": ["world"]})
| >>> df2 = pl.DataFrame({"foo": ["bar", "baz"]})
| >>> ctx = pl.SQLContext(hello_data=df1, foo_bar=df2)
| >>> ctx.tables()
| ['foo_bar', 'hello_data']
|
| unregister(self, names: 'str | Collection[str]') -> 'Self'
| Unregister one or more eager/lazy frames by name.
|
| Parameters
| ----------
| names
| Names of the tables to unregister.
|
| Notes
| -----
| You can also control table registration lifetime by using `SQLContext` as a
| context manager; this can often be more useful when such control is wanted:
|
| >>> df0 = pl.DataFrame({"colx": [0, 1, 2]})
| >>> df1 = pl.DataFrame({"colx": [1, 2, 3]})
| >>> df2 = pl.DataFrame({"colx": [2, 3, 4]})
|
| Frames registered in-scope are automatically unregistered on scope-exit. Note
| that frames registered on construction will persist through subsequent scopes.
|
| >>> # register one frame at construction time, and the other two in-scope
| >>> with pl.SQLContext(tbl0=df0) as ctx:
| ... ctx.register_many(tbl1=df1, tbl2=df2).tables()
| ['tbl0', 'tbl1', 'tbl2']
|
| After scope exit, none of the tables registered in-scope remain:
|
| >>> ctx.tables()
| ['tbl0']
|
| See Also
| --------
| register
| register_globals
| register_many
|
| Examples
| --------
| >>> df0 = pl.DataFrame({"ints": [9, 8, 7, 6, 5]})
| >>> lf1 = pl.LazyFrame({"text": ["a", "b", "c"]})
| >>> lf2 = pl.LazyFrame({"misc": ["testing1234"]})
|
| Register with a SQLContext object:
|
| >>> ctx = pl.SQLContext(test1=df0, test2=lf1, test3=lf2)
| >>> ctx.tables()
| ['test1', 'test2', 'test3']
|
| Unregister one or more of the tables:
|
| >>> ctx.unregister(["test1", "test3"]).tables()
| ['test2']
| >>> ctx.unregister("test2").tables()
| []
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| execute_global(query: 'str', *, eager: 'bool' = False) -> 'DataFrame | LazyFrame'
| Immediately execute a SQL query, automatically registering frame globals.
|
| Notes
| -----
| * This convenience method automatically registers all compatible objects in
| the local stack that are referenced in the query, mapping their variable name
| to a table name. Note that in addition to polars DataFrame, LazyFrame, and
| Series this method *also* registers pandas DataFrame, Series, and pyarrow
| Table and RecordBatch objects.
| * Instead of calling this classmethod you should consider using `pl.sql`,
| which will use this code internally.
|
| Parameters
| ----------
| query
| A valid SQL query string.
| eager
| If True, returns execution results as `DataFrame` instead of `LazyFrame`.
| (Note that the query itself is always executed in lazy-mode).
|
| Examples
| --------
| >>> import pandas as pd
| >>> df = pl.LazyFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
| >>> df_pandas = pd.DataFrame({"a": [2, 3, 4], "c": [7, 8, 9]})
|
| Join a polars LazyFrame with a pandas DataFrame (note use of the preferred
| `pl.sql` method, which is equivalent to `SQLContext.execute_global`):
|
| >>> pl.sql("SELECT df.*, c FROM df JOIN df_pandas USING(a)").collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 2 ┆ 5 ┆ 7 │
| │ 3 ┆ 6 ┆ 8 │
| └─────┴─────┴─────┘
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_ctxt': 'PySQLContext', '_eager_execution': 'Final...
|
| __orig_bases__ = (typing.Generic[~FrameType],)
|
| __parameters__ = (~FrameType,)
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Generic:
|
| __class_getitem__(...)
| Parameterizes a generic class.
|
| At least, parameterizing a generic class is the *main* thing this
| method does. For example, for some generic class `Foo`, this is called
| when we do `Foo[int]` - there, with `cls=Foo` and `params=int`.
|
| However, note that this method is also called when defining generic
| classes in the first place with `class Foo[T]: ...`.
|
| __init_subclass__(...)
| Function to initialize subclasses.
polars.ScanCastOptions
Python Library Documentation: class ScanCastOptions in module polars.io.scan_options.cast_options class ScanCastOptions(builtins.object) | ScanCastOptions(*, integer_cast: 'IntegerCastOption | Collection[IntegerCastOption]' = 'forbid', float_cast: 'FloatCastOption | Collection[FloatCastOption]' = 'forbid', datetime_cast: 'DatetimeCastOption | Collection[DatetimeCastOption]' = 'forbid', missing_struct_fields: "Literal['insert', 'raise']" = 'raise', extra_struct_fields: "Literal['ignore', 'raise']" = 'raise', categorical_to_string: "Literal['allow', 'forbid']" = 'forbid', _internal_call: 'bool' = False) -> 'None' | | Cast options applied when scanning files. | | Methods defined here: | | __init__(self, *, integer_cast: 'IntegerCastOption | Collection[IntegerCastOption]' = 'forbid', float_cast: 'FloatCastOption | Collection[FloatCastOption]' = 'forbid', datetime_cast: 'DatetimeCastOption | Collection[DatetimeCastOption]' = 'forbid', missing_struct_fields: "Literal['insert', 'raise']" = 'raise', extra_struct_fields: "Literal['ignore', 'raise']" = 'raise', categorical_to_string: "Literal['allow', 'forbid']" = 'forbid', _internal_call: 'bool' = False) -> 'None' | Common configuration for scanning files. | | .. warning:: | This functionality is considered **unstable**. It may be changed | at any point without it being considered a breaking change. | | Parameters | ---------- | integer_cast | Configuration for casting from integer types: | | * `upcast`: Allow lossless casting to wider integer types. | * `allow-float`: Allow casting integers to float types. | * `forbid`: Raises an error if dtypes do not match. | | float_cast | Configuration for casting from float types: | | * `upcast`: Allow casting to higher precision float types. | * `downcast`: Allow casting to lower precision float types. | * `forbid`: Raises an error if dtypes do not match. | | datetime_cast | Configuration for casting from datetime types: | | * `nanosecond-downcast`: Allow nanosecond precision datetime to be | downcasted to any lower precision. This has a similar effect to | PyArrow's `coerce_int96_timestamp_unit`. | * `microsecond-downcast`: Allow microsecond precision datetime to be | downcasted to millisecond precision. | * `microsecond-upcast`: Allow microsecond precision datetime to be | upcasted to nanosecond precision. | * `millisecond-upcast`: Allow millisecond precision datetime to be | upcasted to microsecond or nanosecond precision. | * `downcast`: Allow downcasting to any lower precision (convenience | aggregate of `nanosecond-downcast` and `microsecond-downcast`). | * `upcast`: Allow upcasting to any higher precision (convenience | aggregate of `millisecond-upcast` and `microsecond-upcast`). | * `convert-timezone`: Allow casting to a different timezone. | * `forbid`: Raises an error if dtypes do not match. | | missing_struct_fields | Configuration for behavior when struct fields defined in the schema | are missing from the data: | | * `insert`: Inserts the missing fields. | * `raise`: Raises an error. | | extra_struct_fields | Configuration for behavior when extra struct fields outside of the | defined schema are encountered in the data: | | * `ignore`: Silently ignores. | * `raise`: Raises an error. | | categorical_to_string | Configuration for behavior when reading in a column whose expected | type is string, but type in the file is categorical. | | * `allow`: Categorical is casted to string. | * `forbid`: Raises an error. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
polars.Schema
Python Library Documentation: class Schema in module polars.schema
class Schema(collections.OrderedDict)
| Schema(schema: 'Mapping[str, SchemaInitDataType] | Iterable[tuple[str, SchemaInitDataType] | ArrowSchemaExportable] | ArrowSchemaExportable | None' = None, *, check_dtypes: 'bool' = True) -> 'None'
|
| Ordered mapping of column names to their data type.
|
| Parameters
| ----------
| schema
| The schema definition given by column names and their associated
| Polars data type. Accepts a mapping, or an iterable of tuples, or any
| object implementing the `__arrow_c_schema__` PyCapsule interface
| (e.g. pyarrow schemas).
|
| Examples
| --------
| Define a schema by passing instantiated data types.
|
| >>> schema = pl.Schema(
| ... {
| ... "foo": pl.String(),
| ... "bar": pl.Duration("us"),
| ... "baz": pl.Array(pl.Int8, 4),
| ... }
| ... )
| >>> schema
| Schema({'foo': String, 'bar': Duration(time_unit='us'), 'baz': Array(Int8, shape=(4,))})
|
| Access the data type associated with a specific column name.
|
| >>> schema["baz"]
| Array(Int8, shape=(4,))
|
| Access various schema properties using the `names`, `dtypes`, and `len` methods.
|
| >>> schema.names()
| ['foo', 'bar', 'baz']
| >>> schema.dtypes()
| [String, Duration(time_unit='us'), Array(Int8, shape=(4,))]
| >>> schema.len()
| 3
|
| Import a pyarrow schema.
|
| >>> import pyarrow as pa
| >>> pl.Schema(pa.schema([pa.field("x", pa.int32())]))
| Schema({'x': Int32})
|
| Export a schema to pyarrow.
|
| >>> pa.schema(pl.Schema({"x": pl.Int32}))
| x: int32
|
| Method resolution order:
| Schema
| collections.OrderedDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __arrow_c_schema__(self) -> 'object'
| Export a Schema via the Arrow PyCapsule Interface.
|
| https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html
|
| __eq__(self, other: 'object') -> 'bool'
| Return self==value.
|
| __init__(self, schema: 'Mapping[str, SchemaInitDataType] | Iterable[tuple[str, SchemaInitDataType] | ArrowSchemaExportable] | ArrowSchemaExportable | None' = None, *, check_dtypes: 'bool' = True) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __ne__(self, other: 'object') -> 'bool'
| Return self!=value.
|
| __setitem__(self, name: 'str', dtype: 'DataType | DataTypeClass | PythonDataType') -> 'None'
| Set self[key] to value.
|
| contains_dtype(self, dtype: 'DataType', *, recursive: 'bool') -> 'bool'
| Check if the schema contains the given data type.
|
| Parameters
| ----------
| dtype
| The data type to search for.
| recursive
| If False, only check top-level column dtypes.
| If True, also search within nested types (List, Array, Struct).
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.Int64(), "y": pl.List(pl.Float64)})
| >>> s.contains_dtype(pl.Int64, recursive=False)
| True
| >>> s.contains_dtype(pl.Float64, recursive=False)
| False
| >>> s.contains_dtype(pl.Float64, recursive=True)
| True
|
| dtypes(self) -> 'list[DataType]'
| Get the data types of the schema.
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.UInt8(), "y": pl.List(pl.UInt8)})
| >>> s.dtypes()
| [UInt8, List(UInt8)]
|
| len(self) -> 'int'
| Get the number of schema entries.
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.Int32(), "y": pl.List(pl.String)})
| >>> s.len()
| 2
| >>> len(s)
| 2
|
| names(self) -> 'list[str]'
| Get the column names of the schema.
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.Float64(), "y": pl.Datetime(time_zone="UTC")})
| >>> s.names()
| ['x', 'y']
|
| to_arrow(self, *, compat_level: 'CompatLevel | None' = None) -> 'pa.Schema'
| Convert the schema to a pyarrow schema.
|
| Parameters
| ----------
| compat_level
| Use a specific compatibility level
| when exporting Polars' internal data types.
|
| Examples
| --------
| >>> pl.Schema({"x": pl.String}).to_arrow()
| x: string_view
|
| to_frame(self, *, eager: 'bool' = True) -> 'DataFrame | LazyFrame'
| Create an empty DataFrame (or LazyFrame) from this Schema.
|
| Parameters
| ----------
| eager
| If True, create a DataFrame; otherwise, create a LazyFrame.
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.Int32(), "y": pl.String()})
| >>> s.to_frame()
| shape: (0, 2)
| ┌─────┬─────┐
| │ x ┆ y │
| │ --- ┆ --- │
| │ i32 ┆ str │
| ╞═════╪═════╡
| └─────┴─────┘
| >>> s.to_frame(eager=False) # doctest: +IGNORE_RESULT
| <LazyFrame at 0x11BC0AD80>
|
| to_python(self) -> 'dict[str, type]'
| Return a dictionary of column names and Python types.
|
| Examples
| --------
| >>> s = pl.Schema(
| ... {
| ... "x": pl.Int8(),
| ... "y": pl.String(),
| ... "z": pl.Duration("us"),
| ... }
| ... )
| >>> s.to_python()
| {'x': <class 'int'>, 'y': <class 'str'>, 'z': <class 'datetime.timedelta'>}
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __hash__ = None
|
| __orig_bases__ = (collections.OrderedDict[str, polars.datatypes.classe...
|
| ----------------------------------------------------------------------
| Methods inherited from collections.OrderedDict:
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __ge__(self, value, /)
| Return self>=value.
|
| __gt__(self, value, /)
| Return self>value.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __lt__(self, value, /)
| Return self<value.
|
| __or__(self, value, /)
| Return self|value.
|
| __reduce__(...)
| Return state information for pickling
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(...)
| od.__reversed__() <==> reversed(od)
|
| __ror__(self, value, /)
| Return value|self.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| od.clear() -> None. Remove all items from od.
|
| copy(...)
| od.copy() -> a shallow copy of od
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| move_to_end(self, /, key, last=True)
| Move an existing element to the end (or beginning if last is false).
|
| Raise KeyError if the element does not exist.
|
| pop(...)
| od.pop(key[,default]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /, last=True)
| Remove and return a (key, value) pair from the dictionary.
|
| Pairs are returned in LIFO order if last is true or FIFO order if false.
|
| setdefault(self, /, key, default=None)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.OrderedDict:
|
| fromkeys(iterable, value=None)
| Create a new ordered dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from collections.OrderedDict:
|
| __dict__
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __len__(self, /)
| Return len(self).
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
polars.Series
Python Library Documentation: class Series in module polars.series.series
class Series(builtins.object)
| Series(name: 'str_ | ArrayLike | None' = None, values: 'ArrayLike | None' = None, dtype: 'PolarsDataType | None' = None, *, strict: 'bool' = True, nan_to_null: 'bool' = False) -> 'None'
|
| A Series represents a single column in a Polars DataFrame.
|
| Parameters
| ----------
| name : str, default None
| Name of the Series. Will be used as a column name when used in a DataFrame.
| When not specified, name is set to an empty string.
| values : ArrayLike, default None
| One-dimensional data in various forms. Supported are: Sequence, Series,
| pyarrow Array, and numpy ndarray.
| dtype : DataType, default None
| Data type of the resulting Series. If set to `None` (default), the data type is
| inferred from the `values` input. The strategy for data type inference depends
| on the `strict` parameter:
|
| - If `strict` is set to True (default), the inferred data type is equal to the
| first non-null value, or `Null` if all values are null.
| - If `strict` is set to False, the inferred data type is the supertype of the
| values, or :class:`Object` if no supertype can be found. **WARNING**: A full
| pass over the values is required to determine the supertype.
| - If no values were passed, the resulting data type is :class:`Null`.
|
| strict : bool, default True
| Throw an error if any value does not exactly match the given or inferred data
| type. If set to `False`, values that do not match the data type are cast to
| that data type or, if casting is not possible, set to null instead.
| nan_to_null : bool, default False
| In case a numpy array is used to create this Series, indicate how to deal
| with np.nan values. (This parameter is a no-op on non-numpy data).
|
| Examples
| --------
| Constructing a Series by specifying name and values positionally:
|
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| Notice that the dtype is automatically inferred as a polars Int64:
|
| >>> s.dtype
| Int64
|
| Constructing a Series with a specific dtype:
|
| >>> s2 = pl.Series("a", [1, 2, 3], dtype=pl.Float32)
| >>> s2
| shape: (3,)
| Series: 'a' [f32]
| [
| 1.0
| 2.0
| 3.0
| ]
|
| It is possible to construct a Series with values as the first positional argument.
| This syntax considered an anti-pattern, but it can be useful in certain
| scenarios. You must specify any other arguments through keywords.
|
| >>> s3 = pl.Series([1, 2, 3])
| >>> s3
| shape: (3,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| ]
|
| Methods defined here:
|
| __abs__(self) -> 'Series'
|
| __add__(self, other: 'Any') -> 'Series | DataFrame | Expr'
|
| __and__(self, other: 'Any') -> 'Expr | Series'
|
| __array__(self, dtype: 'np.dtype[Any] | None' = None, copy: 'bool | None' = None) -> 'np.ndarray[Any, Any]'
| Return a NumPy ndarray with the given data type.
|
| This method ensures a Polars Series can be treated as a NumPy ndarray.
| It enables `np.asarray` and NumPy universal functions.
|
| See the NumPy documentation for more information:
| https://numpy.org/doc/stable/user/basics.interoperability.html#the-array-method
|
| See Also
| --------
| __array_ufunc__
|
| __array_ufunc__(self, ufunc: 'np.ufunc', method: 'str_', *inputs: 'Any', **kwargs: 'Any') -> 'Series'
| Numpy universal functions.
|
| __arrow_c_stream__(self, requested_schema: 'object | None' = None) -> 'object'
| Export a Series via the Arrow PyCapsule Interface.
|
| https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html
|
| __bool__(self) -> 'NoReturn'
|
| __contains__(self, item: 'Any') -> 'bool'
|
| __copy__(self) -> 'Self'
|
| __deepcopy__(self, memo: 'None' = None) -> 'Self'
|
| __eq__(self, other: 'object') -> 'Series | Expr'
| Return self==value.
|
| __floordiv__(self, other: 'Any') -> 'Series | Expr'
|
| __ge__(self, other: 'Any') -> 'Series | Expr'
| Return self>=value.
|
| __getitem__(self, key: 'SingleIndexSelector | MultiIndexSelector') -> 'Any | Series'
| Get part of the Series as a new Series or scalar.
|
| Parameters
| ----------
| key
| Row(s) to select.
|
| Returns
| -------
| Series or scalar, depending on `key`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 4, 2])
| >>> s[0]
| 1
| >>> s[0:2]
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 4
| ]
|
| __getstate__(self) -> 'bytes'
| Helper for pickle.
|
| __gt__(self, other: 'Any') -> 'Series | Expr'
| Return self>value.
|
| __init__(self, name: 'str_ | ArrayLike | None' = None, values: 'ArrayLike | None' = None, dtype: 'PolarsDataType | None' = None, *, strict: 'bool' = True, nan_to_null: 'bool' = False) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __invert__(self) -> 'Series'
|
| __iter__(self) -> 'Generator[Any]'
|
| __le__(self, other: 'Any') -> 'Series | Expr'
| Return self<=value.
|
| __len__(self) -> 'int'
|
| __lt__(self, other: 'Any') -> 'Series | Expr'
| Return self<value.
|
| __matmul__(self, other: 'Any') -> 'float | Series | None'
|
| __mod__(self, other: 'Any') -> 'Series | Expr'
|
| __mul__(self, other: 'Any') -> 'Series | DataFrame | Expr'
|
| __ne__(self, other: 'object') -> 'Series | Expr'
| Return self!=value.
|
| __neg__(self) -> 'Series'
|
| __or__(self, other: 'Any') -> 'Expr | Series'
| Return self|value.
|
| __pos__(self) -> 'Series'
|
| __pow__(self, exponent: 'int | float | Series') -> 'Series'
|
| __radd__(self, other: 'Any') -> 'Series'
|
| __rand__(self, other: 'Any') -> 'Expr | Series'
|
| __repr__(self) -> 'str_'
| Return repr(self).
|
| __rfloordiv__(self, other: 'Any') -> 'Series'
|
| __rmatmul__(self, other: 'Any') -> 'float | Series | None'
|
| __rmod__(self, other: 'Any') -> 'Series'
|
| __rmul__(self, other: 'Any') -> 'Series'
|
| __ror__(self, other: 'Any') -> 'Expr | Series'
| Return value|self.
|
| __rpow__(self, other: 'Any') -> 'Series'
|
| __rsub__(self, other: 'Any') -> 'Series'
|
| __rtruediv__(self, other: 'Any') -> 'Series'
|
| __rxor__(self, other: 'Any') -> 'Expr | Series'
|
| __setitem__(self, key: 'int | Series | np.ndarray[Any, Any] | Sequence[object] | tuple[object]', value: 'Any') -> 'None'
| Set Series values in-place using a single index, boolean mask, or index array.
|
| Parameters
| ----------
| key
| Determines which elements to update:
|
| - ``int``: a single row index.
| - ``Series`` (Boolean): a boolean mask.
| - ``Series`` (Integer): an index array.
| - ``ndarray``: a NumPy boolean mask or integer index array.
| - ``list`` / ``tuple``: an index sequence (cast to UInt32).
| value
| Scalar or sequence of values to assign.
|
| Examples
| --------
| Set a single element by index:
|
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s[0] = 10
| >>> s
| shape: (3,)
| Series: 'a' [i64]
| [
| 10
| 2
| 3
| ]
|
| Set elements with a boolean mask:
|
| >>> s[pl.Series([False, True, True])] = 99
| >>> s
| shape: (3,)
| Series: 'a' [i64]
| [
| 10
| 99
| 99
| ]
|
| __setstate__(self, state: 'bytes') -> 'None'
|
| __str__(self) -> 'str_'
| Return str(self).
|
| __sub__(self, other: 'Any') -> 'Series | Expr'
|
| __truediv__(self, other: 'Any') -> 'Series | Expr'
|
| __xor__(self, other: 'Any') -> 'Expr | Series'
|
| abs(self) -> 'Series'
| Compute absolute values.
|
| Same as `abs(series)`.
|
| Examples
| --------
| >>> s = pl.Series([1, -2, -3])
| >>> s.abs()
| shape: (3,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| ]
|
| alias(self, name: 'str_') -> 'Series'
| Rename the series.
|
| Parameters
| ----------
| name
| The new name.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.alias("b")
| shape: (3,)
| Series: 'b' [i64]
| [
| 1
| 2
| 3
| ]
|
| all(self, *, ignore_nulls: 'bool' = True) -> 'bool | None'
| Return whether all values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `True`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `False` values,
| the output is `None`.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| bool or None
|
| Examples
| --------
| >>> pl.Series([True, True]).all()
| True
| >>> pl.Series([False, True]).all()
| False
| >>> pl.Series([None, True]).all()
| True
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> pl.Series([None, True]).all(ignore_nulls=False) # Returns None
|
| any(self, *, ignore_nulls: 'bool' = True) -> 'bool | None'
| Return whether any of the values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `False`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `True` values,
| the output is `None`.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| bool or None
|
| Examples
| --------
| >>> pl.Series([True, False]).any()
| True
| >>> pl.Series([False, False]).any()
| False
| >>> pl.Series([None, False]).any()
| False
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> pl.Series([None, False]).any(ignore_nulls=False) # Returns None
|
| append(self, other: 'Series') -> 'Self'
| Append a Series to this one.
|
| The resulting series will consist of multiple chunks.
|
| Parameters
| ----------
| other
| Series to append.
|
| Warnings
| --------
| This method modifies the series in-place. The series is returned for
| convenience only.
|
| See Also
| --------
| extend
|
| Examples
| --------
| >>> a = pl.Series("a", [1, 2, 3])
| >>> b = pl.Series("b", [4, 5])
| >>> a.append(b)
| shape: (5,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| ]
|
| The resulting series will consist of multiple chunks.
|
| >>> a.n_chunks()
| 2
|
| approx_n_unique(self) -> 'PythonLiteral | None'
| Approximate count of unique values.
|
| This is done using the HyperLogLog++ algorithm for cardinality estimation.
|
| arccos(self) -> 'Series'
| Compute the element-wise value for the inverse cosine.
|
| Notes
| -----
| The returned value is in radians.
| To convert from radians to degrees,
| call :meth:`.degrees() <polars.Series.degrees>`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.arccos()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.0
| 1.570796
| 3.141593
| ]
|
| arccosh(self) -> 'Series'
| Compute the element-wise value for the inverse hyperbolic cosine.
|
| Examples
| --------
| >>> s = pl.Series("a", [5.0, 1.0, 0.0, -1.0])
| >>> s.arccosh()
| shape: (4,)
| Series: 'a' [f64]
| [
| 2.292432
| 0.0
| NaN
| NaN
| ]
|
| arcsin(self) -> 'Series'
| Compute the element-wise value for the inverse sine.
|
| Notes
| -----
| The returned value is in radians.
| To convert from radians to degrees,
| call :meth:`.degrees() <polars.Series.degrees>`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.arcsin()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.570796
| 0.0
| -1.570796
| ]
|
| arcsinh(self) -> 'Series'
| Compute the element-wise value for the inverse hyperbolic sine.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.arcsinh()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.881374
| 0.0
| -0.881374
| ]
|
| arctan(self) -> 'Series'
| Compute the element-wise value for the inverse tangent.
|
| Notes
| -----
| The returned value is in radians.
| To convert from radians to degrees,
| call :meth:`.degrees() <polars.Series.degrees>`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.arctan()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.785398
| 0.0
| -0.785398
| ]
|
| arctanh(self) -> 'Series'
| Compute the element-wise value for the inverse hyperbolic tangent.
|
| Examples
| --------
| >>> s = pl.Series("a", [2.0, 1.0, 0.5, 0.0, -0.5, -1.0, -1.1])
| >>> s.arctanh()
| shape: (7,)
| Series: 'a' [f64]
| [
| NaN
| inf
| 0.549306
| 0.0
| -0.549306
| -inf
| NaN
| ]
|
| arg_max(self) -> 'int | None'
| Get the index of the maximal value.
|
| Returns
| -------
| int
|
| Examples
| --------
| >>> s = pl.Series("a", [3, 2, 1])
| >>> s.arg_max()
| 0
|
| arg_min(self) -> 'int | None'
| Get the index of the minimal value.
|
| Returns
| -------
| int
|
| Examples
| --------
| >>> s = pl.Series("a", [3, 2, 1])
| >>> s.arg_min()
| 2
|
| arg_sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Series'
| Get the index values that would sort this Series.
|
| Parameters
| ----------
| descending
| Sort in descending order.
| nulls_last
| Place null values last instead of first.
|
| See Also
| --------
| Series.gather: Take values by index.
| Series.rank : Get the rank of each row.
|
| Examples
| --------
| >>> s = pl.Series("a", [5, 3, 4, 1, 2])
| >>> s.arg_sort()
| shape: (5,)
| Series: 'a' [u32]
| [
| 3
| 4
| 1
| 2
| 0
| ]
|
| arg_true(self) -> 'Series'
| Get index values where Boolean Series evaluate True.
|
| Returns
| -------
| Series
| Series of data type :class:`UInt32`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> (s == 2).arg_true()
| shape: (1,)
| Series: 'a' [u32]
| [
| 1
| ]
|
| arg_unique(self) -> 'Series'
| Get unique index as Series.
|
| Returns
| -------
| Series
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.arg_unique()
| shape: (3,)
| Series: 'a' [u32]
| [
| 0
| 1
| 3
| ]
|
| backward_fill(self, limit: 'int | None' = None) -> 'Series'
| Fill missing values with the next non-null value.
|
| This is an alias of `.fill_null(strategy="backward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to backward fill.
|
| See Also
| --------
| fill_null
| forward_fill
| shift
|
| bitwise_and(self) -> 'PythonLiteral | None'
| Perform an aggregation of bitwise ANDs.
|
| bitwise_count_ones(self) -> 'Self'
| Evaluate the number of set bits.
|
| bitwise_count_zeros(self) -> 'Self'
| Evaluate the number of unset bits.
|
| bitwise_leading_ones(self) -> 'Self'
| Evaluate the number most-significant set bits before seeing an unset bit.
|
| bitwise_leading_zeros(self) -> 'Self'
| Evaluate the number most-significant unset bits before seeing a set bit.
|
| bitwise_or(self) -> 'PythonLiteral | None'
| Perform an aggregation of bitwise ORs.
|
| bitwise_trailing_ones(self) -> 'Self'
| Evaluate the number least-significant set bits before seeing an unset bit.
|
| bitwise_trailing_zeros(self) -> 'Self'
| Evaluate the number least-significant unset bits before seeing a set bit.
|
| bitwise_xor(self) -> 'PythonLiteral | None'
| Perform an aggregation of bitwise XORs.
|
| bottom_k(self, k: 'int' = 5) -> 'Series'
| Return the `k` smallest elements.
|
| Non-null elements are always preferred over null elements. The output is
| not guaranteed to be in any particular order, call :func:`sort` after
| this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k_by
|
| Examples
| --------
| >>> s = pl.Series("a", [2, 5, 1, 4, 3])
| >>> s.bottom_k(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| bottom_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Series'
| Return the `k` smallest elements of the `by` column.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| Parameters
| ----------
| by
| Column used to determine the smallest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` largest elements of the `by` column( (instead of the `k`
| smallest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k
|
| Examples
| --------
| >>> s = pl.Series("a", [2, 5, 1, 4, 3])
| >>> s.bottom_k_by("a", 3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| cast(self, dtype: 'type[int | float | str_ | bool] | PolarsDataType', *, strict: 'bool' = True, wrap_numerical: 'bool' = False) -> 'Self'
| Cast between data types.
|
| Parameters
| ----------
| dtype
| DataType to cast to.
| strict
| If True invalid casts generate exceptions instead of `null`\s.
| wrap_numerical
| If True numeric casts wrap overflowing values instead of
| marking the cast as invalid.
|
| Examples
| --------
| >>> s = pl.Series("a", [True, False, True])
| >>> s
| shape: (3,)
| Series: 'a' [bool]
| [
| true
| false
| true
| ]
|
| >>> s.cast(pl.UInt32)
| shape: (3,)
| Series: 'a' [u32]
| [
| 1
| 0
| 1
| ]
|
| cbrt(self) -> 'Series'
| Compute the cube root of the elements.
|
| Optimization for
|
| >>> pl.Series([1, 2]) ** (1.0 / 3)
| shape: (2,)
| Series: '' [f64]
| [
| 1.0
| 1.259921
| ]
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.cbrt()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 1.259921
| 1.44225
| ]
|
| ceil(self) -> 'Series'
| Rounds up to the nearest integer value.
|
| Only works on floating point Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.12345, 2.56789, 3.901234])
| >>> s.ceil()
| shape: (3,)
| Series: 'a' [f64]
| [
| 2.0
| 3.0
| 4.0
| ]
|
| chunk_lengths(self) -> 'list_[int]'
| Get the length of each individual chunk.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("a", [4, 5, 6])
|
| Concatenate Series with rechunk = True
|
| >>> pl.concat([s, s2], rechunk=True).chunk_lengths()
| [6]
|
| Concatenate Series with rechunk = False
|
| >>> pl.concat([s, s2], rechunk=False).chunk_lengths()
| [3, 3]
|
| clear(self, n: 'int' = 0) -> 'Series'
| Create an empty copy of the current Series, with zero to 'n' elements.
|
| The copy has an identical name/dtype, but no data.
|
| Parameters
| ----------
| n
| Number of (empty) elements to return in the cleared frame.
|
| See Also
| --------
| clone : Cheap deepcopy/clone.
|
| Examples
| --------
| >>> s = pl.Series("a", [None, True, False])
| >>> s.clear()
| shape: (0,)
| Series: 'a' [bool]
| [
| ]
|
| >>> s.clear(n=2)
| shape: (2,)
| Series: 'a' [bool]
| [
| null
| null
| ]
|
| clip(self, lower_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None, upper_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None) -> 'Series'
| Set values outside the given boundaries to the boundary value.
|
| Parameters
| ----------
| lower_bound
| Lower bound. Accepts expression input.
| Non-expression inputs are parsed as literals.
| If set to `None` (default), no lower bound is applied.
| upper_bound
| Upper bound. Accepts expression input.
| Non-expression inputs are parsed as literals.
| If set to `None` (default), no upper bound is applied.
|
| See Also
| --------
| when
|
| Notes
| -----
| This method only works for numeric and temporal columns. To clip other data
| types, consider writing a `when-then-otherwise` expression. See :func:`when`.
|
| Examples
| --------
| Specifying both a lower and upper bound:
|
| >>> s = pl.Series([-50, 5, 50, None])
| >>> s.clip(1, 10)
| shape: (4,)
| Series: '' [i64]
| [
| 1
| 5
| 10
| null
| ]
|
| Specifying only a single bound:
|
| >>> s.clip(upper_bound=10)
| shape: (4,)
| Series: '' [i64]
| [
| -50
| 5
| 10
| null
| ]
|
| clone(self) -> 'Self'
| Create a copy of this Series.
|
| This is a cheap operation that does not copy data.
|
| See Also
| --------
| clear : Create an empty copy of the current Series, with identical
| schema but no data.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.clone()
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| cos(self) -> 'Series'
| Compute the element-wise value for the cosine.
|
| Notes
| -----
| Input values are interpreted as radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Series.radians>`.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [0.0, math.pi / 2.0, math.pi])
| >>> s.cos()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.0
| 6.1232e-17
| -1.0
| ]
|
| cosh(self) -> 'Series'
| Compute the element-wise value for the hyperbolic cosine.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.cosh()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.543081
| 1.0
| 1.543081
| ]
|
| cot(self) -> 'Series'
| Compute the element-wise value for the cotangent.
|
| Notes
| -----
| Input values are interpreted as radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Series.radians>`.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [0.0, math.pi / 2.0, math.pi])
| >>> s.cot()
| shape: (3,)
| Series: 'a' [f64]
| [
| inf
| 6.1232e-17
| -8.1656e15
| ]
|
| count(self) -> 'int'
| Return the number of non-null elements in the column.
|
| See Also
| --------
| len
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, None])
| >>> s.count()
| 2
|
| cum_count(self, *, reverse: 'bool' = False) -> 'Self'
| Return the cumulative count of the non-null values in the column.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> s = pl.Series(["x", "k", None, "d"])
| >>> s.cum_count()
| shape: (4,)
| Series: '' [u32]
| [
| 1
| 2
| 2
| 3
| ]
|
| cum_max(self, *, reverse: 'bool' = False) -> 'Series'
| Get an array with the cumulative max computed at every element.
|
| Parameters
| ----------
| reverse
| reverse the operation.
|
| Examples
| --------
| >>> s = pl.Series("s", [3, 5, 1])
| >>> s.cum_max()
| shape: (3,)
| Series: 's' [i64]
| [
| 3
| 5
| 5
| ]
|
| cum_min(self, *, reverse: 'bool' = False) -> 'Series'
| Get an array with the cumulative min computed at every element.
|
| Parameters
| ----------
| reverse
| reverse the operation.
|
| Examples
| --------
| >>> s = pl.Series("s", [1, 2, 3])
| >>> s.cum_min()
| shape: (3,)
| Series: 's' [i64]
| [
| 1
| 1
| 1
| ]
|
| cum_prod(self, *, reverse: 'bool' = False) -> 'Series'
| Get an array with the cumulative product computed at every element.
|
| Parameters
| ----------
| reverse
| reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.cum_prod()
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 6
| ]
|
| cum_sum(self, *, reverse: 'bool' = False) -> 'Series'
| Get an array with the cumulative sum computed at every element.
|
| Parameters
| ----------
| reverse
| reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.cum_sum()
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 3
| 6
| ]
|
| cumulative_eval(self, expr: 'Expr', *, min_samples: 'int' = 1, parallel: 'bool' = False) -> 'Series'
| Run an expression over a sliding window that increases `1` slot every iteration.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| expr
| Expression to evaluate
| min_samples
| Number of valid values there should be in the window before the expression
| is evaluated. valid values = `length - null_count`
| parallel
| Run in parallel. Don't do this in a group by or another operation that
| already has much parallelization.
|
| Warnings
| --------
| This can be really slow as it can have `O(n^2)` complexity. Don't use this
| for operations that visit all elements.
|
| Examples
| --------
| >>> s = pl.Series("values", [1, 2, 3, 4, 5])
| >>> s.cumulative_eval(pl.element().first() - pl.element().last() ** 2)
| shape: (5,)
| Series: 'values' [i64]
| [
| 0
| -3
| -8
| -15
| -24
| ]
|
| cut(self, breaks: 'Sequence[float]', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, include_breaks: 'bool' = False) -> 'Series'
| Bin continuous values into discrete categories.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| breaks
| List of unique cut points.
| labels
| Names of the categories. The number of labels must be equal to the number
| of cut points plus one.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from an
| :class:`Enum` to a :class:`Struct`.
|
| Returns
| -------
| Series
| Series of data type :class:`Enum` if `include_breaks` is set to
| `False` (default), otherwise a Series of data type :class:`Struct`.
|
| See Also
| --------
| qcut
|
| Examples
| --------
| Divide the column into three categories.
|
| >>> s = pl.Series("foo", [-2, -1, 0, 1, 2])
| >>> s.cut([-1, 1], labels=["a", "b", "c"])
| shape: (5,)
| Series: 'foo' [enum]
| [
| "a"
| "a"
| "b"
| "b"
| "c"
| ]
|
| Create a DataFrame with the breakpoint and category for each value.
|
| >>> cut = s.cut([-1, 1], include_breaks=True).alias("cut")
| >>> s.to_frame().with_columns(cut).unnest("cut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ enum │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| degrees(self) -> 'Series'
| Convert from radians to degrees.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [x * math.pi for x in range(-4, 5)])
| >>> s.degrees()
| shape: (9,)
| Series: 'a' [f64]
| [
| -720.0
| -540.0
| -360.0
| -180.0
| 0.0
| 180.0
| 360.0
| 540.0
| 720.0
| ]
|
| describe(self, percentiles: 'Sequence[float] | float | None' = (0.25, 0.5, 0.75), interpolation: 'QuantileMethod' = 'nearest') -> 'DataFrame'
| Quick summary statistics of a Series.
|
| Series with mixed datatypes will return summary statistics for the datatype of
| the first value.
|
| Parameters
| ----------
| percentiles
| One or more percentiles to include in the summary statistics (if the
| Series has a numeric dtype). All values must be in the range `[0, 1]`.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method used when calculating percentiles.
|
| Notes
| -----
| The median is included by default as the 50% percentile.
|
| Returns
| -------
| DataFrame
| Mapping with summary statistics of a Series.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3, 4, 5])
| >>> s.describe()
| shape: (9, 2)
| ┌────────────┬──────────┐
| │ statistic ┆ value │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞════════════╪══════════╡
| │ count ┆ 5.0 │
| │ null_count ┆ 0.0 │
| │ mean ┆ 3.0 │
| │ std ┆ 1.581139 │
| │ min ┆ 1.0 │
| │ 25% ┆ 2.0 │
| │ 50% ┆ 3.0 │
| │ 75% ┆ 4.0 │
| │ max ┆ 5.0 │
| └────────────┴──────────┘
|
| Non-numeric data types may not have all statistics available.
|
| >>> s = pl.Series(["aa", "aa", None, "bb", "cc"])
| >>> s.describe()
| shape: (4, 2)
| ┌────────────┬───────┐
| │ statistic ┆ value │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞════════════╪═══════╡
| │ count ┆ 4 │
| │ null_count ┆ 1 │
| │ min ┆ aa │
| │ max ┆ cc │
| └────────────┴───────┘
|
| diff(self, n: 'int' = 1, null_behavior: 'NullBehavior' = 'ignore') -> 'Series'
| Calculate the first discrete difference between shifted items.
|
| Parameters
| ----------
| n
| Number of slots to shift.
| null_behavior : {'ignore', 'drop'}
| How to handle null values.
|
| Examples
| --------
| >>> s = pl.Series("s", values=[20, 10, 30, 25, 35], dtype=pl.Int8)
| >>> s.diff()
| shape: (5,)
| Series: 's' [i8]
| [
| null
| -10
| 20
| -5
| 10
| ]
|
| >>> s.diff(n=2)
| shape: (5,)
| Series: 's' [i8]
| [
| null
| null
| 10
| 15
| 5
| ]
|
| >>> s.diff(n=2, null_behavior="drop")
| shape: (3,)
| Series: 's' [i8]
| [
| 10
| 15
| 5
| ]
|
| dot(self, other: 'Series | ArrayLike') -> 'int | float | None'
| Compute the dot/inner product between two Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("b", [4.0, 5.0, 6.0])
| >>> s.dot(s2)
| 32.0
|
| Parameters
| ----------
| other
| Series (or array) to compute dot product with.
|
| drop_nans(self) -> 'Series'
| Drop all floating point NaN values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nulls
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To drop null values, use :func:`drop_nulls`.
|
| Examples
| --------
| >>> s = pl.Series([1.0, None, 3.0, float("nan")])
| >>> s.drop_nans()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| null
| 3.0
| ]
|
| drop_nulls(self) -> 'Series'
| Drop all null values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nans
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To drop NaN values, use :func:`drop_nans`.
|
| Examples
| --------
| >>> s = pl.Series([1.0, None, 3.0, float("nan")])
| >>> s.drop_nulls()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 3.0
| NaN
| ]
|
| entropy(self, base: 'float' = 2.718281828459045, *, normalize: 'bool' = True) -> 'float | None'
| Computes the entropy.
|
| Uses the formula `-sum(pk * log(pk))` where `pk` are discrete probabilities.
|
| Parameters
| ----------
| base
| Given base, defaults to `e`
| normalize
| Normalize pk if it doesn't sum to 1.
|
| Examples
| --------
| >>> a = pl.Series([0.99, 0.005, 0.005])
| >>> a.entropy(normalize=True)
| 0.06293300616044681
| >>> b = pl.Series([0.65, 0.10, 0.25])
| >>> b.entropy(normalize=True)
| 0.8568409950394724
|
| eq(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series == other`.
|
| eq_missing(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of equality operator `series == other` where `None == None`.
|
| This differs from the standard `eq` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| See Also
| --------
| ne_missing
| eq
|
| Examples
| --------
| >>> s1 = pl.Series("a", [333, 200, None])
| >>> s2 = pl.Series("a", [100, 200, None])
| >>> s1.eq(s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| true
| null
| ]
| >>> s1.eq_missing(s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| true
| true
| ]
|
| equals(self, other: 'Series', *, check_dtypes: 'bool' = False, check_names: 'bool' = False, null_equal: 'bool' = True) -> 'bool'
| Check whether the Series is equal to another Series.
|
| .. versionchanged:: 0.20.31
| The `strict` parameter was renamed `check_dtypes`.
|
| Parameters
| ----------
| other
| Series to compare with.
| check_dtypes
| Require data types to match.
| check_names
| Require names to match.
| null_equal
| Consider null values as equal.
|
| See Also
| --------
| polars.testing.assert_series_equal
|
| Examples
| --------
| >>> s1 = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("b", [4, 5, 6])
| >>> s1.equals(s1)
| True
| >>> s1.equals(s2)
| False
|
| estimated_size(self, unit: 'SizeUnit' = 'b') -> 'int | float'
| Return an estimation of the total (heap) allocated size of the Series.
|
| Estimated size is given in the specified unit (bytes by default).
|
| This estimation is the sum of the size of its buffers, validity, including
| nested arrays. Multiple arrays may share buffers and bitmaps. Therefore, the
| size of 2 arrays is not the sum of the sizes computed from this function. In
| particular, [`StructArray`]'s size is an upper bound.
|
| When an array is sliced, its allocated size remains constant because the buffer
| unchanged. However, this function will yield a smaller number. This is because
| this function returns the visible size of the buffer, not its total capacity.
|
| FFI buffers are included in this estimation.
|
| Notes
| -----
| For data with Object dtype, the estimated size only reports the pointer
| size, which is a huge underestimation.
|
| Parameters
| ----------
| unit : {'b', 'kb', 'mb', 'gb', 'tb'}
| Scale the returned size to the given unit.
|
| Examples
| --------
| >>> s = pl.Series("values", list(range(1_000_000)), dtype=pl.UInt32)
| >>> s.estimated_size()
| 4000000
| >>> s.estimated_size("mb")
| 3.814697265625
|
| ewm_mean(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Series'
| Compute exponentially-weighted moving average.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.ewm_mean(com=1, ignore_nulls=False)
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 1.666667
| 2.428571
| ]
|
| ewm_mean_by(self, by: 'IntoExpr', *, half_life: 'str_ | timedelta') -> 'Series'
| Compute time-based exponentially weighted moving average.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMA is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \alpha_i &= 1 - \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= \alpha_i x_i + (1 - \alpha_i) y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Times to calculate average by. Should be ``DateTime``, ``Date``, ``UInt64``,
| ``UInt32``, ``Int64``, or ``Int32`` data type.
| half_life
| Unit over which observation decays to half its value.
|
| Can be created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 day)
| - 1w (1 week)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| Note that `half_life` is treated as a constant duration - calendar
| durations such as months (or even days in the time-zone-aware case)
| are not supported, please express your duration in an approximately
| equivalent number of hours (e.g. '370h' instead of '1mo').
|
| Returns
| -------
| Expr
| :class:`Float16` if input is `Float16`, :class:`.Float32` if input is
| `Float32`, otherwise :class:`.Float64`.
|
| Examples
| --------
| >>> from datetime import date, timedelta
| >>> df = pl.DataFrame(
| ... {
| ... "values": [0, 1, 2, None, 4],
| ... "times": [
| ... date(2020, 1, 1),
| ... date(2020, 1, 3),
| ... date(2020, 1, 10),
| ... date(2020, 1, 15),
| ... date(2020, 1, 17),
| ... ],
| ... }
| ... ).sort("times")
| >>> df["values"].ewm_mean_by(df["times"], half_life="4d")
| shape: (5,)
| Series: 'values' [f64]
| [
| 0.0
| 0.292893
| 1.492474
| null
| 3.254508
| ]
|
| ewm_std(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Series'
| Compute exponentially-weighted moving standard deviation.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.ewm_std(com=1, ignore_nulls=False)
| shape: (3,)
| Series: 'a' [f64]
| [
| null
| 0.707107
| 0.963624
| ]
|
| ewm_sum(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Series'
| Compute exponentially-weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.ewm_sum(alpha=0.5)
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 2.5
| 4.25
| ]
|
| ewm_sum_by(self, by: 'IntoExpr', *, half_life: 'str_ | timedelta') -> 'Series'
| Compute time-based exponentially weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMS is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \lambda_i &= \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= x_i + \lambda_i y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Times to calculate the sum by. Should be ``DateTime``, ``Date``, ``UInt64``,
| ``UInt32``, ``Int64``, or ``Int32`` data type.
| half_life
| Unit over which observation decays to half its value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "values": [1, 2, 3, 4, 5],
| ... "times": [0, 1, 2, 5, 6],
| ... }
| ... )
| >>> df["values"].ewm_sum_by(df["times"], half_life="1i")
| shape: (5,)
| Series: 'values' [f64]
| [
| 1.0
| 2.5
| 4.25
| 4.53125
| 7.265625
| ]
|
| ewm_var(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Series'
| Compute exponentially-weighted moving variance.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.ewm_var(com=1, ignore_nulls=False)
| shape: (3,)
| Series: 'a' [f64]
| [
| null
| 0.5
| 0.928571
| ]
|
| exp(self) -> 'Series'
| Compute the exponential, element-wise.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.exp()
| shape: (3,)
| Series: '' [f64]
| [
| 2.718282
| 7.389056
| 20.085537
| ]
|
| explode(self, *, empty_as_null: 'bool | None' = True, keep_nulls: 'bool' = True) -> 'Series'
| Explode a list Series.
|
| This means that every item is expanded to a new row.
|
| Parameters
| ----------
| empty_as_null
| Explode an empty list into a `null`.
| keep_nulls
| Explode a `null` list into a `null`.
|
| Returns
| -------
| Series
| Series with the data type of the list elements.
|
| See Also
| --------
| Series.list.explode : Explode a list column.
|
| Examples
| --------
| >>> s = pl.Series("a", [[1, 2, 3], [4, 5, 6]])
| >>> s
| shape: (2,)
| Series: 'a' [list[i64]]
| [
| [1, 2, 3]
| [4, 5, 6]
| ]
| >>> s.explode(empty_as_null=False)
| shape: (6,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| ]
|
| extend(self, other: 'Series') -> 'Self'
| Extend the memory backed by this Series with the values from another.
|
| Different from `append`, which adds the chunks from `other` to the chunks of
| this series, `extend` appends the data from `other` to the underlying memory
| locations and thus may cause a reallocation (which is expensive).
|
| If this does `not` cause a reallocation, the resulting data structure will not
| have any extra chunks and thus will yield faster queries.
|
| Prefer `extend` over `append` when you want to do a query after a single
| append. For instance, during online operations where you add `n` rows
| and rerun a query.
|
| Prefer `append` over `extend` when you want to append many times
| before doing a query. For instance, when you read in multiple files and want
| to store them in a single `Series`. In the latter case, finish the sequence
| of `append` operations with a `rechunk`.
|
| Parameters
| ----------
| other
| Series to extend the series with.
|
| Warnings
| --------
| This method modifies the series in-place. The series is returned for
| convenience only.
|
| See Also
| --------
| append
|
| Examples
| --------
| >>> a = pl.Series("a", [1, 2, 3])
| >>> b = pl.Series("b", [4, 5])
| >>> a.extend(b)
| shape: (5,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| ]
|
| The resulting series will consist of a single chunk.
|
| >>> a.n_chunks()
| 1
|
| extend_constant(self, value: 'IntoExpr', n: 'int | IntoExprColumn') -> 'Series'
| Extremely fast method for extending the Series with 'n' copies of a value.
|
| Parameters
| ----------
| value
| A constant literal value or a unit expression with which to extend the
| expression result Series; can pass None to extend with nulls.
| n
| The number of additional values that will be added.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.extend_constant(99, n=2)
| shape: (5,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| 99
| 99
| ]
|
| fill_nan(self, value: 'int | float | Expr | None') -> 'Series'
| Fill floating point NaN value with a fill value.
|
| Parameters
| ----------
| value
| Value used to fill NaN values.
|
| See Also
| --------
| fill_null
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To fill null values, use :func:`fill_null`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, float("nan")])
| >>> s.fill_nan(0)
| shape: (4,)
| Series: 'a' [f64]
| [
| 1.0
| 2.0
| 3.0
| 0.0
| ]
|
| fill_null(self, value: 'Any | Expr | None' = None, strategy: 'FillNullStrategy | None' = None, limit: 'int | None' = None) -> 'Series'
| Fill null values using the specified value or strategy.
|
| Parameters
| ----------
| value
| Value used to fill null values.
| strategy : {None, 'forward', 'backward', 'min', 'max', 'mean', 'zero', 'one'}
| Strategy used to fill null values.
| limit
| Number of consecutive null values to fill when using the 'forward' or
| 'backward' strategy.
|
| See Also
| --------
| backward_fill
| fill_nan
| forward_fill
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To fill NaN values, use :func:`fill_nan`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, None])
| >>> s.fill_null(strategy="forward")
| shape: (4,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 3
| ]
| >>> s.fill_null(strategy="min")
| shape: (4,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 1
| ]
| >>> s = pl.Series("b", ["x", None, "z"])
| >>> s.fill_null(pl.lit(""))
| shape: (3,)
| Series: 'b' [str]
| [
| "x"
| ""
| "z"
| ]
|
| filter(self, predicate: 'Series | Iterable[bool]') -> 'Self'
| Filter elements by a boolean mask.
|
| The original order of the remaining elements is preserved.
|
| Elements where the filter does not evaluate to True are discarded, including
| nulls.
|
| Parameters
| ----------
| predicate
| Boolean mask.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> mask = pl.Series("", [True, False, True])
| >>> s.filter(mask)
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 3
| ]
|
| first(self, *, ignore_nulls: 'bool' = False) -> 'PythonLiteral | None'
| Get the first element of the Series.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the first non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Returns `None` if the Series is empty.
|
| floor(self) -> 'Series'
| Rounds down to the nearest integer value.
|
| Only works on floating point Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.12345, 2.56789, 3.901234])
| >>> s.floor()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.0
| 2.0
| 3.0
| ]
|
| forward_fill(self, limit: 'int | None' = None) -> 'Series'
| Fill missing values with the last non-null value.
|
| This is an alias of `.fill_null(strategy="forward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to forward fill.
|
| See Also
| --------
| backward_fill
| fill_null
| shift
|
| gather(self, indices: 'int | list_[int] | Expr | Series | np.ndarray[Any, Any]', *, null_on_oob: 'bool' = False) -> 'Series'
| Take values by index.
|
| Parameters
| ----------
| indices
| Index location used for selection.
| null_on_oob
| Behavior if an index is out of bounds:
|
| - True -> set the result to null
| - False -> raise an error
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4])
| >>> s.gather([1, 3])
| shape: (2,)
| Series: 'a' [i64]
| [
| 2
| 4
| ]
|
| Use `null_on_oob=True` to return null for out-of-bounds indices.
|
| >>> s.gather([1, 10], null_on_oob=True)
| shape: (2,)
| Series: 'a' [i64]
| [
| 2
| null
| ]
|
| gather_every(self, n: 'int', offset: 'int' = 0) -> 'Series'
| Take every nth value in the Series and return as new Series.
|
| Parameters
| ----------
| n
| Gather every *n*-th row.
| offset
| Start the row index at this offset.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4])
| >>> s.gather_every(2)
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 3
| ]
| >>> s.gather_every(2, offset=1)
| shape: (2,)
| Series: 'a' [i64]
| [
| 2
| 4
| ]
|
| ge(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series >= other`.
|
| get_chunks(self) -> 'list_[Series]'
| Get the chunks of this Series as a list of Series.
|
| Examples
| --------
| >>> s1 = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("a", [4, 5, 6])
| >>> s = pl.concat([s1, s2], rechunk=False)
| >>> s.get_chunks()
| [shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ], shape: (3,)
| Series: 'a' [i64]
| [
| 4
| 5
| 6
| ]]
|
| gt(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series > other`.
|
| has_nulls(self) -> 'bool'
| Check whether the Series contains one or more null values.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, None])
| >>> s.has_nulls()
| True
| >>> s[:2].has_nulls()
| False
|
| has_validity(self) -> 'bool'
| Check whether the Series contains one or more null values.
|
| .. deprecated:: 0.20.30
| Use the :meth:`has_nulls` method instead.
|
| hash(self, seed: 'int' = 0, seed_1: 'int | None' = None, seed_2: 'int | None' = None, seed_3: 'int | None' = None) -> 'Series'
| Hash the Series.
|
| The hash value is of type `UInt64`.
|
| Parameters
| ----------
| seed
| Random seed parameter. Defaults to 0.
| seed_1
| Random seed parameter. Defaults to `seed` if not set.
| seed_2
| Random seed parameter. Defaults to `seed` if not set.
| seed_3
| Random seed parameter. Defaults to `seed` if not set.
|
| Notes
| -----
| This implementation of `hash` does not guarantee stable results
| across different Polars versions. Its stability is only guaranteed within a
| single version.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.hash(seed=42) # doctest: +IGNORE_RESULT
| shape: (3,)
| Series: 'a' [u64]
| [
| 10734580197236529959
| 3022416320763508302
| 13756996518000038261
| ]
|
| head(self, n: 'int' = 10) -> 'Series'
| Get the first `n` elements.
|
| Parameters
| ----------
| n
| Number of elements to return. If a negative value is passed, return all
| elements except the last `abs(n)`.
|
| See Also
| --------
| tail, slice
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.head(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| Pass a negative value to get all rows `except` the last `abs(n)`.
|
| >>> s.head(-3)
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 2
| ]
|
| hist(self, bins: 'list_[float] | None' = None, *, bin_count: 'int | None' = None, include_category: 'bool' = True, include_breakpoint: 'bool' = True) -> 'DataFrame'
| Bin values into buckets and count their occurrences.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| bins
| Bin edges. If None given, we determine the edges based on the data.
| bin_count
| If `bins` is not provided, `bin_count` uniform bins are created that fully
| encompass the data.
| include_breakpoint
| Include a column that indicates the upper breakpoint.
| include_category
| Include a column that shows the intervals as categories.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> a = pl.Series("a", [1, 3, 8, 8, 2, 1, 3])
| >>> a.hist(bin_count=4)
| shape: (4, 3)
| ┌────────────┬─────────────┬───────┐
| │ breakpoint ┆ category ┆ count │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ cat ┆ u32 │
| ╞════════════╪═════════════╪═══════╡
| │ 2.75 ┆ [1.0, 2.75] ┆ 3 │
| │ 4.5 ┆ (2.75, 4.5] ┆ 2 │
| │ 6.25 ┆ (4.5, 6.25] ┆ 0 │
| │ 8.0 ┆ (6.25, 8.0] ┆ 2 │
| └────────────┴─────────────┴───────┘
|
| implode(self) -> 'Self'
| Aggregate values into a list.
|
| The returned list itself is a scalar value of `list` dtype.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.implode()
| shape: (1,)
| Series: 'a' [list[i64]]
| [
| [1, 2, 3]
| ]
|
| index_of(self, element: 'IntoExpr') -> 'int | None'
| Get the index of the first occurrence of a value, or ``None`` if it's not found.
|
| Parameters
| ----------
| element
| Value to find.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, None, 17])
| >>> s.index_of(17)
| 2
| >>> s.index_of(None) # search for a null
| 1
| >>> s.index_of(55) is None
| True
|
| interpolate(self, method: 'InterpolationMethod' = 'linear') -> 'Series'
| Interpolate intermediate values.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| method : {'linear', 'nearest'}
| Interpolation method.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, None, None, 5])
| >>> s.interpolate()
| shape: (5,)
| Series: 'a' [f64]
| [
| 1.0
| 2.0
| 3.0
| 4.0
| 5.0
| ]
|
| interpolate_by(self, by: 'IntoExpr') -> 'Series'
| Interpolate intermediate values with x-coordinate based on another column.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| by
| Column to interpolate values based on.
|
| Examples
| --------
| Fill null values using linear interpolation.
|
| >>> s = pl.Series([1, None, None, 3])
| >>> by = pl.Series([1, 2, 7, 8])
| >>> s.interpolate_by(by)
| shape: (4,)
| Series: '' [f64]
| [
| 1.0
| 1.285714
| 2.714286
| 3.0
| ]
|
| is_between(self, lower_bound: 'IntoExpr', upper_bound: 'IntoExpr', closed: 'ClosedInterval' = 'both') -> 'Series'
| Get a boolean mask of the values that are between the given lower/upper bounds.
|
| Parameters
| ----------
| lower_bound
| Lower bound value. Accepts expression input. Non-expression inputs
| (including strings) are parsed as literals.
| upper_bound
| Upper bound value. Accepts expression input. Non-expression inputs
| (including strings) are parsed as literals.
| closed : {'both', 'left', 'right', 'none'}
| Define which sides of the interval are closed (inclusive).
|
| Notes
| -----
| If the value of the `lower_bound` is greater than that of the `upper_bound`
| then the result will be False, as no value can satisfy the condition.
|
| Examples
| --------
| >>> s = pl.Series("num", [1, 2, 3, 4, 5])
| >>> s.is_between(2, 4)
| shape: (5,)
| Series: 'num' [bool]
| [
| false
| true
| true
| true
| false
| ]
|
| Use the `closed` argument to include or exclude the values at the bounds:
|
| >>> s.is_between(2, 4, closed="left")
| shape: (5,)
| Series: 'num' [bool]
| [
| false
| true
| true
| false
| false
| ]
|
| You can also use strings as well as numeric/temporal values:
|
| >>> s = pl.Series("s", ["a", "b", "c", "d", "e"])
| >>> s.is_between("b", "d", closed="both")
| shape: (5,)
| Series: 's' [bool]
| [
| false
| true
| true
| true
| false
| ]
|
| is_close(self, other: 'IntoExpr', *, abs_tol: 'float' = 0.0, rel_tol: 'float' = 1e-09, nans_equal: 'bool' = False) -> 'Series'
| Get a boolean mask of the values being close to the other values.
|
| Two values `a` and `b` are considered close if the following condition holds:
|
| .. math::
| |a-b| \le max \{ \text{rel_tol} \cdot max \{ |a|, |b| \}, \text{abs_tol} \}
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
| abs_tol
| Absolute tolerance. This is the maximum allowed absolute difference between
| two values. Must be non-negative.
| rel_tol
| Relative tolerance. This is the maximum allowed difference between two
| values, relative to the larger absolute value. Must be non-negative.
| nans_equal
| Whether NaN values should be considered equal.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Notes
| -----
| The implementation of this method is symmetric and mirrors the behavior of
| :meth:`math.isclose`. Specifically note that this behavior is different to
| :meth:`numpy.isclose`.
|
| Examples
| --------
| >>> s = pl.Series("s", [1.0, 1.2, 1.4, 1.45, 1.6])
| >>> s.is_close(1.4, abs_tol=0.1)
| shape: (5,)
| Series: 's' [bool]
| [
| false
| false
| true
| true
| false
| ]
|
| is_duplicated(self) -> 'Series'
| Get mask of all duplicated values.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.is_duplicated()
| shape: (4,)
| Series: 'a' [bool]
| [
| false
| true
| true
| false
| ]
|
| is_empty(self, *, ignore_nulls: 'bool' = False) -> 'bool'
| Check if the Series is empty.
|
| Parameters
| ----------
| ignore_nulls
| If true a series containing only nulls will also be considered empty.
| The default is false.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Examples
| --------
| >>> s = pl.Series("a", [], dtype=pl.Float32)
| >>> s.is_empty()
| True
| >>> s = pl.Series("a", [None], dtype=pl.Float32)
| >>> s.is_empty()
| False
| >>> s.is_empty(ignore_nulls=True)
| True
|
| is_finite(self) -> 'Series'
| Returns a boolean Series indicating which values are finite.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> import numpy as np
| >>> s = pl.Series("a", [1.0, 2.0, np.inf])
| >>> s.is_finite()
| shape: (3,)
| Series: 'a' [bool]
| [
| true
| true
| false
| ]
|
| is_first_distinct(self) -> 'Series'
| Return a boolean mask indicating the first occurrence of each distinct value.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series([1, 1, 2, 3, 2])
| >>> s.is_first_distinct()
| shape: (5,)
| Series: '' [bool]
| [
| true
| false
| true
| true
| false
| ]
|
| is_in(self, other: 'Series | Collection[Any]', *, nulls_equal: 'bool' = False) -> 'Series'
| Check if elements of this Series are in the other Series.
|
| Parameters
| ----------
| other
| A Series or collection to search in.
| nulls_equal : bool, default False
| If True, treat null as a distinct value. Null values will not propagate.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("b", [2, 4, None])
| >>> s2.is_in(s)
| shape: (3,)
| Series: 'b' [bool]
| [
| true
| false
| null
| ]
| >>> # when nulls_equal=True, None is treated as a distinct value
| >>> s2.is_in(s, nulls_equal=True)
| shape: (3,)
| Series: 'b' [bool]
| [
| true
| false
| false
| ]
|
| >>> # check if some values are a member of sublists
| >>> sets = pl.Series("sets", [[1, 2, 3], [1, 2], [9, 10]])
| >>> optional_members = pl.Series("optional_members", [1, 2, 3])
| >>> print(sets)
| shape: (3,)
| Series: 'sets' [list[i64]]
| [
| [1, 2, 3]
| [1, 2]
| [9, 10]
| ]
| >>> print(optional_members)
| shape: (3,)
| Series: 'optional_members' [i64]
| [
| 1
| 2
| 3
| ]
| >>> optional_members.is_in(sets)
| shape: (3,)
| Series: 'optional_members' [bool]
| [
| true
| true
| false
| ]
|
| is_infinite(self) -> 'Series'
| Returns a boolean Series indicating which values are infinite.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> import numpy as np
| >>> s = pl.Series("a", [1.0, 2.0, np.inf])
| >>> s.is_infinite()
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| false
| true
| ]
|
| is_last_distinct(self) -> 'Series'
| Return a boolean mask indicating the last occurrence of each distinct value.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series([1, 1, 2, 3, 2])
| >>> s.is_last_distinct()
| shape: (5,)
| Series: '' [bool]
| [
| false
| true
| false
| true
| true
| ]
|
| is_nan(self) -> 'Series'
| Returns a boolean Series indicating which values are NaN.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> import numpy as np
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, np.nan])
| >>> s.is_nan()
| shape: (4,)
| Series: 'a' [bool]
| [
| false
| false
| false
| true
| ]
|
| is_not_nan(self) -> 'Series'
| Returns a boolean Series indicating which values are not NaN.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> import numpy as np
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, np.nan])
| >>> s.is_not_nan()
| shape: (4,)
| Series: 'a' [bool]
| [
| true
| true
| true
| false
| ]
|
| is_not_null(self) -> 'Series'
| Returns a boolean Series indicating which values are not null.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, None])
| >>> s.is_not_null()
| shape: (4,)
| Series: 'a' [bool]
| [
| true
| true
| true
| false
| ]
|
| is_null(self) -> 'Series'
| Returns a boolean Series indicating which values are null.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, None])
| >>> s.is_null()
| shape: (4,)
| Series: 'a' [bool]
| [
| false
| false
| false
| true
| ]
|
| is_sorted(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'bool'
| Check if the Series is sorted.
|
| Parameters
| ----------
| descending
| Check if the Series is sorted in descending order
| nulls_last
| Set nulls at the end of the Series in sorted check.
|
| Examples
| --------
| >>> s = pl.Series([1, 3, 2])
| >>> s.is_sorted()
| False
|
| >>> s = pl.Series([3, 2, 1])
| >>> s.is_sorted(descending=True)
| True
|
| is_unique(self) -> 'Series'
| Get mask of all unique values.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.is_unique()
| shape: (4,)
| Series: 'a' [bool]
| [
| true
| false
| false
| true
| ]
|
| item(self, index: 'int | None' = None) -> 'Any'
| Return the Series as a scalar, or return the element at the given index.
|
| If no index is provided, this is equivalent to `s[0]`, with a check
| that the shape is (1,). With an index, this is equivalent to `s[index]`.
|
| Examples
| --------
| >>> s1 = pl.Series("a", [1])
| >>> s1.item()
| 1
| >>> s2 = pl.Series("a", [9, 8, 7])
| >>> s2.cum_sum().item(-1)
| 24
|
| kurtosis(self, *, fisher: 'bool' = True, bias: 'bool' = True) -> 'float | None'
| Compute the kurtosis (Fisher or Pearson) of a dataset.
|
| Kurtosis is the fourth central moment divided by the square of the
| variance. If Fisher's definition is used, then 3.0 is subtracted from
| the result to give 0.0 for a normal distribution.
| If bias is False then the kurtosis is calculated using k statistics to
| eliminate bias coming from biased moment estimators
|
| See scipy.stats for more information
|
| Parameters
| ----------
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Examples
| --------
| >>> s = pl.Series("grades", [66, 79, 54, 97, 96, 70, 69, 85, 93, 75])
| >>> s.kurtosis()
| -1.0522623626787952
| >>> s.kurtosis(fisher=False)
| 1.9477376373212048
| >>> s.kurtosis(fisher=False, bias=False)
| 2.1040361802642717
|
| last(self, *, ignore_nulls: 'bool' = False) -> 'PythonLiteral | None'
| Get the last element of the Series.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the last non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Returns `None` if the Series is empty.
|
| le(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series <= other`.
|
| len(self) -> 'int'
| Return the number of elements in the Series.
|
| Null values count towards the total.
|
| See Also
| --------
| count
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, None])
| >>> s.len()
| 3
|
| limit(self, n: 'int' = 10) -> 'Series'
| Get the first `n` elements.
|
| Alias for :func:`Series.head`.
|
| Parameters
| ----------
| n
| Number of elements to return. If a negative value is passed, return all
| elements except the last `abs(n)`.
|
| See Also
| --------
| head
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.limit(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| Pass a negative value to get all rows `except` the last `abs(n)`.
|
| >>> s.limit(-3)
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 2
| ]
|
| log(self, base: 'float | Series' = 2.718281828459045) -> 'Series'
| Compute the logarithm to a given base.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.log()
| shape: (3,)
| Series: '' [f64]
| [
| 0.0
| 0.693147
| 1.098612
| ]
|
| log10(self) -> 'Series'
| Compute the base 10 logarithm of the input array, element-wise.
|
| Examples
| --------
| >>> s = pl.Series([10, 100, 1000])
| >>> s.log10()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 2.0
| 3.0
| ]
|
| log1p(self) -> 'Series'
| Compute the natural logarithm of the input array plus one, element-wise.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.log1p()
| shape: (3,)
| Series: '' [f64]
| [
| 0.693147
| 1.098612
| 1.386294
| ]
|
| lower_bound(self) -> 'Self'
| Return the lower bound of this Series' dtype as a unit Series.
|
| See Also
| --------
| upper_bound : return the upper bound of the given Series' dtype.
|
| Examples
| --------
| >>> s = pl.Series("s", [-1, 0, 1], dtype=pl.Int32)
| >>> s.lower_bound()
| shape: (1,)
| Series: 's' [i32]
| [
| -2147483648
| ]
|
| >>> s = pl.Series("s", [1.0, 2.5, 3.0], dtype=pl.Float32)
| >>> s.lower_bound()
| shape: (1,)
| Series: 's' [f32]
| [
| -inf
| ]
|
| lt(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series < other`.
|
| map_elements(self, function: 'Callable[[Any], Any]', return_dtype: 'PolarsDataType | None' = None, *, skip_nulls: 'bool' = True, _disable_inefficient_map_warning: 'bool' = False) -> 'Self'
| Map a custom/user-defined function (UDF) over elements in this Series.
|
| .. warning::
| This method is much slower than the native expressions API.
| Only use it if you cannot implement your logic otherwise.
|
| Suppose that the function is: `x ↦ sqrt(x)`:
|
| - For mapping elements of a series, consider: `s.sqrt()`.
| - For mapping inner elements of lists, consider:
| `s.list.eval(pl.element().sqrt())`.
| - For mapping elements of struct fields, consider:
| `s.struct.field("field_name").sqrt()`.
|
| If the function returns a different datatype, the return_dtype arg should
| be set, otherwise the method will fail.
|
| Implementing logic using a Python function is almost always *significantly*
| slower and more memory intensive than implementing the same logic using
| the native expression API because:
|
| - The native expression engine runs in Rust; UDFs run in Python.
| - Use of Python UDFs forces the DataFrame to be materialized in memory.
| - Polars-native expressions can be parallelised (UDFs typically cannot).
| - Polars-native expressions can be logically optimised (UDFs cannot).
|
| Wherever possible you should strongly prefer the native expression API
| to achieve the best performance.
|
| Parameters
| ----------
| function
| Custom function or lambda.
| return_dtype
| Output datatype.
| If not set, the dtype will be inferred based on the first non-null value
| that is returned by the function.
| skip_nulls
| Nulls will be skipped and not passed to the python function.
| This is faster because python can be skipped and because we call
| more specialized functions.
|
| Warnings
| --------
| If `return_dtype` is not provided, this may lead to unexpected results.
| We allow this, but it is considered a bug in the user's query.
|
| Notes
| -----
| * If your function is expensive and you don't want it to be called more than
| once for a given input, consider applying an `@lru_cache` decorator to it.
| If your data is suitable you may achieve *significant* speedups.
|
| * A UDF passed to `map_elements` must be pure, meaning that it cannot modify
| or depend on state other than its arguments.
|
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.map_elements(lambda x: x + 10, return_dtype=pl.Int64) # doctest: +SKIP
| shape: (3,)
| Series: 'a' [i64]
| [
| 11
| 12
| 13
| ]
|
| Returns
| -------
| Series
|
| max(self) -> 'PythonLiteral | None'
| Get the maximum value in this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.max()
| 3
|
| max_by(self, by: 'IntoExpr') -> 'Expr'
| Get the maximum value in this Series, ordered by an expression.
|
| If the by expression has multiple values equal to the maximum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the largest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> s = pl.Series("a", [-2.0, float("nan"), 1.0])
| >>> s.max_by(pl.col.a.abs())
| -2.0
|
| mean(self) -> 'PythonLiteral | None'
| Reduce this Series to the mean value.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.mean()
| 2.0
|
| median(self) -> 'PythonLiteral | None'
| Get the median of this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.median()
| 2.0
|
| min(self) -> 'PythonLiteral | None'
| Get the minimal value in this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.min()
| 1
|
| min_by(self, by: 'IntoExpr') -> 'Expr'
| Get the minimum value in this Series, ordered by an expression.
|
| If the by expression has multiple values equal to the minimum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the smallest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> s = pl.Series("a", [-2.0, float("nan"), 1.0])
| >>> s.min_by(pl.col.a.abs())
| 1.0
|
| mode(self, *, maintain_order: 'bool' = False) -> 'Series'
| Compute the most occurring value(s).
|
| Can return multiple Values.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.mode()
| shape: (1,)
| Series: 'a' [i64]
| [
| 2
| ]
|
| n_chunks(self) -> 'int'
| Get the number of chunks that this Series contains.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.n_chunks()
| 1
| >>> s2 = pl.Series("a", [4, 5, 6])
|
| Concatenate Series with rechunk = True
|
| >>> pl.concat([s, s2], rechunk=True).n_chunks()
| 1
|
| Concatenate Series with rechunk = False
|
| >>> pl.concat([s, s2], rechunk=False).n_chunks()
| 2
|
| n_unique(self) -> 'int'
| Count the number of unique values in this Series.
|
| `null` is considered to be a unique value for the purposes of this operation.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, None])
| >>> s.n_unique()
| 3
|
| nan_max(self) -> 'int | float | date | datetime | timedelta | str_'
| Get maximum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 3, 4])
| >>> s.nan_max()
| 4
|
| >>> s = pl.Series("a", [1.0, float("nan"), 4.0])
| >>> s.nan_max()
| nan
|
| nan_min(self) -> 'int | float | date | datetime | timedelta | str_'
| Get minimum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 3, 4])
| >>> s.nan_min()
| 1
|
| >>> s = pl.Series("a", [1.0, float("nan"), 4.0])
| >>> s.nan_min()
| nan
|
| ne(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series != other`.
|
| ne_missing(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of equality operator `series != other` where `None == None`.
|
| This differs from the standard `ne` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| See Also
| --------
| eq_missing
| ne
|
| Examples
| --------
| >>> s1 = pl.Series("a", [333, 200, None])
| >>> s2 = pl.Series("a", [100, 200, None])
| >>> s1.ne(s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| true
| false
| null
| ]
| >>> s1.ne_missing(s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| true
| false
| false
| ]
|
| new_from_index(self, index: 'int', length: 'int') -> 'Self'
| Create a new Series filled with values from the given index.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.new_from_index(1, 3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 2
| 2
| 2
| ]
|
| not_(self) -> 'Series'
| Negate a boolean Series.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [True, False, False])
| >>> s.not_()
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| true
| true
| ]
|
| null_count(self) -> 'int'
| Count the null values in this Series.
|
| Examples
| --------
| >>> s = pl.Series([1, None, None])
| >>> s.null_count()
| 2
|
| pct_change(self, n: 'int | IntoExprColumn' = 1) -> 'Series'
| Computes percentage change between values.
|
| Percentage change (as fraction) between current element and most-recent
| non-null element at least `n` period(s) before the current element.
|
| Computes the change from the previous row by default.
|
| Parameters
| ----------
| n
| periods to shift for forming percent change.
|
| Notes
| -----
| Null values are preserved. If you're coming from pandas, this matches
| their ``fill_method=None`` behaviour.
|
| Examples
| --------
| >>> pl.Series(range(10)).pct_change()
| shape: (10,)
| Series: '' [f64]
| [
| null
| inf
| 1.0
| 0.5
| 0.333333
| 0.25
| 0.2
| 0.166667
| 0.142857
| 0.125
| ]
|
| >>> pl.Series([1, 2, 4, 8, 16, 32, 64, 128, 256, 512]).pct_change(2)
| shape: (10,)
| Series: '' [f64]
| [
| null
| null
| 3.0
| 3.0
| 3.0
| 3.0
| 3.0
| 3.0
| 3.0
| 3.0
| ]
|
| peak_max(self) -> 'Self'
| Get a boolean mask of the local maximum peaks.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.peak_max()
| shape: (5,)
| Series: 'a' [bool]
| [
| false
| false
| false
| false
| true
| ]
|
| peak_min(self) -> 'Self'
| Get a boolean mask of the local minimum peaks.
|
| Examples
| --------
| >>> s = pl.Series("a", [4, 1, 3, 2, 5])
| >>> s.peak_min()
| shape: (5,)
| Series: 'a' [bool]
| [
| false
| true
| false
| true
| false
| ]
|
| pow(self, exponent: 'int | float | Series') -> 'Series'
| Raise to the power of the given exponent.
|
| If the exponent is float, the result follows the dtype of exponent.
| Otherwise, it follows dtype of base.
|
| Parameters
| ----------
| exponent
| The exponent. Accepts Series input.
|
| Examples
| --------
| Raising integers to positive integers results in integers:
|
| >>> s = pl.Series("foo", [1, 2, 3, 4])
| >>> s.pow(3)
| shape: (4,)
| Series: 'foo' [i64]
| [
| 1
| 8
| 27
| 64
| ]
|
| In order to raise integers to negative integers, you can cast either the
| base or the exponent to float:
|
| >>> s.pow(-3.0)
| shape: (4,)
| Series: 'foo' [f64]
| [
| 1.0
| 0.125
| 0.037037
| 0.015625
| ]
|
| product(self) -> 'int | float'
| Reduce this Series to the product value.
|
| Notes
| -----
| If there are no non-null values, then the output is `1`.
| If you would prefer empty products to return `None`, you can
| use `s.product() if s.count() else None` instead
| of `s.product()`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.product()
| 6
|
| qcut(self, quantiles: 'Sequence[float] | int', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, allow_duplicates: 'bool' = False, include_breaks: 'bool' = False) -> 'Series'
| Bin continuous values into discrete categories based on their quantiles.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| quantiles
| Either a list of quantile probabilities between 0 and 1 or a positive
| integer determining the number of bins with uniform probability.
| labels
| Names of the categories. The number of labels must be equal to the number
| of cut points plus one.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| allow_duplicates
| If set to `True`, duplicates in the resulting quantiles are dropped,
| rather than raising a `DuplicateError`. This can happen even with unique
| probabilities, depending on the data.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from a
| :class:`Categorical` to a :class:`Struct`.
|
| Returns
| -------
| Series
| Series of data type :class:`Categorical` if `include_breaks` is set to
| `False` (default), otherwise a Series of data type :class:`Struct`.
|
| See Also
| --------
| cut
|
| Examples
| --------
| Divide a column into three categories according to pre-defined quantile
| probabilities.
|
| >>> s = pl.Series("foo", [-2, -1, 0, 1, 2])
| >>> s.qcut([0.25, 0.75], labels=["a", "b", "c"])
| shape: (5,)
| Series: 'foo' [cat]
| [
| "a"
| "a"
| "b"
| "b"
| "c"
| ]
|
| Divide a column into two categories using uniform quantile probabilities.
|
| >>> s.qcut(2, labels=["low", "high"], left_closed=True)
| shape: (5,)
| Series: 'foo' [cat]
| [
| "low"
| "low"
| "high"
| "high"
| "high"
| ]
|
| Create a DataFrame with the breakpoint and category for each value.
|
| >>> cut = s.qcut([0.25, 0.75], include_breaks=True).alias("cut")
| >>> s.to_frame().with_columns(cut).unnest("cut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ cat │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| quantile(self, quantile: 'float | list_[float]', interpolation: 'QuantileMethod' = 'nearest') -> 'float | None | list_[float] | list_[None]'
| Get the quantile value of this Series.
|
| Parameters
| ----------
| quantile
| Quantile(s) between 0.0 and 1.0. Can be a single float or a list of floats.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
|
| Returns
| -------
| float | None | list[float] | list[None]
| A single quantile value if a float is provided, or a list of quantile values if a list is provided.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.quantile(0.5)
| 2.0
|
| Return a list of quantiles:
|
| >>> s.quantile([0.25, 0.75], interpolation="linear")
| [1.5, 2.5]
|
| radians(self) -> 'Series'
| Convert from degrees to radians.
|
| Examples
| --------
| >>> s = pl.Series("a", [-720, -540, -360, -180, 0, 180, 360, 540, 720])
| >>> s.radians()
| shape: (9,)
| Series: 'a' [f64]
| [
| -12.566371
| -9.424778
| -6.283185
| -3.141593
| 0.0
| 3.141593
| 6.283185
| 9.424778
| 12.566371
| ]
|
| rank(self, method: 'RankMethod' = 'average', *, descending: 'bool' = False, seed: 'int | None' = None) -> 'Series'
| Assign ranks to data, dealing with ties appropriately.
|
| Parameters
| ----------
| method : {'average', 'min', 'max', 'dense', 'ordinal', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'ordinal' : All values are given a distinct rank, corresponding to
| the order that the values occur in the Series.
| - 'random' : Like 'ordinal', but the rank for ties is not dependent
| on the order that the values occur in the Series.
| descending
| Rank in descending order.
| seed
| If `method="random"`, use this as seed.
|
| Notes
| -----
| If you're coming from SQL, you may be expecting null values to be ranked last.
| Polars, however, only ranks non-null values and preserves the null ones.
|
| Examples
| --------
| The 'average' method:
|
| >>> s = pl.Series("a", [3, 6, 1, 1, 6])
| >>> s.rank()
| shape: (5,)
| Series: 'a' [f64]
| [
| 3.0
| 4.5
| 1.5
| 1.5
| 4.5
| ]
|
| The 'ordinal' method:
|
| >>> s = pl.Series("a", [3, 6, 1, 1, 6])
| >>> s.rank("ordinal")
| shape: (5,)
| Series: 'a' [u32]
| [
| 3
| 4
| 1
| 2
| 5
| ]
|
| rechunk(self, *, in_place: 'bool' = False) -> 'Self'
| Create a single chunk of memory for this Series.
|
| Parameters
| ----------
| in_place
| In place or not.
|
| Examples
| --------
| >>> s1 = pl.Series("a", [1, 2, 3])
| >>> s1.n_chunks()
| 1
| >>> s2 = pl.Series("a", [4, 5, 6])
| >>> s = pl.concat([s1, s2], rechunk=False)
| >>> s.n_chunks()
| 2
| >>> s.rechunk(in_place=True)
| shape: (6,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| ]
| >>> s.n_chunks()
| 1
|
| reinterpret(self, *, signed: 'bool | None' = None, dtype: 'type[int | float] | PolarsDataType | None' = None) -> 'Series'
| Reinterpret the underlying bits as a signed/unsigned integer or float.
|
| This operation is only allowed for numeric types of the same size.
| For lower bits numbers, you can safely use the cast operation.
|
| Either `signed` or `dtype` can be specified.
| Defaults to `signed=True` otherwise.
|
| Parameters
| ----------
| signed
| If True, reinterpret as signed integer. Otherwise, reinterpret
| as unsigned integer.
| dtype
| DataType to reinterpret to.
|
| Examples
| --------
| >>> s = pl.Series("a", [-(2**60), -2, 3])
| >>> s
| shape: (3,)
| Series: 'a' [i64]
| [
| -1152921504606846976
| -2
| 3
| ]
| >>> s.reinterpret(signed=False)
| shape: (3,)
| Series: 'a' [u64]
| [
| 17293822569102704640
| 18446744073709551614
| 3
| ]
| >>> s.reinterpret(dtype=pl.Int64)
| shape: (3,)
| Series: 'a' [i64]
| [
| -1152921504606846976
| -2
| 3
| ]
|
| rename(self, name: 'str_') -> 'Series'
| Rename this Series.
|
| Alias for :func:`Series.alias`.
|
| Parameters
| ----------
| name
| New name.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.rename("b")
| shape: (3,)
| Series: 'b' [i64]
| [
| 1
| 2
| 3
| ]
|
| repeat_by(self, by: 'int | IntoExprColumn') -> 'Self'
| Repeat the elements in this Series as specified in the given expression.
|
| The repeated elements are expanded into a List.
|
| Parameters
| ----------
| by
| Numeric column that determines how often the values will be repeated.
| The column will be coerced to UInt32. Give this dtype to make the coercion
| a no-op.
|
| Returns
| -------
| Expr
| Expression of data type List, where the inner data type is equal to the
| original data type.
|
| replace(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | None' = None) -> 'Self'
| Replace values by different values of the same data type.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Length must match the length of `old` or have length 1.
|
| default
| Set values that were not replaced to this value.
| Defaults to keeping the original value.
| Accepts expression input. Non-expression inputs are parsed as literals.
|
| .. deprecated:: 0.20.31
| Use :meth:`replace_strict` instead to set a default while
| replacing values.
|
| return_dtype
| The data type of the resulting expression. If set to `None` (default),
| the data type is determined automatically based on the other inputs.
|
| .. deprecated:: 0.20.31
| Use :meth:`replace_strict` instead to set a return data type while
| replacing values.
|
|
| See Also
| --------
| replace_strict
| str.replace
|
| Examples
| --------
| Replace a single value by another value. Values that were not replaced remain
| unchanged.
|
| >>> s = pl.Series([1, 2, 2, 3])
| >>> s.replace(2, 100)
| shape: (4,)
| Series: '' [i64]
| [
| 1
| 100
| 100
| 3
| ]
|
| Replace multiple values by passing sequences to the `old` and `new` parameters.
|
| >>> s.replace([2, 3], [100, 200])
| shape: (4,)
| Series: '' [i64]
| [
| 1
| 100
| 100
| 200
| ]
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {2: 100, 3: 200}
| >>> s.replace(mapping)
| shape: (4,)
| Series: '' [i64]
| [
| 1
| 100
| 100
| 200
| ]
|
| The original data type is preserved when replacing by values of a different
| data type. Use :meth:`replace_strict` to replace and change the return data
| type.
|
| >>> s = pl.Series(["x", "y", "z"])
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> s.replace(mapping)
| shape: (3,)
| Series: '' [str]
| [
| "1"
| "2"
| "3"
| ]
|
| replace_strict(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | None' = None) -> 'Self'
| Replace all values by different values.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace_strict(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Length must match the length of `old` or have length 1.
| default
| Set values that were not replaced to this value. If no default is specified,
| (default), an error is raised if any values were not replaced.
| Accepts expression input. Non-expression inputs are parsed as literals.
| return_dtype
| The data type of the resulting Series. If set to `None` (default),
| the data type is determined automatically based on the other inputs.
|
| Raises
| ------
| InvalidOperationError
| If any non-null values in the original column were not replaced, and no
| `default` was specified.
|
| See Also
| --------
| replace
| str.replace
|
| Examples
| --------
| Replace values by passing sequences to the `old` and `new` parameters.
|
| >>> s = pl.Series([1, 2, 2, 3])
| >>> s.replace_strict([1, 2, 3], [100, 200, 300])
| shape: (4,)
| Series: '' [i64]
| [
| 100
| 200
| 200
| 300
| ]
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {1: 100, 2: 200, 3: 300}
| >>> s.replace_strict(mapping)
| shape: (4,)
| Series: '' [i64]
| [
| 100
| 200
| 200
| 300
| ]
|
| By default, an error is raised if any non-null values were not replaced.
| Specify a default to set all values that were not matched.
|
| >>> mapping = {2: 200, 3: 300}
| >>> s.replace_strict(mapping) # doctest: +SKIP
| Traceback (most recent call last):
| ...
| polars.exceptions.InvalidOperationError: incomplete mapping specified for `replace_strict`
| >>> s.replace_strict(mapping, default=-1)
| shape: (4,)
| Series: '' [i64]
| [
| -1
| 200
| 200
| 300
| ]
|
| The default can be another Series.
|
| >>> default = pl.Series([2.5, 5.0, 7.5, 10.0])
| >>> s.replace_strict(2, 200, default=default)
| shape: (4,)
| Series: '' [f64]
| [
| 2.5
| 200.0
| 200.0
| 10.0
| ]
|
| Replacing by values of a different data type sets the return type based on
| a combination of the `new` data type and the `default` data type.
|
| >>> s = pl.Series(["x", "y", "z"])
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> s.replace_strict(mapping)
| shape: (3,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| ]
| >>> s.replace_strict(mapping, default="x")
| shape: (3,)
| Series: '' [str]
| [
| "1"
| "2"
| "3"
| ]
|
| Set the `return_dtype` parameter to control the resulting data type directly.
|
| >>> s.replace_strict(mapping, return_dtype=pl.UInt8)
| shape: (3,)
| Series: '' [u8]
| [
| 1
| 2
| 3
| ]
|
| reshape(self, dimensions: 'tuple[int, ...]') -> 'Series'
| Reshape this Series to a flat Series or an Array Series.
|
| Parameters
| ----------
| dimensions
| Tuple of the dimension sizes. If a -1 is used in any of the dimensions, that
| dimension is inferred.
|
| Returns
| -------
| Series
| If a single dimension is given, results in a Series of the original
| data type.
| If a multiple dimensions are given, results in a Series of data type
| :class:`Array` with shape `dimensions`.
|
| See Also
| --------
| Series.list.explode : Explode a list column.
|
| Examples
| --------
| >>> s = pl.Series("foo", [1, 2, 3, 4, 5, 6, 7, 8, 9])
| >>> square = s.reshape((3, 3))
| >>> square
| shape: (3,)
| Series: 'foo' [array[i64, 3]]
| [
| [1, 2, 3]
| [4, 5, 6]
| [7, 8, 9]
| ]
| >>> square.reshape((9,))
| shape: (9,)
| Series: 'foo' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| 7
| 8
| 9
| ]
|
| reverse(self) -> 'Series'
| Return Series in reverse order.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3], dtype=pl.Int8)
| >>> s.reverse()
| shape: (3,)
| Series: 'a' [i8]
| [
| 3
| 2
| 1
| ]
|
| rle(self) -> 'Series'
| Compress the Series data using run-length encoding.
|
| Run-length encoding (RLE) encodes data by storing each *run* of identical values
| as a single value and its length.
|
| Returns
| -------
| Series
| Series of data type `Struct` with fields `len` of data type `UInt32`
| and `value` of the original data type.
|
| Examples
| --------
| >>> s = pl.Series("s", [1, 1, 2, 1, None, 1, 3, 3])
| >>> s.rle().struct.unnest()
| shape: (6, 2)
| ┌─────┬───────┐
| │ len ┆ value │
| │ --- ┆ --- │
| │ u32 ┆ i64 │
| ╞═════╪═══════╡
| │ 2 ┆ 1 │
| │ 1 ┆ 2 │
| │ 1 ┆ 1 │
| │ 1 ┆ null │
| │ 1 ┆ 1 │
| │ 2 ┆ 3 │
| └─────┴───────┘
|
| rle_id(self) -> 'Series'
| Get a distinct integer ID for each run of identical values.
|
| The ID starts at 0 and increases by one each time the value of the column
| changes.
|
| Returns
| -------
| Series
| Series of data type `UInt32`.
|
| See Also
| --------
| rle
|
| Notes
| -----
| This functionality is especially useful for defining a new group for every time
| a column's value changes, rather than for every distinct value of that column.
|
| Examples
| --------
| >>> s = pl.Series("s", [1, 1, 2, 1, None, 1, 3, 3])
| >>> s.rle_id()
| shape: (8,)
| Series: 's' [u32]
| [
| 0
| 0
| 1
| 2
| 3
| 4
| 5
| 5
| ]
|
| rolling_kurtosis(self, window_size: 'int', *, fisher: 'bool' = True, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling kurtosis.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Series.kurtosis
|
| Examples
| --------
| >>> pl.Series([1, 4, 2, 9]).rolling_kurtosis(3)
| shape: (4,)
| Series: '' [f64]
| [
| null
| null
| -1.5
| -1.5
| ]
|
| rolling_map(self, function: 'Callable[[Series], Any]', window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a custom rolling window function.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| function
| Custom aggregation function.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Warnings
| --------
| Computing custom functions is extremely slow. Use specialized rolling
| functions such as :func:`Series.rolling_sum` if at all possible.
|
| Examples
| --------
| >>> from numpy import nansum
| >>> s = pl.Series([11.0, 2.0, 9.0, float("nan"), 8.0])
| >>> s.rolling_map(nansum, window_size=3)
| shape: (5,)
| Series: '' [f64]
| [
| null
| null
| 22.0
| 11.0
| 17.0
| ]
|
| rolling_max(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Apply a rolling max (moving max) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their max.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [100, 200, 300, 400, 500])
| >>> s.rolling_max(window_size=2)
| shape: (5,)
| Series: 'a' [i64]
| [
| null
| 200
| 300
| 400
| 500
| ]
|
| rolling_max_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling max based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling max with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_max_by(d, "3h")
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| rolling_mean(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Apply a rolling mean (moving mean) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their mean.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [100, 200, 300, 400, 500])
| >>> s.rolling_mean(window_size=2)
| shape: (5,)
| Series: 'a' [f64]
| [
| null
| 150.0
| 250.0
| 350.0
| 450.0
| ]
|
| rolling_mean_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling mean based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling mean with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_mean_by(d, "3h")
| shape: (25,)
| Series: 'index' [f64]
| [
| 0.0
| 0.5
| 1.0
| 2.0
| 3.0
| …
| 19.0
| 20.0
| 21.0
| 22.0
| 23.0
| ]
|
| rolling_median(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling median.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
| >>> s.rolling_median(window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 2.0
| 3.0
| 4.0
| 6.0
| ]
|
| rolling_median_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling median based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling median with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_median_by(d, "3h")
| shape: (25,)
| Series: 'index' [f64]
| [
| 0.0
| 0.5
| 1.0
| 2.0
| 3.0
| …
| 19.0
| 20.0
| 21.0
| 22.0
| 23.0
| ]
|
| rolling_min(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Apply a rolling min (moving min) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their min.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [100, 200, 300, 400, 500])
| >>> s.rolling_min(window_size=3)
| shape: (5,)
| Series: 'a' [i64]
| [
| null
| null
| 100
| 200
| 300
| ]
|
| rolling_min_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling min based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling min with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_min_by(d, "3h")
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 0
| 0
| 1
| 2
| …
| 18
| 19
| 20
| 21
| 22
| ]
|
| rolling_quantile(self, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', window_size: 'int' = 2, weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling quantile.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
| >>> s.rolling_quantile(quantile=0.33, window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 2.0
| 3.0
| 4.0
| 6.0
| ]
| >>> s.rolling_quantile(quantile=0.33, interpolation="linear", window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 1.66
| 2.66
| 3.66
| 5.32
| ]
|
| rolling_quantile_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling quantile based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling quantile with the temporal windows from the second series closed on the right:
|
| >>> s.rolling_quantile_by(d, "3h", quantile=0.5)
| shape: (25,)
| Series: 'index' [f64]
| [
| 0.0
| 1.0
| 1.0
| 2.0
| 3.0
| …
| 19.0
| 20.0
| 21.0
| 22.0
| 23.0
| ]
|
| rolling_rank(self, window_size: 'int', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling rank.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| A window of length `window_size` will traverse the array. The values
| that fill this window will be ranked according to the `method`
| parameter. The resulting values will be the rank of the value that is
| at the end of the sliding window.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Returns
| -------
| Series
| A Series of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| Examples
| --------
| >>> pl.Series([1, 4, 4, 1, 9]).rolling_rank(3, method="average")
| shape: (5,)
| Series: '' [f64]
| [
| null
| null
| 2.5
| 1.0
| 3.0
| ]
|
| rolling_rank_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Series'
| Compute a rolling rank based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Returns
| -------
| Series
| A Series of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| rolling_skew(self, window_size: 'int', *, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling skew.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row includes the row itself and the
| `window_size - 1` elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| bias
| If False, the calculations are corrected for statistical bias.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Series.skew
|
| Examples
| --------
| >>> pl.Series([1, 4, 2, 9]).rolling_skew(3)
| shape: (4,)
| Series: '' [f64]
| [
| null
| null
| 0.381802
| 0.47033
| ]
|
| Note how the values match
|
| >>> pl.Series([1, 4, 2]).skew(), pl.Series([4, 2, 9]).skew()
| (0.38180177416060584, 0.47033046033698594)
|
| rolling_std(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Series'
| Compute a rolling std dev.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their std dev.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
| >>> s.rolling_std(window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 1.0
| 1.0
| 1.527525
| 2.0
| ]
|
| rolling_std_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Self'
| Compute a rolling standard deviation based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling std with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_std_by(d, "3h")
| shape: (25,)
| Series: 'index' [f64]
| [
| null
| 0.707107
| 1.0
| 1.0
| 1.0
| …
| 1.0
| 1.0
| 1.0
| 1.0
| 1.0
| ]
|
| rolling_sum(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Apply a rolling sum (moving sum) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their sum.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.rolling_sum(window_size=2)
| shape: (5,)
| Series: 'a' [i64]
| [
| null
| 3
| 5
| 7
| 9
| ]
|
| rolling_sum_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 0, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling sum based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling mean with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_sum_by(d, "3h")
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 3
| 6
| 9
| …
| 57
| 60
| 63
| 66
| 69
| ]
|
| rolling_var(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Series'
| Compute a rolling variance.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their variance.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
| >>> s.rolling_var(window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 1.0
| 1.0
| 2.333333
| 4.0
| ]
|
| rolling_var_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Self'
| Compute a rolling variance based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling std with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_std_by(d, "3h")
| shape: (25,)
| Series: 'index' [f64]
| [
| null
| 0.707107
| 1.0
| 1.0
| 1.0
| …
| 1.0
| 1.0
| 1.0
| 1.0
| 1.0
| ]
|
| round(self, decimals: 'int' = 0, mode: 'RoundMode' = 'half_to_even') -> 'Series'
| Round underlying floating point data by `decimals` digits.
|
| Parameters
| ----------
| decimals
| Number of decimals to round by.
| mode : {'half_to_even', 'half_away_from_zero', 'to_zero'}
| The rounding strategy used. A "rounded value" is a value with at most
| `decimals` decimal places (e.g. integers when ``decimals=0``, multiples
| of 0.1 when ``decimals=1``, 0.01 when ``decimals=2``, and so on).
|
| Strategies that start with ``half_`` round all values to the *nearest*
| rounded value, only using the strategy to break ties when a value falls
| exactly between two rounded values (e.g. 0.5 when ``decimals=0``, 0.05
| when ``decimals=1``). Other rounding strategies specify explicitly
| which rounded value is chosen and always apply (not just for tiebreaks).
|
| * *half_to_even* (default)
| Round to the nearest value; break ties by choosing the nearest
| **even** value. For example, 0.5 rounds to 0, 1.5 rounds to 2,
| 2.5 rounds to 2. Also known as "banker's rounding"; this is the
| default because it tends to minimise cumulative rounding bias.
| * *half_away_from_zero*
| Round to the nearest value; break ties by rounding **away from
| zero**. For example, 0.5 rounds to 1, -0.5 rounds to -1, 2.5
| rounds to 3. Also known as "commercial rounding".
| * *to_zero*
| Always round (truncate) **towards zero**, discarding the fractional
| part beyond `decimals`. For example, 0.9 rounds to 0, -0.9 rounds
| to 0, 1.29 rounds to 1.2 (with ``decimals=1``). Equivalent to the
| :meth:`truncate` method.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.12345, 2.56789, 3.901234])
| >>> s.round(2)
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.12
| 2.57
| 3.9
| ]
|
| >>> s = pl.Series([-3.5, -2.5, -1.5, -0.5, 0.5, 1.5, 2.5, 3.5])
| >>> s.round(mode="half_to_even")
| shape: (8,)
| Series: '' [f64]
| [
| -4.0
| -2.0
| -2.0
| -0.0
| 0.0
| 2.0
| 2.0
| 4.0
| ]
|
| round_sig_figs(self, digits: 'int') -> 'Series'
| Round to a number of significant figures.
|
| Parameters
| ----------
| digits
| Number of significant figures to round to.
|
| Examples
| --------
| >>> s = pl.Series([0.01234, 3.333, 3450.0])
| >>> s.round_sig_figs(2)
| shape: (3,)
| Series: '' [f64]
| [
| 0.012
| 3.3
| 3500.0
| ]
|
| sample(self, n: 'int | None' = None, *, fraction: 'float | None' = None, with_replacement: 'bool' = False, shuffle: 'bool | None' = None, seed: 'int | None' = None) -> 'Series'
| Sample from this Series.
|
| Parameters
| ----------
| n
| Number of items to return. Cannot be used with `fraction`. Defaults to 1 if
| `fraction` is None.
| fraction
| Fraction of items to return. Cannot be used with `n`.
| with_replacement
| Allow values to be sampled more than once.
| shuffle
| Determines the order of the sampled values.
| If True, sampled values are explicitly shuffled.
| If False, the relative order of the sampled values is preserved.
| (i.e. they appear in the same order as the original Series).
| If None (default), no ordering guarantee; uses the most performant
| algorithm.
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated for each sample operation.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.sample(2, shuffle=False, seed=0) # doctest: +IGNORE_RESULT
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 5
| ]
|
| scatter(self, indices: 'Series | Iterable[int] | int | np.ndarray[Any, Any]', values: 'Series | Iterable[PythonLiteral] | PythonLiteral | None') -> 'Series'
| Set values at the index locations.
|
| Parameters
| ----------
| indices
| Integers representing the index locations.
| values
| Replacement values.
|
| Notes
| -----
| Use of this function is frequently an anti-pattern, as it can
| block optimization (predicate pushdown, etc). Consider using
| `pl.when(predicate).then(value).otherwise(self)` instead.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.scatter(1, 10)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 10
| 3
| ]
|
| It is better to implement this as follows:
|
| >>> s.to_frame().with_row_index().select(
| ... pl.when(pl.col("index") == 1).then(10).otherwise(pl.col("a"))
| ... )
| shape: (3, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ i64 │
| ╞═════════╡
| │ 1 │
| │ 10 │
| │ 3 │
| └─────────┘
|
| search_sorted(self, element: 'IntoExpr | np.ndarray[Any, Any] | None', side: 'SearchSortedSide' = 'any', *, descending: 'bool' = False) -> 'int | Series'
| Find indices where elements should be inserted to maintain order.
|
| .. math:: a[i-1] < v <= a[i]
|
| Parameters
| ----------
| element
| Expression or scalar value.
| side : {'any', 'left', 'right'}
| If 'any', the index of the first suitable location found is given.
| If 'left', the index of the leftmost suitable location found is given.
| If 'right', return the rightmost suitable location found is given.
| descending
| Boolean indicating whether the values are descending or not (they
| are required to be sorted either way).
|
| Examples
| --------
| >>> s = pl.Series("set", [1, 2, 3, 4, 4, 5, 6, 7])
| >>> s.search_sorted(4)
| 3
| >>> s.search_sorted(4, "left")
| 3
| >>> s.search_sorted(4, "right")
| 5
| >>> s.search_sorted([1, 4, 5])
| shape: (3,)
| Series: 'set' [u32]
| [
| 0
| 3
| 5
| ]
| >>> s.search_sorted([1, 4, 5], "left")
| shape: (3,)
| Series: 'set' [u32]
| [
| 0
| 3
| 5
| ]
| >>> s.search_sorted([1, 4, 5], "right")
| shape: (3,)
| Series: 'set' [u32]
| [
| 1
| 5
| 6
| ]
|
| set(self, filter: 'Series', value: 'Any') -> 'Series'
| Set masked values.
|
| Parameters
| ----------
| filter
| Boolean mask.
| value
| Value with which to replace the masked values.
|
| Notes
| -----
| Use of this function is frequently an anti-pattern, as it can
| block optimisation (predicate pushdown, etc). Consider using
| `pl.when(predicate).then(value).otherwise(self)` instead.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.set(s == 2, 10)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 10
| 3
| ]
|
| It is better to implement this as follows:
|
| >>> s.to_frame().select(
| ... pl.when(pl.col("a") == 2).then(10).otherwise(pl.col("a"))
| ... )
| shape: (3, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ i64 │
| ╞═════════╡
| │ 1 │
| │ 10 │
| │ 3 │
| └─────────┘
|
| set_sorted(self, *, descending: 'bool' = False) -> 'Self'
| Flags the Series as 'sorted'.
|
| Enables downstream code to user fast paths for sorted arrays.
|
| Parameters
| ----------
| descending
| If the `Series` order is descending.
|
| Warnings
| --------
| This can lead to incorrect results if this `Series` is not sorted!!
| Use with care!
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.set_sorted().max()
| 3
|
| shift(self, n: 'int' = 1, *, fill_value: 'IntoExpr | None' = None) -> 'Series'
| Shift values by the given number of indices.
|
| Parameters
| ----------
| n
| Number of indices to shift forward. If a negative value is passed, values
| are shifted in the opposite direction instead.
| fill_value
| Fill the resulting null values with this value. Accepts scalar expression
| input. Non-expression inputs are parsed as literals.
|
| Notes
| -----
| This method is similar to the `LAG` operation in SQL when the value for `n`
| is positive. With a negative value for `n`, it is similar to `LEAD`.
|
| Examples
| --------
| By default, values are shifted forward by one index.
|
| >>> s = pl.Series([1, 2, 3, 4])
| >>> s.shift()
| shape: (4,)
| Series: '' [i64]
| [
| null
| 1
| 2
| 3
| ]
|
| Pass a negative value to shift in the opposite direction instead.
|
| >>> s.shift(-2)
| shape: (4,)
| Series: '' [i64]
| [
| 3
| 4
| null
| null
| ]
|
| Specify `fill_value` to fill the resulting null values.
|
| >>> s.shift(-2, fill_value=100)
| shape: (4,)
| Series: '' [i64]
| [
| 3
| 4
| 100
| 100
| ]
|
| shrink_dtype(self) -> 'Series'
| Shrink numeric columns to the minimal required datatype.
|
| Shrink to the dtype needed to fit the extrema of this [`Series`].
| This can be used to reduce memory pressure.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5, 6])
| >>> s
| shape: (6,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| ]
| >>> s.shrink_dtype()
| shape: (6,)
| Series: 'a' [i8]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| ]
|
| shrink_to_fit(self, *, in_place: 'bool' = False) -> 'Series'
| Shrink Series memory usage.
|
| Shrinks the underlying array capacity to exactly fit the actual data.
| (Note that this function does not change the Series data type).
|
| shuffle(self, seed: 'int | None' = None) -> 'Series'
| Shuffle the contents of this Series.
|
| Parameters
| ----------
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated each time the shuffle is called.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.shuffle(seed=1)
| shape: (3,)
| Series: 'a' [i64]
| [
| 2
| 3
| 1
| ]
|
| sign(self) -> 'Series'
| Compute the element-wise sign function on numeric types.
|
| The returned value is computed as follows:
|
| * -1 if x < 0.
| * 1 if x > 0.
| * x otherwise (typically 0, but could be NaN if the input is).
|
| Null values are preserved as-is, and the dtype of the input is preserved.
|
| Examples
| --------
| >>> s = pl.Series("a", [-9.0, -0.0, 0.0, 4.0, float("nan"), None])
| >>> s.sign()
| shape: (6,)
| Series: 'a' [f64]
| [
| -1.0
| -0.0
| 0.0
| 1.0
| NaN
| null
| ]
|
| sin(self) -> 'Series'
| Compute the element-wise value for the sine.
|
| Notes
| -----
| Input values are interpreted as radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Series.radians>`.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [0.0, math.pi / 2.0, math.pi])
| >>> s.sin()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.0
| 1.0
| 1.2246e-16
| ]
|
| sinh(self) -> 'Series'
| Compute the element-wise value for the hyperbolic sine.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.sinh()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.175201
| 0.0
| -1.175201
| ]
|
| skew(self, *, bias: 'bool' = True) -> 'float | None'
| Compute the sample skewness of a data set.
|
| For normally distributed data, the skewness should be about zero. For
| unimodal continuous distributions, a skewness value greater than zero means
| that there is more weight in the right tail of the distribution. The
| function `skewtest` can be used to determine if the skewness value
| is close enough to zero, statistically speaking.
|
|
| See scipy.stats for more information.
|
| Parameters
| ----------
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Notes
| -----
| The sample skewness is computed as the Fisher-Pearson coefficient
| of skewness, i.e.
|
| .. math:: g_1=\frac{m_3}{m_2^{3/2}}
|
| where
|
| .. math:: m_i=\frac{1}{N}\sum_{n=1}^N(x[n]-\bar{x})^i
|
| is the biased sample :math:`i\texttt{th}` central moment, and
| :math:`\bar{x}` is
| the sample mean. If `bias` is False, the calculations are
| corrected for bias and the value computed is the adjusted
| Fisher-Pearson standardized moment coefficient, i.e.
|
| .. math::
| G_1 = \frac{k_3}{k_2^{3/2}} = \frac{\sqrt{N(N-1)}}{N-2}\frac{m_3}{m_2^{3/2}}
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 2, 4, 5])
| >>> s.skew()
| 0.34776706224699483
|
| slice(self, offset: 'int', length: 'int | None' = None) -> 'Series'
| Get a slice of this Series.
|
| Parameters
| ----------
| offset
| Start index. Negative indexing is supported.
| length
| Length of the slice. If set to `None`, all rows starting at the offset
| will be selected.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4])
| >>> s.slice(1, 2)
| shape: (2,)
| Series: 'a' [i64]
| [
| 2
| 3
| ]
|
| sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False, multithreaded: 'bool' = True, in_place: 'bool' = False) -> 'Self'
| Sort this Series.
|
| Parameters
| ----------
| descending
| Sort in descending order.
| nulls_last
| Place null values last instead of first.
| multithreaded
| Sort using multiple threads.
| in_place
| Sort in-place.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 3, 4, 2])
| >>> s.sort()
| shape: (4,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| ]
| >>> s.sort(descending=True)
| shape: (4,)
| Series: 'a' [i64]
| [
| 4
| 3
| 2
| 1
| ]
|
| sql(self, query: 'str_', *, table_name: 'str_' = 'self') -> 'DataFrame'
| Execute a SQL query against the Series.
|
| .. versionadded:: 1.37.0
|
| .. warning::
| This functionality is considered **unstable**, although it is close to
| being considered stable. It may be changed at any point without it being
| considered a breaking change.
|
| Parameters
| ----------
| query
| SQL query to execute.
| table_name
| Optionally provide an explicit name for the table that represents the
| calling frame (defaults to "self").
|
| Notes
| -----
| * The calling Series is automatically registered as a table in the SQLContext
| under the name "self". If you want access to the DataFrames, LazyFrames, and
| other Series found in the current globals, use :meth:`pl.sql <polars.sql>`.
| * More control over registration and execution behaviour is available by
| using the :class:`SQLContext` object.
| * The SQL query executes in lazy mode before being collected and returned
| as a DataFrame.
| * It is recommended to name your Series for use with SQL, otherwise the default
| Series name (an empty string) is used; while `""` is valid, it is awkward.
|
| See Also
| --------
| SQLContext
|
| Examples
| --------
| >>> from datetime import date
| >>> s = pl.Series(
| ... name="dt",
| ... values=[date(1999, 12, 31), date(2099, 2, 14), date(2026, 3, 5)],
| ... )
|
| Query the Series using SQL:
|
| >>> s.sql('''
| ... SELECT
| ... EXTRACT('year',dt) AS y,
| ... EXTRACT('month',dt) AS m,
| ... EXTRACT('day',dt) AS d,
| ... FROM self
| ... WHERE dt > '2020-01-01'
| ... ORDER BY dt DESC
| ... ''')
| shape: (2, 3)
| ┌──────┬─────┬─────┐
| │ y ┆ m ┆ d │
| │ --- ┆ --- ┆ --- │
| │ i32 ┆ i8 ┆ i8 │
| ╞══════╪═════╪═════╡
| │ 2099 ┆ 2 ┆ 14 │
| │ 2026 ┆ 3 ┆ 5 │
| └──────┴─────┴─────┘
|
| While you can refer to an unnamed Series column using the default empty
| string, it is not recommended:
|
| >>> s = pl.Series([1, 2, 3])
| >>> s.sql('SELECT "" AS x, "" * 2 AS "2x" FROM self')
| shape: (3, 2)
| ┌─────┬─────┐
| │ x ┆ 2x │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 2 │
| │ 2 ┆ 4 │
| │ 3 ┆ 6 │
| └─────┴─────┘
|
| sqrt(self) -> 'Series'
| Compute the square root of the elements.
|
| Syntactic sugar for
|
| >>> pl.Series([1, 2]) ** 0.5
| shape: (2,)
| Series: '' [f64]
| [
| 1.0
| 1.414214
| ]
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.sqrt()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 1.414214
| 1.732051
| ]
|
| std(self, ddof: 'int' = 1) -> 'float | timedelta | None'
| Get the standard deviation of this Series.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.std()
| 1.0
|
| sum(self) -> 'int | float | PyDecimal'
| Reduce this Series to the sum value.
|
| Notes
| -----
| * Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
| * If there are no non-null values, then the output is `0`.
| If you would prefer empty sums to return `None`, you can
| use `s.sum() if s.count() else None` instead
| of `s.sum()`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.sum()
| 6
|
| tail(self, n: 'int' = 10) -> 'Series'
| Get the last `n` elements.
|
| Parameters
| ----------
| n
| Number of elements to return. If a negative value is passed, return all
| elements except the first `abs(n)`.
|
| See Also
| --------
| head, slice
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.tail(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 3
| 4
| 5
| ]
|
| Pass a negative value to get all rows `except` the first `abs(n)`.
|
| >>> s.tail(-3)
| shape: (2,)
| Series: 'a' [i64]
| [
| 4
| 5
| ]
|
| tan(self) -> 'Series'
| Compute the element-wise value for the tangent.
|
| Notes
| -----
| Input values are interpreted as radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Series.radians>`.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [0.0, math.pi / 2.0, math.pi])
| >>> s.tan()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.0
| 1.6331e16
| -1.2246e-16
| ]
|
| tanh(self) -> 'Series'
| Compute the element-wise value for the hyperbolic tangent.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.tanh()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.761594
| 0.0
| -0.761594
| ]
|
| to_arrow(self, *, compat_level: 'CompatLevel | None' = None) -> 'pa.Array'
| Return the underlying Arrow array.
|
| If the Series contains only a single chunk this operation is zero copy.
|
| .. versionchanged:: 1.24
| The `future` parameter was renamed `compat_level`.
|
| Parameters
| ----------
| compat_level
| Use a specific compatibility level
| when exporting Polars' internal data structures.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s = s.to_arrow()
| >>> s
| <pyarrow.lib.Int64Array object at ...>
| [
| 1,
| 2,
| 3
| ]
|
| to_dummies(self, *, separator: 'str_' = '_', drop_first: 'bool' = False, drop_nulls: 'bool' = False) -> 'DataFrame'
| Get dummy/indicator variables.
|
| Parameters
| ----------
| separator
| Separator/delimiter used when generating column names.
| drop_first
| Remove the first category from the variable being encoded.
| drop_nulls
| If there are `None` values in the series, a `null` column is not generated.
| Null values in the input are represented by zero vectors.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.to_dummies()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a_1 ┆ a_2 ┆ a_3 │
| │ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ u8 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 0 ┆ 0 │
| │ 0 ┆ 1 ┆ 0 │
| │ 0 ┆ 0 ┆ 1 │
| └─────┴─────┴─────┘
|
| >>> s.to_dummies(drop_first=True)
| shape: (3, 2)
| ┌─────┬─────┐
| │ a_2 ┆ a_3 │
| │ --- ┆ --- │
| │ u8 ┆ u8 │
| ╞═════╪═════╡
| │ 0 ┆ 0 │
| │ 1 ┆ 0 │
| │ 0 ┆ 1 │
| └─────┴─────┘
|
| >>> s = pl.Series("a", [1, 2, None, 3])
| >>> s.to_dummies(drop_nulls=True, drop_first=True)
| shape: (4, 2)
| ┌─────┬─────┐
| │ a_2 ┆ a_3 │
| │ --- ┆ --- │
| │ u8 ┆ u8 │
| ╞═════╪═════╡
| │ 0 ┆ 0 │
| │ 1 ┆ 0 │
| │ 0 ┆ 0 │
| │ 0 ┆ 1 │
| └─────┴─────┘
|
| to_frame(self, name: 'str_ | None' = None) -> 'DataFrame'
| Cast this Series to a DataFrame.
|
| Parameters
| ----------
| name
| optionally name/rename the Series column in the new DataFrame.
|
| Examples
| --------
| >>> s = pl.Series("a", [123, 456])
| >>> df = s.to_frame()
| >>> df
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 123 │
| │ 456 │
| └─────┘
|
| >>> df = s.to_frame("xyz")
| >>> df
| shape: (2, 1)
| ┌─────┐
| │ xyz │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 123 │
| │ 456 │
| └─────┘
|
| to_init_repr(self, n: 'int' = 1000) -> 'str_'
| Convert Series to instantiable string representation.
|
| Parameters
| ----------
| n
| Only use first n elements.
|
| See Also
| --------
| polars.Series.to_init_repr
| polars.from_repr
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, None, 4], dtype=pl.Int16)
| >>> print(s.to_init_repr())
| pl.Series('a', [1, 2, None, 4], dtype=pl.Int16)
| >>> s_from_str_repr = eval(s.to_init_repr())
| >>> s_from_str_repr
| shape: (4,)
| Series: 'a' [i16]
| [
| 1
| 2
| null
| 4
| ]
|
| to_jax(self, device: 'jax.Device | str_ | None' = None) -> 'jax.Array'
| Convert this Series to a Jax Array.
|
| .. versionadded:: 0.20.27
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| device
| Specify the jax `Device` on which the array will be created; can provide
| a string (such as "cpu", "gpu", or "tpu") in which case the device is
| retrieved as `jax.devices(string)[0]`. For more specific control you
| can supply the instantiated `Device` directly. If None, arrays are
| created on the default device.
|
| Examples
| --------
| >>> s = pl.Series("x", [10.5, 0.0, -10.0, 5.5])
| >>> s.to_jax()
| Array([ 10.5, 0. , -10. , 5.5], dtype=float32)
|
| to_list(self) -> 'list_[Any]'
| Convert this Series to a Python list.
|
| This operation copies data.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.to_list()
| [1, 2, 3]
| >>> type(s.to_list())
| <class 'list'>
|
| to_numpy(self, *, writable: 'bool' = False, allow_copy: 'bool' = True, use_pyarrow: 'bool | None' = None, zero_copy_only: 'bool | None' = None) -> 'np.ndarray[Any, Any]'
| Convert this Series to a NumPy ndarray.
|
| This operation copies data only when necessary. The conversion is zero copy when
| all of the following hold:
|
| - The data type is an integer, float, `Datetime`, `Duration`, or `Array`.
| - The Series contains no null values.
| - The Series consists of a single chunk.
| - The `writable` parameter is set to `False` (default).
|
| Parameters
| ----------
| writable
| Ensure the resulting array is writable. This will force a copy of the data
| if the array was created without copy as the underlying Arrow data is
| immutable.
| allow_copy
| Allow memory to be copied to perform the conversion. If set to `False`,
| causes conversions that are not zero-copy to fail.
|
| use_pyarrow
| First convert to PyArrow, then call `pyarrow.Array.to_numpy
| <https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy>`_
| to convert to NumPy. If set to `False`, Polars' own conversion logic is
| used.
|
| .. deprecated:: 0.20.28
| Polars now uses its native engine by default for conversion to NumPy.
| To use PyArrow's engine, call `.to_arrow().to_numpy()` instead.
|
| zero_copy_only
| Raise an exception if the conversion to a NumPy would require copying
| the underlying data. Data copy occurs, for example, when the Series contains
| nulls or non-numeric types.
|
| .. deprecated:: 0.20.10
| Use the `allow_copy` parameter instead, which is the inverse of this
| one.
|
| Examples
| --------
| Numeric data without nulls can be converted without copying data.
| The resulting array will not be writable.
|
| >>> s = pl.Series([1, 2, 3], dtype=pl.Int8)
| >>> arr = s.to_numpy()
| >>> arr
| array([1, 2, 3], dtype=int8)
| >>> arr.flags.writeable
| False
|
| Set `writable=True` to force data copy to make the array writable.
|
| >>> s.to_numpy(writable=True).flags.writeable
| True
|
| Integer Series containing nulls will be cast to a float type with `nan`
| representing a null value. This requires data to be copied.
|
| >>> s = pl.Series([1, 2, None], dtype=pl.UInt16)
| >>> s.to_numpy()
| array([ 1., 2., nan], dtype=float32)
|
| Set `allow_copy=False` to raise an error if data would be copied.
|
| >>> s.to_numpy(allow_copy=False) # doctest: +SKIP
| Traceback (most recent call last):
| ...
| RuntimeError: copy not allowed: cannot convert to a NumPy array without copying data
|
| Series of data type `Array` and `Struct` will result in an array with more than
| one dimension.
|
| >>> s = pl.Series([[1, 2, 3], [4, 5, 6]], dtype=pl.Array(pl.Int64, 3))
| >>> s.to_numpy()
| array([[1, 2, 3],
| [4, 5, 6]])
|
| to_pandas(self, *, use_pyarrow_extension_array: 'bool' = False, **kwargs: 'Any') -> 'pd.Series[Any]'
| Convert this Series to a pandas Series.
|
| This operation copies data if `use_pyarrow_extension_array` is not enabled.
|
| Parameters
| ----------
| use_pyarrow_extension_array
| Use a PyArrow-backed extension array instead of a NumPy array for the pandas
| Series. This allows zero copy operations and preservation of null values.
| Subsequent operations on the resulting pandas Series may trigger conversion
| to NumPy if those operations are not supported by PyArrow compute functions.
| **kwargs
| Additional keyword arguments to be passed to
| :meth:`pyarrow.Array.to_pandas`.
|
| Returns
| -------
| :class:`pandas.Series`
|
| Notes
| -----
| This operation requires that both :mod:`pandas` and :mod:`pyarrow` are
| installed.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.to_pandas()
| 0 1
| 1 2
| 2 3
| Name: a, dtype: int64
|
| Null values are converted to `NaN`.
|
| >>> s = pl.Series("b", [1, 2, None])
| >>> s.to_pandas()
| 0 1.0
| 1 2.0
| 2 NaN
| Name: b, dtype: float64
|
| Pass `use_pyarrow_extension_array=True` to get a pandas Series backed by a
| PyArrow extension array. This will preserve null values.
|
| >>> s.to_pandas(use_pyarrow_extension_array=True)
| 0 1
| 1 2
| 2 <NA>
| Name: b, dtype: int64[pyarrow]
|
| to_physical(self) -> 'Series'
| Cast to physical representation of the logical dtype.
|
| - :func:`polars.datatypes.Date` -> :func:`polars.datatypes.Int32`
| - :func:`polars.datatypes.Datetime` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Time` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Duration` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Categorical` -> :func:`polars.datatypes.UInt32`
| - `List(inner)` -> `List(physical of inner)`
| - `Array(inner)` -> `Array(physical of inner)`
| - `Struct(fields)` -> `Struct(physical of fields)`
| - Other data types will be left unchanged.
|
| Warnings
| --------
| The physical representations are an implementation detail
| and not guaranteed to be stable.
|
| Examples
| --------
| Replicating the pandas
| `pd.Series.factorize
| <https://pandas.pydata.org/docs/reference/api/pandas.Series.factorize.html>`_
| method.
|
| >>> s = pl.Series("values", ["a", None, "x", "a"])
| >>> s.cast(pl.Categorical).to_physical()
| shape: (4,)
| Series: 'values' [u32]
| [
| 0
| null
| 1
| 0
| ]
|
| to_torch(self) -> 'torch.Tensor'
| Convert this Series to a PyTorch Tensor.
|
| .. versionadded:: 0.20.23
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Notes
| -----
| PyTorch tensors do not support UInt16, UInt32, or UInt64; these dtypes
| will be automatically cast to Int32, Int64, and Int64, respectively.
|
| Examples
| --------
| >>> s = pl.Series("x", [1, 0, 1, 2, 0], dtype=pl.UInt8)
| >>> s.to_torch()
| tensor([1, 0, 1, 2, 0], dtype=torch.uint8)
| >>> s = pl.Series("x", [5.5, -10.0, 2.5], dtype=pl.Float32)
| >>> s.to_torch()
| tensor([ 5.5000, -10.0000, 2.5000])
|
| top_k(self, k: 'int' = 5) -> 'Series'
| Return the `k` largest elements.
|
| Non-null elements are always preferred over null elements. The output is
| not guaranteed to be in any particular order, call :func:`sort` after
| this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k_by
| bottom_k
| bottom_k_by
|
| Examples
| --------
| >>> s = pl.Series("a", [2, 5, 1, 4, 3])
| >>> s.top_k(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 5
| 4
| 3
| ]
|
| top_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Series'
| Return the `k` largest elements of the `by` column.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| Parameters
| ----------
| by
| Column used to determine the largest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` smallest elements of the `by` column (instead of the `k`
| largest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| bottom_k
| bottom_k_by
|
| Examples
| --------
| >>> s = pl.Series("a", [2, 5, 1, 4, 3])
| >>> s.top_k_by("a", 3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 5
| 4
| 3
| ]
|
| truncate(self, decimals: 'int' = 0) -> 'Series'
| Truncate numeric data toward zero to `decimals` number of decimal places.
|
| Parameters
| ----------
| decimals
| Number of decimal places to truncate to.
|
| Notes
| -----
| * Truncation discards the fractional part beyond the given number of decimals.
| For example, when rounding to 0 decimals 0.25, -0.25, 0.99, and -0.99 will
| all round to 0. When rounding to 1 decimal 1.9999 rounds to 1.9 and -1.9999
| rounds to -1.9. There is no tiebreak behaviour at midpoint values as there
| is with :meth:`round` so 0.5 and -0.5 will also round to 0 when decimals=1.
|
| * This method performs numeric truncation. For truncating temporal
| data (dates/datetimes), use :func:`Series.dt.truncate` instead.
|
| See Also
| --------
| round : Round to a given number of decimals.
| floor : Round down to the nearest integer.
| ceil : Round up to the nearest integer.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.12345, 2.56789, 3.991234])
| >>> s.truncate(2)
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.12
| 2.56
| 3.99
| ]
|
| >>> s = pl.Series("a", [-1.78, 2.56, -3.99])
| >>> s.truncate(0)
| shape: (3,)
| Series: 'a' [f64]
| [
| -1.0
| 2.0
| -3.0
| ]
|
| unique(self, *, maintain_order: 'bool' = False) -> 'Series'
| Get unique elements in series.
|
| `null` is considered to be a unique value for the purposes of this operation.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.unique().sort()
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| unique_counts(self) -> 'Series'
| Return a count of the unique values in the order of appearance.
|
| Examples
| --------
| >>> s = pl.Series("id", ["a", "b", "b", "c", "c", "c"])
| >>> s.unique_counts()
| shape: (3,)
| Series: 'id' [u32]
| [
| 1
| 2
| 3
| ]
|
| upper_bound(self) -> 'Self'
| Return the upper bound of this Series' dtype as a unit Series.
|
| See Also
| --------
| lower_bound : return the lower bound of the given Series' dtype.
|
| Examples
| --------
| >>> s = pl.Series("s", [-1, 0, 1], dtype=pl.Int8)
| >>> s.upper_bound()
| shape: (1,)
| Series: 's' [i8]
| [
| 127
| ]
|
| >>> s = pl.Series("s", [1.0, 2.5, 3.0], dtype=pl.Float64)
| >>> s.upper_bound()
| shape: (1,)
| Series: 's' [f64]
| [
| inf
| ]
|
| value_counts(self, *, sort: 'bool' = False, parallel: 'bool' = False, name: 'str_ | None' = None, normalize: 'bool' = False) -> 'DataFrame'
| Count the occurrences of unique values.
|
| Parameters
| ----------
| sort
| Sort the output by count, in descending order.
| If set to `False` (default), the order is non-deterministic.
| parallel
| Execute the computation in parallel.
|
| .. note::
| This option should likely *not* be enabled in a `group_by` context,
| as the computation will already be parallelized per group.
| name
| Give the resulting count column a specific name; if `normalize` is
| True this defaults to "proportion", otherwise defaults to "count".
| normalize
| If True, the count is returned as the relative frequency of unique
| values normalized to 1.0.
|
| Returns
| -------
| DataFrame
| Columns map the unique values to their count (or proportion).
|
| Examples
| --------
| >>> s = pl.Series("color", ["red", "blue", "red", "green", "blue", "blue"])
| >>> s.value_counts() # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌───────┬───────┐
| │ color ┆ count │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═══════╡
| │ red ┆ 2 │
| │ green ┆ 1 │
| │ blue ┆ 3 │
| └───────┴───────┘
|
| Sort the output by count and customize the count column name.
|
| >>> s.value_counts(sort=True, name="n")
| shape: (3, 2)
| ┌───────┬─────┐
| │ color ┆ n │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═════╡
| │ blue ┆ 3 │
| │ red ┆ 2 │
| │ green ┆ 1 │
| └───────┴─────┘
|
| Return the count as a relative frequency, normalized to 1.0:
|
| >>> s.value_counts(sort=True, normalize=True, name="fraction")
| shape: (3, 2)
| ┌───────┬──────────┐
| │ color ┆ fraction │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪══════════╡
| │ blue ┆ 0.5 │
| │ red ┆ 0.333333 │
| │ green ┆ 0.166667 │
| └───────┴──────────┘
|
| var(self, ddof: 'int' = 1) -> 'float | timedelta | None'
| Get variance of this Series.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.var()
| 1.0
|
| zip_with(self, mask: 'Series', other: 'Series') -> 'Self'
| Take values from self or other based on the given mask.
|
| Where mask evaluates true, take values from self. Where mask evaluates false,
| take values from other.
|
| Parameters
| ----------
| mask
| Boolean Series.
| other
| Series of same type.
|
| Returns
| -------
| Series
|
| Examples
| --------
| >>> s1 = pl.Series([1, 2, 3, 4, 5])
| >>> s2 = pl.Series([5, 4, 3, 2, 1])
| >>> s1.zip_with(s1 < s2, s2)
| shape: (5,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| 2
| 1
| ]
| >>> mask = pl.Series([True, False, True, False, True])
| >>> s1.zip_with(mask, s2)
| shape: (5,)
| Series: '' [i64]
| [
| 1
| 4
| 3
| 2
| 5
| ]
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| arr
| Create an object namespace of all array related methods.
|
| bin
| Create an object namespace of all binary related methods.
|
| cat
| Create an object namespace of all categorical related methods.
|
| dt
| Create an object namespace of all datetime related methods.
|
| dtype
| Get the data type of this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.dtype
| Int64
|
| ext
| Create an object namespace of all extension type related methods.
|
| flags
| Get flags that are set on the Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.flags
| {'SORTED_ASC': False, 'SORTED_DESC': False}
|
| list
| Create an object namespace of all list related methods.
|
| name
| Get the name of this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.name
| 'a'
|
| plot
| Create a plot namespace.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.6.0
| In prior versions of Polars, HvPlot was the plotting backend. If you would
| like to restore the previous plotting functionality, all you need to do
| is add `import hvplot.polars` at the top of your script and replace
| `df.plot` with `df.hvplot`.
|
| Polars does not implement plotting logic itself, but instead defers to
| Altair:
|
| - `s.plot.hist(**kwargs)`
| is shorthand for
| `alt.Chart(s.to_frame()).mark_bar(tooltip=True).encode(x=alt.X(f'{s.name}:Q', bin=True), y='count()', **kwargs).interactive()`
| - `s.plot.kde(**kwargs)`
| is shorthand for
| `alt.Chart(s.to_frame()).transform_density(s.name, as_=[s.name, 'density']).mark_area(tooltip=True).encode(x=s.name, y='density:Q', **kwargs).interactive()`
| - for any other attribute `attr`, `s.plot.attr(**kwargs)`
| is shorthand for
| `alt.Chart(s.to_frame().with_row_index()).mark_attr(tooltip=True).encode(x='index', y=s.name, **kwargs).interactive()`
|
| For configuration, we suggest reading
| `Chart Configuration <https://altair-viz.github.io/altair-tutorial/notebooks/08-Configuration.html>`_.
| For example, you can:
|
| - Change the width/height/title with ``.properties(width=500, height=350, title="My amazing plot")``.
| - Change the x-axis label rotation with ``.configure_axisX(labelAngle=30)``.
| - Change the opacity of the points in your scatter plot with ``.configure_point(opacity=.5)``.
|
| Examples
| --------
| Histogram:
|
| >>> s = pl.Series([1, 4, 4, 6, 2, 4, 3, 5, 5, 7, 1])
| >>> s.plot.hist() # doctest: +SKIP
|
| KDE plot:
|
| >>> s.plot.kde() # doctest: +SKIP
|
| Line plot:
|
| >>> s.plot.line() # doctest: +SKIP
|
| shape
| Shape of this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.shape
| (3,)
|
| str
| Create an object namespace of all string related methods.
|
| struct
| Create an object namespace of all struct related methods.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set_[str_]]', '_s': 'PySeri...
|
| __hash__ = None
polars.String
Python Library Documentation: class String in module polars.datatypes.classes
class String(DataType)
| UTF-8 encoded string type.
|
| Method resolution order:
| String
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.StringCache
Python Library Documentation: class StringCache in module polars.string_cache
class StringCache(contextlib.ContextDecorator)
| StringCache(*args, **kwargs)
|
| Does nothing.
|
| .. deprecated:: 1.41.0
| The string cache was used to maintain the mapping for the Categorical
| dtype, this is now done through ``pl.Categories``.
|
| Method resolution order:
| StringCache
| contextlib.ContextDecorator
| builtins.object
|
| Methods defined here:
|
| __enter__(self) -> 'Self'
|
| __exit__(self, exc_type: 'type[BaseException] | None', exc_val: 'BaseException | None', exc_tb: 'TracebackType | None') -> 'None'
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __init_subclass__(*args, **kwargs) from typing_extensions.deprecated.__call__.<locals>
| This method is called when a class is subclassed.
|
| The default implementation does nothing. It may be
| overridden to extend subclasses.
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs) from object
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| __deprecated__ = 'the string cache has been replaced by pl.Categories'
|
| ----------------------------------------------------------------------
| Methods inherited from contextlib.ContextDecorator:
|
| __call__(self, func)
| Call self as a function.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from contextlib.ContextDecorator:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Struct
Python Library Documentation: class Struct in module polars.datatypes.classes
class Struct(NestedType)
| Struct(fields: 'Sequence[Field] | SchemaDict') -> 'None'
|
| Struct composite type.
|
| Parameters
| ----------
| fields
| The fields that make up the struct. Can be either a sequence of Field
| objects or a mapping of column names to data types.
|
| Examples
| --------
| Initialize using a dictionary:
|
| >>> dtype = pl.Struct({"a": pl.Int8, "b": pl.List(pl.String)})
| >>> dtype
| Struct({'a': Int8, 'b': List(String)})
|
| Initialize using a list of Field objects:
|
| >>> dtype = pl.Struct([pl.Field("a", pl.Int8), pl.Field("b", pl.List(pl.String))])
| >>> dtype
| Struct({'a': Int8, 'b': List(String)})
|
| When initializing a Series, Polars can infer a struct data type from the data.
|
| >>> s = pl.Series([{"a": 1, "b": ["x", "y"]}, {"a": 2, "b": ["z"]}])
| >>> s
| shape: (2,)
| Series: '' [struct[2]]
| [
| {1,["x", "y"]}
| {2,["z"]}
| ]
| >>> s.dtype
| Struct({'a': Int64, 'b': List(String)})
|
| Method resolution order:
| Struct
| NestedType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, fields: 'Sequence[Field] | SchemaDict') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> 'Iterator[tuple[str, PolarsDataType]]'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __reversed__(self) -> 'Iterator[tuple[str, PolarsDataType]]'
|
| to_schema(self) -> 'OrderedDict[str, PolarsDataType]'
| Return Struct dtype as a schema dict.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'fields': 'list[Field]'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.TYPE_CHECKING
Python Library Documentation: bool object class bool(int) | bool(x) -> bool | | Returns True when the argument x is true, False otherwise. | The builtins True and False are the only two instances of the class bool. | The class bool is a subclass of the class int, and cannot be subclassed. | | Method resolution order: | bool | int | object | | Methods defined here: | | __and__(self, value, /) | Return self&value. | | __invert__(self, /) | ~self | | __or__(self, value, /) | Return self|value. | | __rand__(self, value, /) | Return value&self. | | __repr__(self, /) | Return repr(self). | | __ror__(self, value, /) | Return value|self. | | __rxor__(self, value, /) | Return value^self. | | __xor__(self, value, /) | Return self^value. | | ---------------------------------------------------------------------- | Static methods defined here: | | __new__(*args, **kwargs) | Create and return a new object. See help(type) for accurate signature. | | ---------------------------------------------------------------------- | Methods inherited from int: | | __abs__(self, /) | abs(self) | | __add__(self, value, /) | Return self+value. | | __bool__(self, /) | True if self else False | | __ceil__(...) | Ceiling of an Integral returns itself. | | __divmod__(self, value, /) | Return divmod(self, value). | | __eq__(self, value, /) | Return self==value. | | __float__(self, /) | float(self) | | __floor__(...) | Flooring an Integral returns itself. | | __floordiv__(self, value, /) | Return self//value. | | __format__(self, format_spec, /) | Convert to a string according to format_spec. | | __ge__(self, value, /) | Return self>=value. | | __getattribute__(self, name, /) | Return getattr(self, name). | | __getnewargs__(self, /) | | __gt__(self, value, /) | Return self>value. | | __hash__(self, /) | Return hash(self). | | __index__(self, /) | Return self converted to an integer, if self is suitable for use as an index into a list. | | __int__(self, /) | int(self) | | __le__(self, value, /) | Return self<=value. | | __lshift__(self, value, /) | Return self<<value. | | __lt__(self, value, /) | Return self<value. | | __mod__(self, value, /) | Return self%value. | | __mul__(self, value, /) | Return self*value. | | __ne__(self, value, /) | Return self!=value. | | __neg__(self, /) | -self | | __pos__(self, /) | +self | | __pow__(self, value, mod=None, /) | Return pow(self, value, mod). | | __radd__(self, value, /) | Return value+self. | | __rdivmod__(self, value, /) | Return divmod(value, self). | | __rfloordiv__(self, value, /) | Return value//self. | | __rlshift__(self, value, /) | Return value<<self. | | __rmod__(self, value, /) | Return value%self. | | __rmul__(self, value, /) | Return value*self. | | __round__(...) | Rounding an Integral returns itself. | | Rounding with an ndigits argument also returns an integer. | | __rpow__(self, value, mod=None, /) | Return pow(value, self, mod). | | __rrshift__(self, value, /) | Return value>>self. | | __rshift__(self, value, /) | Return self>>value. | | __rsub__(self, value, /) | Return value-self. | | __rtruediv__(self, value, /) | Return value/self. | | __sizeof__(self, /) | Returns size in memory, in bytes. | | __sub__(self, value, /) | Return self-value. | | __truediv__(self, value, /) | Return self/value. | | __trunc__(...) | Truncating an Integral returns itself. | | as_integer_ratio(self, /) | Return a pair of integers, whose ratio is equal to the original int. | | The ratio is in lowest terms and has a positive denominator. | | >>> (10).as_integer_ratio() | (10, 1) | >>> (-10).as_integer_ratio() | (-10, 1) | >>> (0).as_integer_ratio() | (0, 1) | | bit_count(self, /) | Number of ones in the binary representation of the absolute value of self. | | Also known as the population count. | | >>> bin(13) | '0b1101' | >>> (13).bit_count() | 3 | | bit_length(self, /) | Number of bits necessary to represent self in binary. | | >>> bin(37) | '0b100101' | >>> (37).bit_length() | 6 | | conjugate(...) | Returns self, the complex conjugate of any int. | | is_integer(self, /) | Returns True. Exists for duck type compatibility with float.is_integer. | | to_bytes(self, /, length=1, byteorder='big', *, signed=False) | Return an array of bytes representing an integer. | | length | Length of bytes object to use. An OverflowError is raised if the | integer is not representable with the given number of bytes. Default | is length 1. | byteorder | The byte order used to represent the integer. If byteorder is 'big', | the most significant byte is at the beginning of the byte array. If | byteorder is 'little', the most significant byte is at the end of the | byte array. To request the native byte order of the host system, use | `sys.byteorder' as the byte order value. Default is to use 'big'. | signed | Determines whether two's complement is used to represent the integer. | If signed is False and a negative integer is given, an OverflowError | is raised. | | ---------------------------------------------------------------------- | Class methods inherited from int: | | from_bytes(bytes, byteorder='big', *, signed=False) | Return the integer represented by the given array of bytes. | | bytes | Holds the array of bytes to convert. The argument must either | support the buffer protocol or be an iterable object producing bytes. | Bytes and bytearray are examples of built-in objects that support the | buffer protocol. | byteorder | The byte order used to represent the integer. If byteorder is 'big', | the most significant byte is at the beginning of the byte array. If | byteorder is 'little', the most significant byte is at the end of the | byte array. To request the native byte order of the host system, use | `sys.byteorder' as the byte order value. Default is to use 'big'. | signed | Indicates whether two's complement is used to represent the integer. | | ---------------------------------------------------------------------- | Data descriptors inherited from int: | | denominator | the denominator of a rational number in lowest terms | | imag | the imaginary part of a complex number | | numerator | the numerator of a rational number in lowest terms | | real | the real part of a complex number
polars.Time
Python Library Documentation: class Time in module polars.datatypes.classes
class Time(TemporalType)
| Data type representing the time of day.
|
| Notes
| -----
| The underlying representation of this type is a 64-bit signed integer.
| The integer indicates the number of nanoseconds since midnight.
|
| Method resolution order:
| Time
| TemporalType
| DataType
| builtins.object
|
| Class methods defined here:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Time.max() == 86_399_999_999_999)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Time.min() == 0)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.UInt128
Python Library Documentation: class UInt128 in module polars.datatypes.classes
class UInt128(UnsignedIntegerType)
| 128-bit unsigned integer type.
|
| .. warning::
| This functionality is considered **unstable**.
| It is a work-in-progress feature and may not always work as expected.
| It may be changed at any point without it being considered a breaking change.
|
| Method resolution order:
| UInt128
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.UInt16
Python Library Documentation: class UInt16 in module polars.datatypes.classes
class UInt16(UnsignedIntegerType)
| 16-bit unsigned integer type.
|
| Method resolution order:
| UInt16
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.UInt32
Python Library Documentation: class UInt32 in module polars.datatypes.classes
class UInt32(UnsignedIntegerType)
| 32-bit unsigned integer type.
|
| Method resolution order:
| UInt32
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.UInt64
Python Library Documentation: class UInt64 in module polars.datatypes.classes
class UInt64(UnsignedIntegerType)
| 64-bit unsigned integer type.
|
| Method resolution order:
| UInt64
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.UInt8
Python Library Documentation: class UInt8 in module polars.datatypes.classes
class UInt8(UnsignedIntegerType)
| 8-bit unsigned integer type.
|
| Method resolution order:
| UInt8
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Unknown
Python Library Documentation: class Unknown in module polars.datatypes.classes
class Unknown(DataType)
| Type representing DataType values that could not be determined statically.
|
| Method resolution order:
| Unknown
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.Utf8
Python Library Documentation: class String in module polars.datatypes.classes
class String(DataType)
| UTF-8 encoded string type.
|
| Method resolution order:
| String
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
polars.align_frames
Python Library Documentation: function align_frames in module polars.functions.eager
align_frames(*frames: 'FrameType | Iterable[FrameType]', on: 'str | Expr | Sequence[str] | Sequence[Expr] | Sequence[str | Expr]', how: 'JoinStrategy' = 'full', select: 'str | Expr | Sequence[str | Expr] | None' = None, descending: 'bool | Sequence[bool]' = False) -> 'list[FrameType]'
Align a sequence of frames using common values from one or more columns as a key.
Frames that do not contain the given key values have rows injected (with nulls
filling the non-key columns), and each resulting frame is sorted by the key.
The original column order of input frames is not changed unless `select` is
specified (in which case the final column order is determined from that). In the
case where duplicate key values exist, the alignment behaviour is determined by
the given alignment strategy specified in the `how` parameter (by default this
is a full outer join, but if your data is suitable you can get a large speedup
by setting `how="left"` instead).
Note that this function does not result in a joined frame - you receive the same
number of frames back that you passed in, but each is now aligned by key and has
the same number of rows.
Parameters
----------
frames
Sequence of DataFrames or LazyFrames.
on
One or more columns whose unique values will be used to align the frames.
select
Optional post-alignment column select to constrain and/or order
the columns returned from the newly aligned frames.
descending
Sort the alignment column values in descending order; can be a single
boolean or a list of booleans associated with each column in `on`.
how
By default the row alignment values are determined using a full outer join
strategy across all frames; if you know that the first frame contains all
required keys, you can set `how="left"` for a large performance increase.
Examples
--------
>>> from datetime import date
>>> df1 = pl.DataFrame(
... {
... "dt": [date(2022, 9, 1), date(2022, 9, 2), date(2022, 9, 3)],
... "x": [3.5, 4.0, 1.0],
... "y": [10.0, 2.5, 1.5],
... }
... )
>>> df2 = pl.DataFrame(
... {
... "dt": [date(2022, 9, 2), date(2022, 9, 3), date(2022, 9, 1)],
... "x": [8.0, 1.0, 3.5],
... "y": [1.5, 12.0, 5.0],
... }
... )
>>> df3 = pl.DataFrame(
... {
... "dt": [date(2022, 9, 3), date(2022, 9, 2)],
... "x": [2.0, 5.0],
... "y": [2.5, 2.0],
... }
... ) # doctest: +IGNORE_RESULT
>>> pl.Config.set_tbl_formatting("UTF8_FULL") # doctest: +IGNORE_RESULT
#
# df1 df2 df3
# shape: (3, 3) shape: (3, 3) shape: (2, 3)
# ┌────────────┬─────┬──────┐ ┌────────────┬─────┬──────┐ ┌────────────┬─────┬─────┐
# │ dt ┆ x ┆ y │ │ dt ┆ x ┆ y │ │ dt ┆ x ┆ y │
# │ --- ┆ --- ┆ --- │ │ --- ┆ --- ┆ --- │ │ --- ┆ --- ┆ --- │
# │ date ┆ f64 ┆ f64 │ │ date ┆ f64 ┆ f64 │ │ date ┆ f64 ┆ f64 │
# ╞════════════╪═════╪══════╡ ╞════════════╪═════╪══════╡ ╞════════════╪═════╪═════╡
# │ 2022-09-01 ┆ 3.5 ┆ 10.0 │\ ,->│ 2022-09-02 ┆ 8.0 ┆ 1.5 │\ ,->│ 2022-09-03 ┆ 2.0 ┆ 2.5 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ \/ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ \/ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┤
# │ 2022-09-02 ┆ 4.0 ┆ 2.5 │_/\,->│ 2022-09-03 ┆ 1.0 ┆ 12.0 │_/`-->│ 2022-09-02 ┆ 5.0 ┆ 2.0 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ /\ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ └────────────┴─────┴─────┘
# │ 2022-09-03 ┆ 1.0 ┆ 1.5 │_/ `>│ 2022-09-01 ┆ 3.5 ┆ 5.0 │-//-
# └────────────┴─────┴──────┘ └────────────┴─────┴──────┘
...
Align frames by the "dt" column:
>>> af1, af2, af3 = pl.align_frames(
... df1, df2, df3, on="dt"
... ) # doctest: +IGNORE_RESULT
#
# df1 df2 df3
# shape: (3, 3) shape: (3, 3) shape: (3, 3)
# ┌────────────┬─────┬──────┐ ┌────────────┬─────┬──────┐ ┌────────────┬──────┬──────┐
# │ dt ┆ x ┆ y │ │ dt ┆ x ┆ y │ │ dt ┆ x ┆ y │
# │ --- ┆ --- ┆ --- │ │ --- ┆ --- ┆ --- │ │ --- ┆ --- ┆ --- │
# │ date ┆ f64 ┆ f64 │ │ date ┆ f64 ┆ f64 │ │ date ┆ f64 ┆ f64 │
# ╞════════════╪═════╪══════╡ ╞════════════╪═════╪══════╡ ╞════════════╪══════╪══════╡
# │ 2022-09-01 ┆ 3.5 ┆ 10.0 │----->│ 2022-09-01 ┆ 3.5 ┆ 5.0 │----->│ 2022-09-01 ┆ null ┆ null │
# ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┤
# │ 2022-09-02 ┆ 4.0 ┆ 2.5 │----->│ 2022-09-02 ┆ 8.0 ┆ 1.5 │----->│ 2022-09-02 ┆ 5.0 ┆ 2.0 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┤
# │ 2022-09-03 ┆ 1.0 ┆ 1.5 │----->│ 2022-09-03 ┆ 1.0 ┆ 12.0 │----->│ 2022-09-03 ┆ 2.0 ┆ 2.5 │
# └────────────┴─────┴──────┘ └────────────┴─────┴──────┘ └────────────┴──────┴──────┘
...
Align frames by "dt" using "left" alignment, but keep only cols "x" and "y":
>>> af1, af2, af3 = pl.align_frames(
... df1, df2, df3, on="dt", select=["x", "y"], how="left"
... ) # doctest: +IGNORE_RESULT
#
# af1 af2 af3
# shape: (3, 3) shape: (3, 3) shape: (3, 3)
# ┌─────┬──────┐ ┌─────┬──────┐ ┌──────┬──────┐
# │ x ┆ y │ │ x ┆ y │ │ x ┆ y │
# │ --- ┆ --- │ │ --- ┆ --- │ │ --- ┆ --- │
# │ f64 ┆ f64 │ │ f64 ┆ f64 │ │ f64 ┆ f64 │
# ╞═════╪══════╡ ╞═════╪══════╡ ╞══════╪══════╡
# │ 3.5 ┆ 10.0 │ │ 3.5 ┆ 5.0 │ │ null ┆ null │
# ├╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌┼╌╌╌╌╌╌┤
# │ 4.0 ┆ 2.5 │ │ 8.0 ┆ 1.5 │ │ 5.0 ┆ 2.0 │
# ├╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌┼╌╌╌╌╌╌┤
# │ 1.0 ┆ 1.5 │ │ 1.0 ┆ 12.0 │ │ 2.0 ┆ 2.5 │
# └─────┴──────┘ └─────┴──────┘ └──────┴──────┘
...
Now data is aligned, and you can easily calculate the row-wise dot product:
>>> (af1 * af2 * af3).fill_null(0).select(pl.sum_horizontal("*").alias("dot"))
shape: (3, 1)
┌───────┐
│ dot │
│ --- │
│ f64 │
╞═══════╡
│ 0.0 │
├╌╌╌╌╌╌╌┤
│ 167.5 │
├╌╌╌╌╌╌╌┤
│ 47.0 │
└───────┘
polars.all
Python Library Documentation: function all in module polars.functions.aggregation.vertical
all(*names: 'str', ignore_nulls: 'bool' = True) -> 'Expr'
Either return an expression representing all columns, or evaluate a bitwise AND operation.
If no arguments are passed, this function is syntactic sugar for `col("*")`.
Otherwise, this function is syntactic sugar for `col(names).all()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
ignore_nulls
* If set to `True` (default), null values are ignored. If there
are no non-null values, the output is `True`.
* If set to `False`, `Kleene logic`_ is used to deal with nulls:
if the column contains any null values and no `False` values,
the output is null.
.. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
See Also
--------
all_horizontal
Examples
--------
Selecting all columns.
>>> df = pl.DataFrame(
... {
... "a": [True, False, True],
... "b": [False, False, False],
... }
... )
>>> df.select(pl.all().sum())
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ u32 ┆ u32 │
╞═════╪═════╡
│ 2 ┆ 0 │
└─────┴─────┘
Evaluate bitwise AND for a column.
>>> df.select(pl.all("a"))
shape: (1, 1)
┌───────┐
│ a │
│ --- │
│ bool │
╞═══════╡
│ false │
└───────┘
polars.all_horizontal
Python Library Documentation: function all_horizontal in module polars.functions.aggregation.horizontal
all_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Compute the logical AND horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Notes
-----
`Kleene logic`_ is used to deal with nulls: if the column contains any null values
and no `False` values, the output is null.
.. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [False, False, True, True, False, None],
... "b": [False, True, True, None, None, None],
... "c": ["u", "v", "w", "x", "y", "z"],
... }
... )
>>> df.with_columns(all=pl.all_horizontal("a", "b"))
shape: (6, 4)
┌───────┬───────┬─────┬───────┐
│ a ┆ b ┆ c ┆ all │
│ --- ┆ --- ┆ --- ┆ --- │
│ bool ┆ bool ┆ str ┆ bool │
╞═══════╪═══════╪═════╪═══════╡
│ false ┆ false ┆ u ┆ false │
│ false ┆ true ┆ v ┆ false │
│ true ┆ true ┆ w ┆ true │
│ true ┆ null ┆ x ┆ null │
│ false ┆ null ┆ y ┆ false │
│ null ┆ null ┆ z ┆ null │
└───────┴───────┴─────┴───────┘
polars.any
Python Library Documentation: function any in module polars.functions.aggregation.vertical
any(*names: 'str', ignore_nulls: 'bool' = True) -> 'Expr | bool | None'
Evaluate a bitwise OR operation.
Syntactic sugar for `col(names).any()`.
See Also
--------
any_horizontal
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
ignore_nulls
* If set to `True` (default), null values are ignored. If there
are no non-null values, the output is `False`.
* If set to `False`, `Kleene logic`_ is used to deal with nulls:
if the column contains any null values and no `True` values,
the output is null.
.. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [True, False, True],
... "b": [False, False, False],
... }
... )
>>> df.select(pl.any("a"))
shape: (1, 1)
┌──────┐
│ a │
│ --- │
│ bool │
╞══════╡
│ true │
└──────┘
polars.any_horizontal
Python Library Documentation: function any_horizontal in module polars.functions.aggregation.horizontal
any_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Compute the logical OR horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Notes
-----
`Kleene logic`_ is used to deal with nulls: if the column contains any null values
and no `True` values, the output is null.
.. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [False, False, True, True, False, None],
... "b": [False, True, True, None, None, None],
... "c": ["u", "v", "w", "x", "y", "z"],
... }
... )
>>> df.with_columns(any=pl.any_horizontal("a", "b"))
shape: (6, 4)
┌───────┬───────┬─────┬───────┐
│ a ┆ b ┆ c ┆ any │
│ --- ┆ --- ┆ --- ┆ --- │
│ bool ┆ bool ┆ str ┆ bool │
╞═══════╪═══════╪═════╪═══════╡
│ false ┆ false ┆ u ┆ false │
│ false ┆ true ┆ v ┆ true │
│ true ┆ true ┆ w ┆ true │
│ true ┆ null ┆ x ┆ true │
│ false ┆ null ┆ y ┆ null │
│ null ┆ null ┆ z ┆ null │
└───────┴───────┴─────┴───────┘
polars.api
Python Library Documentation: module polars.api in polars
NAME
polars.api
FUNCTIONS
register_dataframe_namespace(name: 'str') -> 'Callable[[type[NS]], type[NS]]'
Decorator for registering custom functionality with a Polars DataFrame.
Parameters
----------
name
Name under which the functionality will be accessed.
See Also
--------
register_expr_namespace : Register functionality on an Expr.
register_lazyframe_namespace : Register functionality on a LazyFrame.
register_series_namespace : Register functionality on a Series.
Examples
--------
>>> @pl.api.register_dataframe_namespace("split")
... class SplitFrame:
... def __init__(self, df: pl.DataFrame) -> None:
... self._df = df
...
... def by_first_letter_of_column_names(self) -> list[pl.DataFrame]:
... return [
... self._df.select([col for col in self._df.columns if col[0] == f])
... for f in dict.fromkeys(col[0] for col in self._df.columns)
... ]
...
... def by_first_letter_of_column_values(self, col: str) -> list[pl.DataFrame]:
... return [
... self._df.filter(pl.col(col).str.starts_with(c))
... for c in sorted(
... set(df.select(pl.col(col).str.slice(0, 1)).to_series())
... )
... ]
>>>
>>> df = pl.DataFrame(
... data=[["xx", 2, 3, 4], ["xy", 4, 5, 6], ["yy", 5, 6, 7], ["yz", 6, 7, 8]],
... schema=["a1", "a2", "b1", "b2"],
... orient="row",
... )
>>> df
shape: (4, 4)
┌─────┬─────┬─────┬─────┐
│ a1 ┆ a2 ┆ b1 ┆ b2 │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
│ xx ┆ 2 ┆ 3 ┆ 4 │
│ xy ┆ 4 ┆ 5 ┆ 6 │
│ yy ┆ 5 ┆ 6 ┆ 7 │
│ yz ┆ 6 ┆ 7 ┆ 8 │
└─────┴─────┴─────┴─────┘
>>> df.split.by_first_letter_of_column_names()
[shape: (4, 2)
┌─────┬─────┐
│ a1 ┆ a2 │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═════╡
│ xx ┆ 2 │
│ xy ┆ 4 │
│ yy ┆ 5 │
│ yz ┆ 6 │
└─────┴─────┘,
shape: (4, 2)
┌─────┬─────┐
│ b1 ┆ b2 │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 3 ┆ 4 │
│ 5 ┆ 6 │
│ 6 ┆ 7 │
│ 7 ┆ 8 │
└─────┴─────┘]
>>> df.split.by_first_letter_of_column_values("a1")
[shape: (2, 4)
┌─────┬─────┬─────┬─────┐
│ a1 ┆ a2 ┆ b1 ┆ b2 │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
│ xx ┆ 2 ┆ 3 ┆ 4 │
│ xy ┆ 4 ┆ 5 ┆ 6 │
└─────┴─────┴─────┴─────┘, shape: (2, 4)
┌─────┬─────┬─────┬─────┐
│ a1 ┆ a2 ┆ b1 ┆ b2 │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
│ yy ┆ 5 ┆ 6 ┆ 7 │
│ yz ┆ 6 ┆ 7 ┆ 8 │
└─────┴─────┴─────┴─────┘]
register_expr_namespace(name: 'str') -> 'Callable[[type[NS]], type[NS]]'
Decorator for registering custom functionality with a Polars Expr.
Parameters
----------
name
Name under which the functionality will be accessed.
See Also
--------
register_dataframe_namespace : Register functionality on a DataFrame.
register_lazyframe_namespace : Register functionality on a LazyFrame.
register_series_namespace : Register functionality on a Series.
Examples
--------
>>> @pl.api.register_expr_namespace("pow_n")
... class PowersOfN:
... def __init__(self, expr: pl.Expr) -> None:
... self._expr = expr
...
... def next(self, p: int) -> pl.Expr:
... return (p ** (self._expr.log(p).ceil()).cast(pl.Int64)).cast(pl.Int64)
...
... def previous(self, p: int) -> pl.Expr:
... return (p ** (self._expr.log(p).floor()).cast(pl.Int64)).cast(pl.Int64)
...
... def nearest(self, p: int) -> pl.Expr:
... return (p ** (self._expr.log(p)).round(0).cast(pl.Int64)).cast(pl.Int64)
>>>
>>> df = pl.DataFrame([1.4, 24.3, 55.0, 64.001], schema=["n"])
>>> df.select(
... pl.col("n"),
... pl.col("n").pow_n.next(p=2).alias("next_pow2"),
... pl.col("n").pow_n.previous(p=2).alias("prev_pow2"),
... pl.col("n").pow_n.nearest(p=2).alias("nearest_pow2"),
... )
shape: (4, 4)
┌────────┬───────────┬───────────┬──────────────┐
│ n ┆ next_pow2 ┆ prev_pow2 ┆ nearest_pow2 │
│ --- ┆ --- ┆ --- ┆ --- │
│ f64 ┆ i64 ┆ i64 ┆ i64 │
╞════════╪═══════════╪═══════════╪══════════════╡
│ 1.4 ┆ 2 ┆ 1 ┆ 1 │
│ 24.3 ┆ 32 ┆ 16 ┆ 32 │
│ 55.0 ┆ 64 ┆ 32 ┆ 64 │
│ 64.001 ┆ 128 ┆ 64 ┆ 64 │
└────────┴───────────┴───────────┴──────────────┘
register_lazyframe_namespace(name: 'str') -> 'Callable[[type[NS]], type[NS]]'
Decorator for registering custom functionality with a Polars LazyFrame.
Parameters
----------
name
Name under which the functionality will be accessed.
See Also
--------
register_expr_namespace : Register functionality on an Expr.
register_dataframe_namespace : Register functionality on a DataFrame.
register_series_namespace : Register functionality on a Series.
Examples
--------
>>> @pl.api.register_lazyframe_namespace("types")
... class DTypeOperations:
... def __init__(self, lf: pl.LazyFrame) -> None:
... self._lf = lf
...
... def split_by_column_dtypes(self) -> list[pl.LazyFrame]:
... return [
... self._lf.select(pl.col(tp))
... for tp in dict.fromkeys(self._lf.collect_schema().dtypes())
... ]
...
... def upcast_integer_types(self) -> pl.LazyFrame:
... return self._lf.with_columns(
... pl.col(tp).cast(pl.Int64) for tp in (pl.Int8, pl.Int16, pl.Int32)
... )
>>>
>>> lf = pl.LazyFrame(
... data={"a": [1, 2], "b": [3, 4], "c": [5.6, 6.7]},
... schema=[("a", pl.Int16), ("b", pl.Int32), ("c", pl.Float32)],
... )
>>> lf.collect()
shape: (2, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i16 ┆ i32 ┆ f32 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 5.6 │
│ 2 ┆ 4 ┆ 6.7 │
└─────┴─────┴─────┘
>>> lf.types.upcast_integer_types().collect()
shape: (2, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ f32 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 5.6 │
│ 2 ┆ 4 ┆ 6.7 │
└─────┴─────┴─────┘
>>> lf = pl.LazyFrame(
... data=[["xx", 2, 3, 4], ["xy", 4, 5, 6], ["yy", 5, 6, 7], ["yz", 6, 7, 8]],
... schema=["a1", "a2", "b1", "b2"],
... orient="row",
... )
>>> lf.collect()
shape: (4, 4)
┌─────┬─────┬─────┬─────┐
│ a1 ┆ a2 ┆ b1 ┆ b2 │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
│ xx ┆ 2 ┆ 3 ┆ 4 │
│ xy ┆ 4 ┆ 5 ┆ 6 │
│ yy ┆ 5 ┆ 6 ┆ 7 │
│ yz ┆ 6 ┆ 7 ┆ 8 │
└─────┴─────┴─────┴─────┘
>>> pl.collect_all(lf.types.split_by_column_dtypes())
[shape: (4, 1)
┌─────┐
│ a1 │
│ --- │
│ str │
╞═════╡
│ xx │
│ xy │
│ yy │
│ yz │
└─────┘, shape: (4, 3)
┌─────┬─────┬─────┐
│ a2 ┆ b1 ┆ b2 │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 2 ┆ 3 ┆ 4 │
│ 4 ┆ 5 ┆ 6 │
│ 5 ┆ 6 ┆ 7 │
│ 6 ┆ 7 ┆ 8 │
└─────┴─────┴─────┘]
register_series_namespace(name: 'str') -> 'Callable[[type[NS]], type[NS]]'
Decorator for registering custom functionality with a polars Series.
Parameters
----------
name
Name under which the functionality will be accessed.
See Also
--------
register_expr_namespace : Register functionality on an Expr.
register_dataframe_namespace : Register functionality on a DataFrame.
register_lazyframe_namespace : Register functionality on a LazyFrame.
Examples
--------
>>> @pl.api.register_series_namespace("math")
... class MathShortcuts:
... def __init__(self, s: pl.Series) -> None:
... self._s = s
...
... def square(self) -> pl.Series:
... return self._s * self._s
...
... def cube(self) -> pl.Series:
... return self._s * self._s * self._s
>>>
>>> s = pl.Series("n", [1.5, 31.0, 42.0, 64.5])
>>> s.math.square().alias("s^2")
shape: (4,)
Series: 's^2' [f64]
[
2.25
961.0
1764.0
4160.25
]
>>> s = pl.Series("n", [1, 2, 3, 4, 5])
>>> s.math.cube().alias("s^3")
shape: (5,)
Series: 's^3' [i64]
[
1
8
27
64
125
]
DATA
__all__ = ['register_dataframe_namespace', 'register_expr_namespace', ...
__annotations__ = {'_reserved_namespaces': 'set[str]'}
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\api.py
polars.approx_n_unique
Python Library Documentation: function approx_n_unique in module polars.functions.lazy
approx_n_unique(*columns: 'str') -> 'Expr'
Approximate count of unique values.
This function is syntactic sugar for `pl.col(columns).approx_n_unique()`, and
uses the HyperLogLog++ algorithm for cardinality estimation.
Parameters
----------
columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 1],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.approx_n_unique("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 2 │
└─────┘
>>> df.select(pl.approx_n_unique("b", "c"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ u32 ┆ u32 │
╞═════╪═════╡
│ 3 ┆ 2 │
└─────┴─────┘
polars.arange
Python Library Documentation: function arange in module polars.functions.range.int_range
arange(start: 'int | IntoExprColumn' = 0, end: 'int | IntoExprColumn | None' = None, step: 'int' = 1, *, dtype: 'PolarsIntegerType | DataTypeExpr' = Int64, eager: 'bool' = False) -> 'Expr | Series'
Generate a range of integers.
Alias for :func:`int_range`.
Parameters
----------
start
Lower bound of the range (inclusive).
end
Upper bound of the range (exclusive).
step
Step size of the range.
dtype
Data type of the range. Defaults to `Int64`.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of integer data type `dtype`.
See Also
--------
int_range : Generate a range of integers.
int_ranges : Generate a range of integers for each row of the input columns.
Examples
--------
>>> pl.arange(0, 3, eager=True)
shape: (3,)
Series: 'literal' [i64]
[
0
1
2
]
polars.arctan2
Python Library Documentation: function arctan2 in module polars.functions.lazy
arctan2(y: 'str | Expr', x: 'str | Expr') -> 'Expr'
Compute two argument arctan in radians.
Returns the angle (in radians) in the plane between the
positive x-axis and the ray from the origin to (x,y).
Parameters
----------
y
Column name or Expression.
x
Column name or Expression.
Examples
--------
>>> c = (2**0.5) / 2
>>> df = pl.DataFrame(
... {
... "y": [c, -c, c, -c],
... "x": [c, c, -c, -c],
... }
... )
>>> df.with_columns(pl.arctan2("y", "x").alias("atan2"))
shape: (4, 3)
┌───────────┬───────────┬───────────┐
│ y ┆ x ┆ atan2 │
│ --- ┆ --- ┆ --- │
│ f64 ┆ f64 ┆ f64 │
╞═══════════╪═══════════╪═══════════╡
│ 0.707107 ┆ 0.707107 ┆ 0.785398 │
│ -0.707107 ┆ 0.707107 ┆ -0.785398 │
│ 0.707107 ┆ -0.707107 ┆ 2.356194 │
│ -0.707107 ┆ -0.707107 ┆ -2.356194 │
└───────────┴───────────┴───────────┘
polars.arctan2d
Python Library Documentation: function arctan2d in module polars.functions.lazy
arctan2d(y: 'str | Expr', x: 'str | Expr') -> 'Expr'
Compute two argument arctan in degrees.
.. deprecated:: 1.0.0
Use `arctan2` followed by :meth:`Expr.degrees` instead.
Returns the angle (in degrees) in the plane between the positive x-axis
and the ray from the origin to (x,y).
Parameters
----------
y
Column name or Expression.
x
Column name or Expression.
Examples
--------
>>> c = (2**0.5) / 2
>>> df = pl.DataFrame(
... {
... "y": [c, -c, c, -c],
... "x": [c, c, -c, -c],
... }
... )
>>> df.select( # doctest: +SKIP
... pl.arctan2d("y", "x").alias("atan2d"),
... pl.arctan2("y", "x").alias("atan2"),
... )
shape: (4, 2)
┌────────┬───────────┐
│ atan2d ┆ atan2 │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞════════╪═══════════╡
│ 45.0 ┆ 0.785398 │
│ -45.0 ┆ -0.785398 │
│ 135.0 ┆ 2.356194 │
│ -135.0 ┆ -2.356194 │
└────────┴───────────┘
polars.arg_sort_by
Python Library Documentation: function arg_sort_by in module polars.functions.lazy
arg_sort_by(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False, multithreaded: 'bool' = True, maintain_order: 'bool' = False) -> 'Expr'
Return the row indices that would sort the column(s).
Parameters
----------
exprs
Column(s) to arg sort by. Accepts expression input. Strings are parsed as column
names.
*more_exprs
Additional columns to arg sort by, specified as positional arguments.
descending
Sort in descending order. When sorting by multiple columns, can be specified
per column by passing a sequence of booleans.
nulls_last
Place null values last.
multithreaded
Sort using multiple threads.
maintain_order
Whether the order should be maintained if elements are equal.
See Also
--------
Expr.gather: Take values by index.
Expr.rank : Get the rank of each row.
Examples
--------
Pass a single column name to compute the arg sort by that column.
>>> df = pl.DataFrame(
... {
... "a": [0, 1, 1, 0],
... "b": [3, 2, 3, 2],
... "c": [1, 2, 3, 4],
... }
... )
>>> df.select(pl.arg_sort_by("a"))
shape: (4, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 0 │
│ 3 │
│ 1 │
│ 2 │
└─────┘
Compute the arg sort by multiple columns by either passing a list of columns, or by
specifying each column as a positional argument.
>>> df.select(pl.arg_sort_by(["a", "b"], descending=True))
shape: (4, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 2 │
│ 1 │
│ 0 │
│ 3 │
└─────┘
Use gather to apply the arg sort to other columns.
>>> df.select(pl.col("c").gather(pl.arg_sort_by("a")))
shape: (4, 1)
┌─────┐
│ c │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 4 │
│ 2 │
│ 3 │
└─────┘
polars.arg_where
Python Library Documentation: function arg_where in module polars.functions.lazy
arg_where(condition: 'Expr | Series', *, eager: 'bool' = False) -> 'Expr | Series'
Return indices where `condition` evaluates `True`.
Parameters
----------
condition
Boolean expression to evaluate
eager
Evaluate immediately and return a `Series`; this requires that the given
condition is itself a `Series`. If set to `False` (default), return
an expression instead.
See Also
--------
Series.arg_true : Return indices where Series is True
Examples
--------
>>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5]})
>>> df.select(
... [
... pl.arg_where(pl.col("a") % 2 == 0),
... ]
... ).to_series()
shape: (2,)
Series: 'a' [u32]
[
1
3
]
polars.build_info
Python Library Documentation: function build_info in module polars.meta.build
build_info() -> 'dict[str, Any]'
Return detailed Polars build information.
The dictionary with build information contains the following keys:
- `"compiler"`
- `"time"`
- `"dependencies"`
- `"features"`
- `"host"`
- `"target"`
- `"git"`
- `"version"`
If Polars was compiled without the `build_info` feature flag, only the `"version"`
key is included.
polars.business_day_count
Python Library Documentation: function business_day_count in module polars.functions.business
business_day_count(start: 'date | IntoExprColumn', end: 'date | IntoExprColumn', *, week_mask: 'Iterable[bool]' = (True, True, True, True, True, False, False), holidays: 'Iterable[date] | Expr | Series' = ()) -> 'Expr'
Count the number of business days between `start` and `end` (not including `end`).
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
.. versionchanged:: 1.27.0
Parameters after `start` and `end` should now be passed as keyword arguments.
Parameters
----------
start
Start dates.
end
End dates.
week_mask
Which days of the week to count. The default is Monday to Friday.
If you wanted to count only Monday to Thursday, you would pass
`(True, True, True, True, False, False, False)`.
holidays
Holidays to exclude from the count. The Python package
`python-holidays <https://github.com/vacanza/python-holidays>`_
may come in handy here. You can install it with ``pip install holidays``,
and then, to get all Dutch holidays for years 2020-2024:
.. code-block:: python
import holidays
my_holidays = holidays.country_holidays("NL", years=range(2020, 2025))
and pass `holidays=my_holidays` when you call `business_day_count`.
Returns
-------
Expr
Examples
--------
>>> from datetime import date
>>> df = pl.DataFrame(
... {
... "start": [date(2020, 1, 1), date(2020, 1, 2)],
... "end": [date(2020, 1, 2), date(2020, 1, 10)],
... }
... )
>>> df.with_columns(
... business_day_count=pl.business_day_count("start", "end"),
... )
shape: (2, 3)
┌────────────┬────────────┬────────────────────┐
│ start ┆ end ┆ business_day_count │
│ --- ┆ --- ┆ --- │
│ date ┆ date ┆ i32 │
╞════════════╪════════════╪════════════════════╡
│ 2020-01-01 ┆ 2020-01-02 ┆ 1 │
│ 2020-01-02 ┆ 2020-01-10 ┆ 6 │
└────────────┴────────────┴────────────────────┘
Note how the business day count is 6 (as opposed a regular day count of 8)
due to the weekend (2020-01-04 - 2020-01-05) not being counted.
You can pass a custom weekend - for example, if you only take Sunday off:
>>> week_mask = (True, True, True, True, True, True, False)
>>> df.with_columns(
... business_day_count=pl.business_day_count(
... "start", "end", week_mask=week_mask
... ),
... )
shape: (2, 3)
┌────────────┬────────────┬────────────────────┐
│ start ┆ end ┆ business_day_count │
│ --- ┆ --- ┆ --- │
│ date ┆ date ┆ i32 │
╞════════════╪════════════╪════════════════════╡
│ 2020-01-01 ┆ 2020-01-02 ┆ 1 │
│ 2020-01-02 ┆ 2020-01-10 ┆ 7 │
└────────────┴────────────┴────────────────────┘
You can also pass a list of holidays to exclude from the count:
>>> from datetime import date
>>> holidays = [date(2020, 1, 1), date(2020, 1, 2)]
>>> df.with_columns(
... business_day_count=pl.business_day_count("start", "end", holidays=holidays)
... )
shape: (2, 3)
┌────────────┬────────────┬────────────────────┐
│ start ┆ end ┆ business_day_count │
│ --- ┆ --- ┆ --- │
│ date ┆ date ┆ i32 │
╞════════════╪════════════╪════════════════════╡
│ 2020-01-01 ┆ 2020-01-02 ┆ 0 │
│ 2020-01-02 ┆ 2020-01-10 ┆ 5 │
└────────────┴────────────┴────────────────────┘
polars.catalog
Python Library Documentation: package polars.catalog in polars
NAME
polars.catalog
PACKAGE CONTENTS
unity (package)
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\catalog\__init__.py
polars.coalesce
Python Library Documentation: function coalesce in module polars.functions.lazy
coalesce(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr', eager: 'bool' = False) -> 'Expr | Series'
Folds the columns from left to right, keeping the first non-null value.
Parameters
----------
exprs
Columns to coalesce. Accepts expression input. Strings are parsed as column
names, other non-expression inputs are parsed as literals.
*more_exprs
Additional columns to coalesce, specified as positional arguments.
eager
Evaluate immediately and return a `Series`; this requires that at least one
of the given arguments is a `Series`. If set to `False` (default), return
an expression instead.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, None, None, None],
... "b": [1, 2, None, None],
... "c": [5, None, 3, None],
... }
... )
>>> df.with_columns(pl.coalesce("a", "b", "c", 10).alias("d"))
shape: (4, 4)
┌──────┬──────┬──────┬─────┐
│ a ┆ b ┆ c ┆ d │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞══════╪══════╪══════╪═════╡
│ 1 ┆ 1 ┆ 5 ┆ 1 │
│ null ┆ 2 ┆ null ┆ 2 │
│ null ┆ null ┆ 3 ┆ 3 │
│ null ┆ null ┆ null ┆ 10 │
└──────┴──────┴──────┴─────┘
>>> df.with_columns(pl.coalesce(pl.col(["a", "b", "c"]), 10.0).alias("d"))
shape: (4, 4)
┌──────┬──────┬──────┬──────┐
│ a ┆ b ┆ c ┆ d │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ f64 │
╞══════╪══════╪══════╪══════╡
│ 1 ┆ 1 ┆ 5 ┆ 1.0 │
│ null ┆ 2 ┆ null ┆ 2.0 │
│ null ┆ null ┆ 3 ┆ 3.0 │
│ null ┆ null ┆ null ┆ 10.0 │
└──────┴──────┴──────┴──────┘
>>> s1 = pl.Series("a", [None, 2, None])
>>> s2 = pl.Series("b", [1, None, 3])
>>> pl.coalesce(s1, s2, eager=True)
shape: (3,)
Series: 'a' [i64]
[
1
2
3
]
polars.col
Python Library Documentation: Col in module polars.functions.col
<Expr ['col("__origin__")']>
polars.collect_all
Python Library Documentation: function collect_all in module polars.functions.lazy
collect_all(lazy_frames: 'Iterable[LazyFrame]', *, type_coercion: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, no_optimization: 'bool' = False, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, engine: 'EngineType' = 'auto', lazy: 'bool' = False) -> 'list[DataFrame] | LazyFrame'
Collect multiple LazyFrames at the same time.
This can run all the computation graphs in parallel or combined.
Common Subplan Elimination is applied on the combined plan, meaning
that diverging queries will run only once.
Parameters
----------
lazy_frames
A list of LazyFrames to collect.
type_coercion
Do type coercion optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
predicate_pushdown
Do predicate pushdown optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
projection_pushdown
Do projection pushdown optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
simplify_expression
Run simplify expressions optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
no_optimization
Turn off optimizations.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
slice_pushdown
Slice pushdown optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
comm_subplan_elim
Will try to cache branching subplans that occur on self-joins or unions.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
comm_subexpr_elim
Common subexpressions will be cached and reused.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
cluster_with_columns
Combine sequential independent calls to with_columns
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
collapse_joins
Collapse a join and filters into a faster join
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
optimizations
The optimization passes done during query optimization.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
engine
Select the engine used to process the query (default ``"auto"``):
* ``"auto"``: use the engine set by
:meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
back to ``"in-memory"`` if unset (this default may change in
a future release).
* ``"in-memory"``: use the in-memory engine, this is the default engine.
* ``"streaming"``: use the streaming engine, which processes
queries in batches, reducing memory pressure and often
outperforming the in-memory engine. This will soon become
the default engine of Polars.
* ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
``cudf-polars``). Pass a :class:`~.GPUEngine` object for
fine-grained control (e.g. device selection on multi-GPU
systems).
If the selected engine cannot run the query, Polars falls back to
the in-memory engine.
.. note::
The GPU engine does not support async, or running in the
background. If either are enabled, then GPU execution is switched off.
lazy:
Return as LazyFrame that can be collected later.
This is only correct if all inputs sink to disk.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Returns
-------
list of DataFrames
The collected DataFrames, returned in the same order as the input LazyFrames.
polars.collect_all_async
Python Library Documentation: function collect_all_async in module polars.functions.lazy
collect_all_async(lazy_frames: 'Iterable[LazyFrame]', *, gevent: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'Awaitable[list[DataFrame]] | _GeventDataFrameResult[list[DataFrame]]'
Collect multiple LazyFrames at the same time asynchronously in thread pool.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Collects into a list of DataFrame (like :func:`polars.collect_all`),
but instead of returning them directly, they are scheduled to be collected
inside thread pool, while this method returns almost instantly.
May be useful if you use gevent or asyncio and want to release control to other
greenlets/tasks while LazyFrames are being collected.
Parameters
----------
lazy_frames
A list of LazyFrames to collect.
gevent
Return wrapper to `gevent.event.AsyncResult` instead of Awaitable
optimizations
The optimization passes done during query optimization.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
engine
Select the engine used to process the query (default ``"auto"``):
* ``"auto"``: use the engine set by
:meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
back to ``"in-memory"`` if unset (this default may change in
a future release).
* ``"in-memory"``: use the in-memory engine, this is the default engine.
* ``"streaming"``: use the streaming engine, which processes
queries in batches, reducing memory pressure and often
outperforming the in-memory engine. This will soon become
the default engine of Polars.
* ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
``cudf-polars``). Pass a :class:`~.GPUEngine` object for
fine-grained control (e.g. device selection on multi-GPU
systems).
If the selected engine cannot run the query, Polars falls back to
the in-memory engine.
.. note::
The GPU engine does not support async, or running in the
background. If either are enabled, then GPU execution is switched off.
See Also
--------
polars.collect_all : Collect multiple LazyFrames at the same time.
LazyFrame.collect_async : To collect single frame.
Notes
-----
In case of error `set_exception` is used on
`asyncio.Future`/`gevent.event.AsyncResult` and will be reraised by them.
Returns
-------
If `gevent=False` (default) then returns awaitable.
If `gevent=True` then returns wrapper that has
`.get(block=True, timeout=None)` method.
polars.concat
Python Library Documentation: function concat in module polars.functions.eager
concat(items: 'Iterable[PolarsType]', *, how: 'ConcatMethod' = 'vertical', rechunk: 'bool' = False, parallel: 'bool' = True, strict: 'bool | None' = None) -> 'PolarsType'
Combine multiple DataFrames, LazyFrames, or Series into a single object.
Parameters
----------
items
DataFrames, LazyFrames, or Series to concatenate.
how : {'vertical', 'vertical_relaxed', 'diagonal', 'diagonal_relaxed', 'horizontal', 'horizontal_extend', 'align', 'align_full', 'align_inner', 'align_left', 'align_right'}
Note that `Series` only support the `vertical` strategy.
* vertical: Applies multiple `vstack` operations.
* vertical_relaxed: Same as `vertical`, but additionally coerces columns to
their common supertype *if* they are mismatched (eg: Int32 → Int64).
* diagonal: Finds a union between the column schemas and fills missing column
values with `null`.
* diagonal_relaxed: Same as `diagonal`, but additionally coerces columns to
their common supertype *if* they are mismatched (eg: Int32 → Int64).
* horizontal: Stacks Series from DataFrames horizontally. All input frames
must have the same height; raises a `ShapeError` otherwise.
* horizontal_extend: Same as `horizontal`, but pads shorter frames with `null`
if the lengths don't match.
* align, align_full, align_left, align_right: Combines frames horizontally,
auto-determining the common key columns and aligning rows using the same
logic as `align_frames` (note that "align" is an alias for "align_full").
The "align" strategy determines the type of join used to align the frames,
equivalent to the "how" parameter on `align_frames`. Note that the common
join columns are automatically coalesced, but other column collisions
will raise an error (if you need more control over this you should use
a suitable `join` method directly).
rechunk
Make sure that the result data is in contiguous memory.
parallel
Only relevant for LazyFrames. This determines if the concatenated
lazy computations may be executed in parallel.
strict
When how=`horizontal`, require all DataFrames to be the same height, raising an error if not.
.. deprecated:: 1.42.1
Use `how='horizontal'` (equal heights) or
`how='horizontal_extend'` (pad with null) instead.
Examples
--------
>>> df1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df2 = pl.DataFrame({"a": [2], "b": [4]})
>>> pl.concat([df1, df2]) # default is 'vertical' strategy
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 3 │
│ 2 ┆ 4 │
└─────┴─────┘
>>> df1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df2 = pl.DataFrame({"a": [2.5], "b": [4]})
>>> pl.concat([df1, df2], how="vertical_relaxed") # 'a' coerced into f64
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ f64 ┆ i64 │
╞═════╪═════╡
│ 1.0 ┆ 3 │
│ 2.5 ┆ 4 │
└─────┴─────┘
>>> df_h1 = pl.DataFrame({"l1": [1, 2], "l2": [3, 4]})
>>> df_h2 = pl.DataFrame({"r1": [5, 6], "r2": [7, 8], "r3": [9, 10]})
>>> pl.concat([df_h1, df_h2], how="horizontal_extend")
shape: (2, 5)
┌─────┬─────┬─────┬─────┬─────┐
│ l1 ┆ l2 ┆ r1 ┆ r2 ┆ r3 │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 5 ┆ 7 ┆ 9 │
│ 2 ┆ 4 ┆ 6 ┆ 8 ┆ 10 │
└─────┴─────┴─────┴─────┴─────┘
The "diagonal" strategy allows for some frames to have missing columns,
the values for which are filled with `null`:
>>> df_d1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df_d2 = pl.DataFrame({"a": [2], "c": [4]})
>>> pl.concat([df_d1, df_d2], how="diagonal")
shape: (2, 3)
┌─────┬──────┬──────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╡
│ 1 ┆ 3 ┆ null │
│ 2 ┆ null ┆ 4 │
└─────┴──────┴──────┘
The "align" strategies require at least one common column to align on:
>>> df_a1 = pl.DataFrame({"id": [1, 2], "x": [3, 4]})
>>> df_a2 = pl.DataFrame({"id": [2, 3], "y": [5, 6]})
>>> df_a3 = pl.DataFrame({"id": [1, 3], "z": [7, 8]})
>>> pl.concat([df_a1, df_a2, df_a3], how="align") # equivalent to "align_full"
shape: (3, 4)
┌─────┬──────┬──────┬──────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╪══════╡
│ 1 ┆ 3 ┆ null ┆ 7 │
│ 2 ┆ 4 ┆ 5 ┆ null │
│ 3 ┆ null ┆ 6 ┆ 8 │
└─────┴──────┴──────┴──────┘
>>> pl.concat([df_a1, df_a2, df_a3], how="align_left")
shape: (2, 4)
┌─────┬─────┬──────┬──────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪══════╪══════╡
│ 1 ┆ 3 ┆ null ┆ 7 │
│ 2 ┆ 4 ┆ 5 ┆ null │
└─────┴─────┴──────┴──────┘
>>> pl.concat([df_a1, df_a2, df_a3], how="align_right")
shape: (2, 4)
┌─────┬──────┬──────┬─────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╪═════╡
│ 1 ┆ null ┆ null ┆ 7 │
│ 3 ┆ null ┆ 6 ┆ 8 │
└─────┴──────┴──────┴─────┘
>>> pl.concat([df_a1, df_a2, df_a3], how="align_inner")
shape: (0, 4)
┌─────┬─────┬─────┬─────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
└─────┴─────┴─────┴─────┘
polars.concat_arr
Python Library Documentation: function concat_arr in module polars.functions.as_datatype
concat_arr(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr') -> 'Expr'
Horizontally concatenate columns into a single array column.
Non-array columns are reshaped to a unit-width array. All columns must have
a dtype of either `pl.Array(<DataType>, width)` or `pl.<DataType>`.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
exprs
Columns to concatenate into a single array column. Accepts expression input.
Strings are parsed as column names, other non-expression inputs are parsed as
literals.
*more_exprs
Additional columns to concatenate into a single array column, specified as
positional arguments.
Examples
--------
Concatenate 2 array columns:
>>> (
... pl.select(
... a=pl.Series([[1], [3], None], dtype=pl.Array(pl.Int64, 1)),
... b=pl.Series([[3], [None], [5]], dtype=pl.Array(pl.Int64, 1)),
... ).with_columns(
... pl.concat_arr("a", "b").alias("concat_arr(a, b)"),
... pl.concat_arr("a", pl.first("b")).alias("concat_arr(a, first(b))"),
... )
... )
shape: (3, 4)
┌───────────────┬───────────────┬──────────────────┬─────────────────────────┐
│ a ┆ b ┆ concat_arr(a, b) ┆ concat_arr(a, first(b)) │
│ --- ┆ --- ┆ --- ┆ --- │
│ array[i64, 1] ┆ array[i64, 1] ┆ array[i64, 2] ┆ array[i64, 2] │
╞═══════════════╪═══════════════╪══════════════════╪═════════════════════════╡
│ [1] ┆ [3] ┆ [1, 3] ┆ [1, 3] │
│ [3] ┆ [null] ┆ [3, null] ┆ [3, 3] │
│ null ┆ [5] ┆ null ┆ null │
└───────────────┴───────────────┴──────────────────┴─────────────────────────┘
Concatenate non-array columns:
>>> (
... pl.select(
... c=pl.Series([None, 5, 6], dtype=pl.Int64),
... )
... .with_columns(d=pl.col("c").reverse())
... .with_columns(
... pl.concat_arr("c", "d").alias("concat_arr(c, d)"),
... )
... )
shape: (3, 3)
┌──────┬──────┬──────────────────┐
│ c ┆ d ┆ concat_arr(c, d) │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ array[i64, 2] │
╞══════╪══════╪══════════════════╡
│ null ┆ 6 ┆ [null, 6] │
│ 5 ┆ 5 ┆ [5, 5] │
│ 6 ┆ null ┆ [6, null] │
└──────┴──────┴──────────────────┘
Concatenate mixed array and non-array columns:
>>> (
... pl.select(
... a=pl.Series([[1], [3], None], dtype=pl.Array(pl.Int64, 1)),
... b=pl.Series([[3], [None], [5]], dtype=pl.Array(pl.Int64, 1)),
... c=pl.Series([None, 5, 6], dtype=pl.Int64),
... ).with_columns(
... pl.concat_arr("a", "b", "c").alias("concat_arr(a, b, c)"),
... )
... )
shape: (3, 4)
┌───────────────┬───────────────┬──────┬─────────────────────┐
│ a ┆ b ┆ c ┆ concat_arr(a, b, c) │
│ --- ┆ --- ┆ --- ┆ --- │
│ array[i64, 1] ┆ array[i64, 1] ┆ i64 ┆ array[i64, 3] │
╞═══════════════╪═══════════════╪══════╪═════════════════════╡
│ [1] ┆ [3] ┆ null ┆ [1, 3, null] │
│ [3] ┆ [null] ┆ 5 ┆ [3, null, 5] │
│ null ┆ [5] ┆ 6 ┆ null │
└───────────────┴───────────────┴──────┴─────────────────────┘
Unit-length columns are broadcasted:
>>> (
... pl.select(
... a=pl.Series([1, 3, None]),
... ).with_columns(
... pl.concat_arr("a", pl.lit(0, dtype=pl.Int64)).alias("concat_arr(a, 0)"),
... pl.concat_arr("a", pl.sum("a")).alias("concat_arr(a, sum(a))"),
... pl.concat_arr("a", pl.max("a")).alias("concat_arr(a, max(a))"),
... )
... )
shape: (3, 4)
┌──────┬──────────────────┬───────────────────────┬───────────────────────┐
│ a ┆ concat_arr(a, 0) ┆ concat_arr(a, sum(a)) ┆ concat_arr(a, max(a)) │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ array[i64, 2] ┆ array[i64, 2] ┆ array[i64, 2] │
╞══════╪══════════════════╪═══════════════════════╪═══════════════════════╡
│ 1 ┆ [1, 0] ┆ [1, 4] ┆ [1, 3] │
│ 3 ┆ [3, 0] ┆ [3, 4] ┆ [3, 3] │
│ null ┆ [null, 0] ┆ [null, 4] ┆ [null, 3] │
└──────┴──────────────────┴───────────────────────┴───────────────────────┘
polars.concat_list
Python Library Documentation: function concat_list in module polars.functions.as_datatype
concat_list(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr') -> 'Expr'
Horizontally concatenate columns into a single list column.
Operates in linear time.
Parameters
----------
exprs
Columns to concatenate into a single list column. Accepts expression input.
Strings are parsed as column names, other non-expression inputs are parsed as
literals.
*more_exprs
Additional columns to concatenate into a single list column, specified as
positional arguments.
Examples
--------
Concatenate two existing list columns. Null values are propagated.
>>> df = pl.DataFrame({"a": [[1, 2], [3], [4, 5]], "b": [[4], [], None]})
>>> df.with_columns(concat_list=pl.concat_list("a", "b"))
shape: (3, 3)
┌───────────┬───────────┬─────────────┐
│ a ┆ b ┆ concat_list │
│ --- ┆ --- ┆ --- │
│ list[i64] ┆ list[i64] ┆ list[i64] │
╞═══════════╪═══════════╪═════════════╡
│ [1, 2] ┆ [4] ┆ [1, 2, 4] │
│ [3] ┆ [] ┆ [3] │
│ [4, 5] ┆ null ┆ null │
└───────────┴───────────┴─────────────┘
Non-list columns are cast to a list before concatenation. The output data type
is the supertype of the concatenated columns.
>>> df.select("a", concat_list=pl.concat_list("a", pl.lit("x")))
shape: (3, 2)
┌───────────┬─────────────────┐
│ a ┆ concat_list │
│ --- ┆ --- │
│ list[i64] ┆ list[str] │
╞═══════════╪═════════════════╡
│ [1, 2] ┆ ["1", "2", "x"] │
│ [3] ┆ ["3", "x"] │
│ [4, 5] ┆ ["4", "5", "x"] │
└───────────┴─────────────────┘
Create lagged columns and collect them into a list. This mimics a rolling window.
>>> df = pl.DataFrame({"A": [1.0, 2.0, 9.0, 2.0, 13.0]})
>>> df = df.select([pl.col("A").shift(i).alias(f"A_lag_{i}") for i in range(3)])
>>> df.select(
... pl.concat_list([f"A_lag_{i}" for i in range(3)][::-1]).alias("A_rolling")
... )
shape: (5, 1)
┌───────────────────┐
│ A_rolling │
│ --- │
│ list[f64] │
╞═══════════════════╡
│ [null, null, 1.0] │
│ [null, 1.0, 2.0] │
│ [1.0, 2.0, 9.0] │
│ [2.0, 9.0, 2.0] │
│ [9.0, 2.0, 13.0] │
└───────────────────┘
polars.concat_str
Python Library Documentation: function concat_str in module polars.functions.as_datatype
concat_str(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr', separator: 'str' = '', ignore_nulls: 'bool' = False) -> 'Expr'
Horizontally concatenate columns into a single string column.
Operates in linear time.
Parameters
----------
exprs
Columns to concatenate into a single string column. Accepts expression input.
Strings are parsed as column names, other non-expression inputs are parsed as
literals. Non-`String` columns are cast to `String`.
*more_exprs
Additional columns to concatenate into a single string column, specified as
positional arguments.
separator
String that will be used to separate the values of each column.
ignore_nulls
Ignore null values (default is ``False``).
If set to ``False``, null values will be propagated.
if the row contains any null values, the output is null.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": ["dogs", "cats", None],
... "c": ["play", "swim", "walk"],
... }
... )
>>> df.with_columns(
... pl.concat_str(
... [
... pl.col("a") * 2,
... pl.col("b"),
... pl.col("c"),
... ],
... separator=" ",
... ).alias("full_sentence"),
... )
shape: (3, 4)
┌─────┬──────┬──────┬───────────────┐
│ a ┆ b ┆ c ┆ full_sentence │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str ┆ str │
╞═════╪══════╪══════╪═══════════════╡
│ 1 ┆ dogs ┆ play ┆ 2 dogs play │
│ 2 ┆ cats ┆ swim ┆ 4 cats swim │
│ 3 ┆ null ┆ walk ┆ null │
└─────┴──────┴──────┴───────────────┘
polars.config
Python Library Documentation: module polars.config in polars
NAME
polars.config
CLASSES
contextlib.ContextDecorator(builtins.object)
Config
class Config(contextlib.ContextDecorator)
| Config(*, restore_defaults: 'bool' = False, apply_on_context_enter: 'bool' = False, **options: 'Unpack[ConfigParameters]') -> 'None'
|
| Configure polars; offers options for table formatting and more.
|
| Notes
| -----
| Can also be used as a context manager OR a function decorator in order to
| temporarily scope the lifetime of specific options. For example:
|
| >>> with pl.Config() as cfg:
| ... # set verbose for more detailed output within the scope
| ... cfg.set_verbose(True) # doctest: +IGNORE_RESULT
| >>> # scope exit - no longer in verbose mode
|
| This can also be written more compactly as:
|
| >>> with pl.Config(verbose=True):
| ... pass
|
| (The compact format is available for all `Config` methods that take a single value).
|
| Alternatively, you can use as a decorator in order to scope the duration of the
| selected options to a specific function:
|
| >>> @pl.Config(verbose=True, apply_on_context_enter=True)
| ... def test():
| ... pass
|
| Method resolution order:
| Config
| contextlib.ContextDecorator
| builtins.object
|
| Methods defined here:
|
| __enter__(self) -> 'Self'
| Support setting Config options that are reset on scope exit.
|
| __eq__(self, other: 'object') -> 'bool'
| Return self==value.
|
| __exit__(self, exc_type: 'type[BaseException] | None', exc_val: 'BaseException | None', exc_tb: 'TracebackType | None') -> 'None'
| Reset any Config options that were set within the scope.
|
| __init__(self, *, restore_defaults: 'bool' = False, apply_on_context_enter: 'bool' = False, **options: 'Unpack[ConfigParameters]') -> 'None'
| Initialise a Config object instance for context manager usage.
|
| Any `options` kwargs should correspond to the available named "set_*"
| methods, but are allowed to omit the "set_" prefix for brevity.
|
| Parameters
| ----------
| restore_defaults
| set all options to their default values (this is applied before
| setting any other options).
| apply_on_context_enter
| defer applying the options until a context is entered. This allows you
| to create multiple `Config` instances with different options, and then
| reuse them independently as context managers or function decorators
| with specific bundles of parameters.
| **options
| keyword args that will set the option; equivalent to calling the
| named "set_<option>" method with the given value.
|
| Notes
| -----
| The `apply_on_context_enter` parameter should almost *always* be set True
| when using `Config` as a decorator to ensure that the options are applied
| only within the scope of the decorated function (and not globally at
| module import time).
|
| Examples
| --------
| Customise Polars table formatting while in context scope:
|
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> with pl.Config(
| ... # these options will be set for scope duration
| ... tbl_formatting="MARKDOWN",
| ... tbl_hide_dataframe_shape=True,
| ... tbl_rows=10,
| ... ):
| ... print(df)
| | abc | xyz |
| | --- | --- |
| | f64 | bool |
| |-----|-------|
| | 1.0 | true |
| | 2.5 | false |
| | 5.0 | true |
|
| Establish several independent Config instances for use in different contexts;
| setting `apply_on_context_enter=True` defers setting the parameters until a
| context (or function, when used as a decorator) is actually entered:
|
| >>> cfg_polars_verbose = pl.Config(
| ... verbose=True,
| ... apply_on_context_enter=True,
| ... )
| >>> cfg_polars_detailed_tables = pl.Config(
| ... tbl_rows=25,
| ... tbl_cols=25,
| ... tbl_width_chars=200,
| ... apply_on_context_enter=True,
| ... )
|
| These Config instances can now be applied independently and re-used:
|
| >>> @cfg_polars_verbose
| ... def traced_function(df: pl.DataFrame) -> pl.DataFrame:
| ... return polars_operations(df)
|
| >>> @cfg_polars_detailed_tables
| ... def print_detailed_frames(*frames: pl.DataFrame) -> None:
| ... for df in frames:
| ... print(df)
|
| __ne__(self, other: 'object') -> 'bool'
| Return self!=value.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| load(cfg: 'str') -> 'Config'
| Load (and set) previously saved Config options from a JSON string.
|
| Parameters
| ----------
| cfg : str
| JSON string produced by `Config.save()`.
|
| See Also
| --------
| load_from_file : Load (and set) Config options from a JSON file.
| save : Save the current set of Config options as a JSON string or file.
|
| load_from_file(file: 'Path | str') -> 'Config'
| Load (and set) previously saved Config options from file.
|
| Parameters
| ----------
| file : Path | str
| File path to a JSON string produced by `Config.save()`.
|
| See Also
| --------
| load : Load (and set) Config options from a JSON string.
| save : Save the current set of Config options as a JSON string or file.
|
| reload_env_vars() -> 'None'
| Update the Polars config from the set environment variables.
|
| Normally you need not call this, it is only necessary when updating
| undocumented (environment-variable-only) config flags from Python after
| importing Polars.
|
| restore_defaults() -> 'type[Config]'
| Reset all polars Config settings to their default state.
|
| Notes
| -----
| This method operates by removing all Config options from the environment,
| and then setting any local (non-env) options back to their default value.
|
| Examples
| --------
| >>> cfg = pl.Config.restore_defaults() # doctest: +SKIP
|
| save(*, if_set: 'bool' = False) -> 'str'
| Save the current set of Config options as a JSON string.
|
| Parameters
| ----------
| if_set
| By default this will save the state of all configuration options; set
| to `False` to save only those that have been set to a non-default value.
|
| See Also
| --------
| load : Load (and set) Config options from a JSON string.
| load_from_file : Load (and set) Config options from a JSON file.
| save_to_file : Save the current set of Config options as a JSON file.
|
| Examples
| --------
| >>> json_state = pl.Config.save()
|
| Returns
| -------
| str
| JSON string containing current Config options.
|
| save_to_file(file: 'Path | str') -> 'None'
| Save the current set of Config options as a JSON file.
|
| Parameters
| ----------
| file
| Optional path to a file into which the JSON string will be written.
| Leave as `None` to return the JSON string directly.
|
| See Also
| --------
| load : Load (and set) Config options from a JSON string.
| load_from_file : Load (and set) Config options from a JSON file.
| save : Save the current set of Config options as a JSON string.
|
| Examples
| --------
| >>> pl.Config().save_to_file("~/polars/config.json") # doctest: +SKIP
|
| set_ascii_tables(active: 'bool | None' = True) -> 'type[Config]'
| Use ASCII characters to display table outlines.
|
| Set False to revert to the standard UTF8_FULL_CONDENSED formatting style.
|
| See Also
| --------
| set_tbl_formatting : Set the table formatting style (includes Markdown option).
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_ascii_tables(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ +-----+-------+
| # │ abc ┆ xyz │ | abc | xyz |
| # │ --- ┆ --- │ | --- | --- |
| # │ f64 ┆ bool │ | f64 | bool |
| # ╞═════╪═══════╡ +=============+
| # │ 1.0 ┆ true │ >> | 1.0 | true |
| # │ 2.5 ┆ false │ | 2.5 | false |
| # │ 5.0 ┆ true │ | 5.0 | true |
| # └─────┴───────┘ +-----+-------+
|
| set_auto_structify(active: 'bool | None' = False) -> 'type[Config]'
| Allow multi-output expressions to be automatically turned into Structs.
|
| .. note::
| Deprecated since 1.32.0.
|
| Examples
| --------
| >>> df = pl.DataFrame({"v": [1, 2, 3], "v2": [4, 5, 6]})
| >>> with pl.Config(set_auto_structify=True): # doctest: +SKIP
| ... out = df.select(pl.all())
| >>> out # doctest: +SKIP
| shape: (3, 1)
| ┌───────────┐
| │ v │
| │ --- │
| │ struct[2] │
| ╞═══════════╡
| │ {1,4} │
| │ {2,5} │
| │ {3,6} │
| └───────────┘
|
| set_decimal_separator(separator: 'str | None' = None) -> 'type[Config]'
| Set the decimal separator character.
|
| Parameters
| ----------
| separator : str, bool
| Character to use as the decimal separator.
| Set to ``None`` to revert to the default (".").
|
| See Also
| --------
| set_thousands_separator : Set the thousands grouping separator character.
|
| Examples
| --------
| >>> df = pl.DataFrame({"v": [9876.54321, 1010101.0, -123456.78]})
| >>> with pl.Config(
| ... tbl_cell_numeric_alignment="RIGHT",
| ... thousands_separator=".",
| ... decimal_separator=",",
| ... float_precision=3,
| ... ):
| ... print(df)
| shape: (3, 1)
| ┌───────────────┐
| │ v │
| │ --- │
| │ f64 │
| ╞═══════════════╡
| │ 9.876,543 │
| │ 1.010.101,000 │
| │ -123.456,780 │
| └───────────────┘
|
| set_default_credential_provider(credential_provider: "CredentialProviderFunction | Literal['auto'] | None") -> 'type[Config]'
| Set a default credential provider.
|
| Sets the default credential provider to be used for functions that
| read / write to cloud storage.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| Can also be set to None, which globally disables auto-initialization
| of credential providers, or "auto" (the default behavior).
|
| Examples
| --------
| >>> pl.Config.set_default_credential_provider(
| ... pl.CredentialProviderAWS(
| ... assume_role={"RoleArn": "...", "RoleSessionName": "..."}
| ... )
| ... )
| <class 'polars.config.Config'>
|
| set_engine_affinity(engine: 'EngineType | None' = None) -> 'type[Config]'
| Set which engine to use by default.
|
| Parameters
| ----------
| engine : {None, 'auto', 'in-memory', 'streaming', 'gpu'}
| The default execution engine Polars will attempt to use
| when calling `.collect()`. However, the query is not
| guaranteed to execute with the specified engine.
|
| Examples
| --------
| >>> pl.Config.set_engine_affinity("streaming") # doctest: +SKIP
| >>> lf = pl.LazyFrame({"v": [1, 2, 3], "v2": [4, 5, 6]}) # doctest: +SKIP
| >>> lf.max().collect() # doctest: +SKIP
| shape: (3, 2)
| ┌─────┬─────┐
| │ v ┆ v2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| │ 3 ┆ 6 │
| └─────┴─────┘
| >>> pl.Config.set_engine_affinity("gpu") # doctest: +SKIP
| >>> lf.max().collect() # doctest: +SKIP
| shape: (3, 2)
| ┌─────┬─────┐
| │ v ┆ v2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| │ 3 ┆ 6 │
| └─────┴─────┘
|
| Raises
| ------
| ValueError: if engine is not recognised.
| NotImplementedError: if engine is a GPUEngine object
|
| set_expr_depth_warning(limit: 'int') -> 'type[Config]'
| Set the expression depth that Polars will accept without triggering a warning.
|
| Having too deep expressions (several 1000s) can lead to overflowing the stack and might be worth a refactor.
|
| set_float_precision(precision: 'int | None' = None) -> 'type[Config]'
| Control the number of decimal places displayed for floating point values.
|
| Parameters
| ----------
| precision : int
| Number of decimal places to display; set to `None` to revert to the
| default/standard behaviour.
|
| Notes
| -----
| When setting this to a larger value you should ensure that you are aware of both
| the limitations of floating point representations, and of the precision of the
| data that you are looking at.
|
| This setting only applies to :class:`.Float16`, :class:`.Float32`, and
| :class:`.Float64` dtypes; it does not cover :class:`.Decimal` dtype values
| (which are displayed at their native level of precision).
|
| Examples
| --------
| Set a large maximum float precision:
|
| >>> from math import pi, e
| >>> df = pl.DataFrame({"const": ["pi", "e"], "value": [pi, e]})
| >>> with pl.Config(float_precision=15):
| ... print(df)
| shape: (2, 2)
| ┌───────┬───────────────────┐
| │ const ┆ value │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪═══════════════════╡
| │ pi ┆ 3.141592653589793 │
| │ e ┆ 2.718281828459045 │
| └───────┴───────────────────┘
|
| Set a fixed float precision and align numeric columns to the
| right in order to cleanly line-up the decimal separator:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["xx", "yy"],
| ... "b": [-11111111, 44444444444],
| ... "c": [100000.987654321, -23456789],
| ... }
| ... )
| >>> with pl.Config(
| ... tbl_cell_numeric_alignment="RIGHT",
| ... thousands_separator=",",
| ... float_precision=3,
| ... ):
| ... print(df)
| shape: (2, 3)
| ┌─────┬────────────────┬─────────────────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ f64 │
| ╞═════╪════════════════╪═════════════════╡
| │ xx ┆ -11,111,111 ┆ 100,000.988 │
| │ yy ┆ 44,444,444,444 ┆ -23,456,789.000 │
| └─────┴────────────────┴─────────────────┘
|
| set_fmt_float(fmt: 'FloatFmt | None' = 'mixed') -> 'type[Config]'
| Control how floating point values are displayed.
|
| Parameters
| ----------
| fmt : {"mixed", "full"}
| How to format floating point numbers:
|
| - "mixed": Limit the number of decimal places and use scientific
| notation for large/small values.
| - "full": Print the full precision of the floating point number.
|
| Examples
| --------
| "mixed" float formatting:
|
| >>> s = pl.Series([1.2304980958725870923, 1e6, 1e-8])
| >>> with pl.Config(set_fmt_float="mixed"):
| ... print(s)
| shape: (3,)
| Series: '' [f64]
| [
| 1.230498
| 1e6
| 1.0000e-8
| ]
|
| "full" float formatting:
|
| >>> with pl.Config(set_fmt_float="full"):
| ... print(s)
| shape: (3,)
| Series: '' [f64]
| [
| 1.230498095872587
| 1000000
| 0.00000001
| ]
|
| set_fmt_str_lengths(n: 'int | None') -> 'type[Config]'
| Set the number of characters used to display string values.
|
| Parameters
| ----------
| n : int
| Number of characters to display.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "txt": [
| ... "Play it, Sam. Play 'As Time Goes By'.",
| ... "This is the beginning of a beautiful friendship.",
| ... ]
| ... }
| ... )
| >>> df.with_columns(pl.col("txt").str.len_bytes().alias("len"))
| shape: (2, 2)
| ┌─────────────────────────────────┬─────┐
| │ txt ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═════════════════════════════════╪═════╡
| │ Play it, Sam. Play 'As Time Go… ┆ 37 │
| │ This is the beginning of a bea… ┆ 48 │
| └─────────────────────────────────┴─────┘
| >>> with pl.Config(fmt_str_lengths=50):
| ... print(df)
| shape: (2, 1)
| ┌──────────────────────────────────────────────────┐
| │ txt │
| │ --- │
| │ str │
| ╞══════════════════════════════════════════════════╡
| │ Play it, Sam. Play 'As Time Goes By'. │
| │ This is the beginning of a beautiful friendship. │
| └──────────────────────────────────────────────────┘
|
| set_fmt_table_cell_list_len(n: 'int | None') -> 'type[Config]'
| Set the number of elements to display for List values.
|
| Empty lists will always print "[]". Negative values will result in all values
| being printed. A value of 0 will always "[…]" for lists with contents. A value
| of 1 will print only the final item in the list.
|
| Parameters
| ----------
| n : int
| Number of values to display.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "nums": [
| ... [1, 2, 3, 4, 5, 6],
| ... ]
| ... }
| ... )
| >>> df
| shape: (1, 1)
| ┌─────────────┐
| │ nums │
| │ --- │
| │ list[i64] │
| ╞═════════════╡
| │ [1, 2, … 6] │
| └─────────────┘
| >>> with pl.Config(fmt_table_cell_list_len=10):
| ... print(df)
| shape: (1, 1)
| ┌────────────────────┐
| │ nums │
| │ --- │
| │ list[i64] │
| ╞════════════════════╡
| │ [1, 2, 3, 4, 5, 6] │
| └────────────────────┘
|
| set_streaming_chunk_size(size: 'int | None') -> 'type[Config]'
| Overwrite chunk size used in `streaming` engine.
|
| By default, the chunk size is determined by the schema
| and size of the thread pool. For some datasets (esp.
| when you have large string elements) this can be too
| optimistic and lead to Out of Memory errors.
|
| Parameters
| ----------
| size
| Number of rows per chunk. Every thread will process chunks
| of this size.
|
| set_tbl_cell_alignment(format: 'Alignment | None') -> 'type[Config]'
| Set table cell alignment.
|
| Parameters
| ----------
| format : str
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"column_abc": [1.0, 2.5, 5.0], "column_xyz": [True, False, True]}
| ... )
| >>> pl.Config.set_tbl_cell_alignment("RIGHT") # doctest: +IGNORE_RESULT
| >>> print(df)
| shape: (3, 2)
| ┌────────────┬────────────┐
| │ column_abc ┆ column_xyz │
| │ --- ┆ --- │
| │ f64 ┆ bool │
| ╞════════════╪════════════╡
| │ 1.0 ┆ true │
| │ 2.5 ┆ false │
| │ 5.0 ┆ true │
| └────────────┴────────────┘
|
| Raises
| ------
| ValueError: if alignment string not recognised.
|
| set_tbl_cell_numeric_alignment(format: 'Alignment | None') -> 'type[Config]'
| Set table cell alignment for numeric columns.
|
| Parameters
| ----------
| format : str
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| Examples
| --------
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "abc": [11, 2, 333],
| ... "mno": [date(2023, 10, 29), None, date(2001, 7, 5)],
| ... "xyz": [True, False, None],
| ... }
| ... )
| >>> pl.Config.set_tbl_cell_numeric_alignment("RIGHT") # doctest: +IGNORE_RESULT
| >>> print(df)
| shape: (3, 3)
| ┌─────┬────────────┬───────┐
| │ abc ┆ mno ┆ xyz │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ date ┆ bool │
| ╞═════╪════════════╪═══════╡
| │ 11 ┆ 2023-10-29 ┆ true │
| │ 2 ┆ null ┆ false │
| │ 333 ┆ 2001-07-05 ┆ null │
| └─────┴────────────┴───────┘
|
| Raises
| ------
| KeyError: if alignment string not recognised.
|
| set_tbl_cols(n: 'int | None') -> 'type[Config]'
| Set the number of columns that are visible when displaying tables.
|
| Parameters
| ----------
| n : int
| Number of columns to display; if `n < 0` (eg: -1), display all columns.
|
| Examples
| --------
| Set number of displayed columns to a low value:
|
| >>> with pl.Config() as cfg:
| ... cfg.set_tbl_cols(5)
| ... df = pl.DataFrame({str(i): [i] for i in range(100)})
| ... print(df)
| <class 'polars.config.Config'>
| shape: (1, 100)
| ┌─────┬─────┬─────┬───┬─────┬─────┐
| │ 0 ┆ 1 ┆ 2 ┆ … ┆ 98 ┆ 99 │
| │ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 ┆ … ┆ 98 ┆ 99 │
| └─────┴─────┴─────┴───┴─────┴─────┘
|
| >>> with pl.Config(tbl_cols=10):
| ... print(df)
| shape: (1, 100)
| ┌─────┬─────┬─────┬─────┬─────┬───┬─────┬─────┬─────┬─────┬─────┐
| │ 0 ┆ 1 ┆ 2 ┆ 3 ┆ 4 ┆ … ┆ 95 ┆ 96 ┆ 97 ┆ 98 ┆ 99 │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╪═════╪═══╪═════╪═════╪═════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 ┆ 3 ┆ 4 ┆ … ┆ 95 ┆ 96 ┆ 97 ┆ 98 ┆ 99 │
| └─────┴─────┴─────┴─────┴─────┴───┴─────┴─────┴─────┴─────┴─────┘
|
| set_tbl_column_data_type_inline(active: 'bool | None' = True) -> 'type[Config]'
| Display the data type next to the column name (to the right, in parentheses).
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_column_data_type_inline(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ ┌───────────┬────────────┐
| # │ abc ┆ xyz │ │ abc (f64) ┆ xyz (bool) │
| # │ --- ┆ --- │ ╞═══════════╪════════════╡
| # │ f64 ┆ bool │ │ 1.0 ┆ true │
| # ╞═════╪═══════╡ >> │ 2.5 ┆ false │
| # │ 1.0 ┆ true │ │ 5.0 ┆ true │
| # │ 2.5 ┆ false │ └───────────┴────────────┘
| # │ 5.0 ┆ true │
| # └─────┴───────┘
|
| set_tbl_dataframe_shape_below(active: 'bool | None' = True) -> 'type[Config]'
| Print the DataFrame shape information below the data when displaying tables.
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_dataframe_shape_below(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) ┌─────┬───────┐
| # ┌─────┬───────┐ │ abc ┆ xyz │
| # │ abc ┆ xyz │ │ --- ┆ --- │
| # │ --- ┆ --- │ │ f64 ┆ bool │
| # │ f64 ┆ bool │ ╞═════╪═══════╡
| # ╞═════╪═══════╡ >> │ 1.0 ┆ true │
| # │ 1.0 ┆ true │ │ 2.5 ┆ false │
| # │ 2.5 ┆ false │ │ 5.0 ┆ true │
| # │ 5.0 ┆ true │ └─────┴───────┘
| # └─────┴───────┘ shape: (3, 2)
|
| set_tbl_formatting(format: 'TableFormatNames | None' = None, rounded_corners: 'bool | None' = False) -> 'type[Config]'
| Set table formatting style.
|
| Parameters
| ----------
| format : str
| * "ASCII_FULL": ASCII, with all borders and lines, including row dividers.
| * "ASCII_FULL_CONDENSED": Same as ASCII_FULL, but with dense row spacing.
| * "ASCII_NO_BORDERS": ASCII, no borders.
| * "ASCII_BORDERS_ONLY": ASCII, borders only.
| * "ASCII_BORDERS_ONLY_CONDENSED": ASCII, borders only, dense row spacing.
| * "ASCII_HORIZONTAL_ONLY": ASCII, horizontal lines only.
| * "ASCII_MARKDOWN": Markdown format (ascii ellipses for truncated values).
| * "MARKDOWN": Markdown format (utf8 ellipses for truncated values).
| * "UTF8_FULL": UTF8, with all borders and lines, including row dividers.
| * "UTF8_FULL_CONDENSED": Same as UTF8_FULL, but with dense row spacing.
| * "UTF8_NO_BORDERS": UTF8, no borders.
| * "UTF8_BORDERS_ONLY": UTF8, borders only.
| * "UTF8_HORIZONTAL_ONLY": UTF8, horizontal lines only.
| * "NOTHING": No borders or other lines.
|
| rounded_corners : bool
| Apply rounded corners to UTF8-styled tables (no-op for ASCII formats).
|
| Notes
| -----
| The UTF8 styles all use one or more of the semigraphic box-drawing characters
| found in the Unicode Box Drawing block, which are not ASCII compatible:
| https://en.wikipedia.org/wiki/Box-drawing_character#Box_Drawing
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"abc": [-2.5, 5.0], "mno": ["hello", "world"], "xyz": [True, False]}
| ... )
| >>> with pl.Config(
| ... tbl_formatting="MARKDOWN",
| ... tbl_hide_column_data_types=True,
| ... tbl_hide_dataframe_shape=True,
| ... ):
| ... print(df)
| | abc | mno | xyz |
| |------|-------|-------|
| | -2.5 | hello | true |
| | 5.0 | world | false |
|
| Raises
| ------
| ValueError: if format string not recognised.
|
| set_tbl_hide_column_data_types(active: 'bool | None' = True) -> 'type[Config]'
| Hide table column data types (i64, f64, str etc.).
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_hide_column_data_types(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ ┌─────┬───────┐
| # │ abc ┆ xyz │ │ abc ┆ xyz │
| # │ --- ┆ --- │ ╞═════╪═══════╡
| # │ f64 ┆ bool │ │ 1.0 ┆ true │
| # ╞═════╪═══════╡ >> │ 2.5 ┆ false │
| # │ 1.0 ┆ true │ │ 5.0 ┆ true │
| # │ 2.5 ┆ false │ └─────┴───────┘
| # │ 5.0 ┆ true │
| # └─────┴───────┘
|
| set_tbl_hide_column_names(active: 'bool | None' = True) -> 'type[Config]'
| Hide table column names.
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_hide_column_names(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ ┌─────┬───────┐
| # │ abc ┆ xyz │ │ f64 ┆ bool │
| # │ --- ┆ --- │ ╞═════╪═══════╡
| # │ f64 ┆ bool │ │ 1.0 ┆ true │
| # ╞═════╪═══════╡ >> │ 2.5 ┆ false │
| # │ 1.0 ┆ true │ │ 5.0 ┆ true │
| # │ 2.5 ┆ false │ └─────┴───────┘
| # │ 5.0 ┆ true │
| # └─────┴───────┘
|
| set_tbl_hide_dataframe_shape(active: 'bool | None' = True) -> 'type[Config]'
| Hide the DataFrame shape information when displaying tables.
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_hide_dataframe_shape(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) ┌─────┬───────┐
| # ┌─────┬───────┐ │ abc ┆ xyz │
| # │ abc ┆ xyz │ │ --- ┆ --- │
| # │ --- ┆ --- │ │ f64 ┆ bool │
| # │ f64 ┆ bool │ ╞═════╪═══════╡
| # ╞═════╪═══════╡ │ 1.0 ┆ true │
| # │ 1.0 ┆ true │ >> │ 2.5 ┆ false │
| # │ 2.5 ┆ false │ │ 5.0 ┆ true │
| # │ 5.0 ┆ true │ └─────┴───────┘
| # └─────┴───────┘
|
| set_tbl_hide_dtype_separator(active: 'bool | None' = True) -> 'type[Config]'
| Hide the '---' separator displayed between the column names and column types.
|
| See Also
| --------
| set_tbl_column_data_type_inline : Display the data type inline with the colname.
|
| Examples
| --------
| >>> df = pl.DataFrame({"abc": [1.0, 2.5, 5.0], "xyz": [True, False, True]})
| >>> pl.Config.set_tbl_hide_dtype_separator(True) # doctest: +SKIP
| # ...
| # shape: (3, 2) shape: (3, 2)
| # ┌─────┬───────┐ ┌─────┬───────┐
| # │ abc ┆ xyz │ │ abc ┆ xyz │
| # │ --- ┆ --- │ │ f64 ┆ bool │
| # │ f64 ┆ bool │ ╞═════╪═══════╡
| # ╞═════╪═══════╡ │ 1.0 ┆ true │
| # │ 1.0 ┆ true │ >> │ 2.5 ┆ false │
| # │ 2.5 ┆ false │ │ 5.0 ┆ true │
| # │ 5.0 ┆ true │ └─────┴───────┘
| # └─────┴───────┘
|
| set_tbl_rows(n: 'int | None') -> 'type[Config]'
| Set the max number of rows used to draw the table (both Dataframe and Series).
|
| Parameters
| ----------
| n : int
| Number of rows to display; if `n < 0` (eg: -1), display all
| rows (DataFrame) and all elements (Series).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"abc": [1.0, 2.5, 3.5, 5.0], "xyz": [True, False, True, False]}
| ... )
| >>> with pl.Config(tbl_rows=2):
| ... print(df)
| shape: (4, 2)
| ┌─────┬───────┐
| │ abc ┆ xyz │
| │ --- ┆ --- │
| │ f64 ┆ bool │
| ╞═════╪═══════╡
| │ 1.0 ┆ true │
| │ … ┆ … │
| │ 5.0 ┆ false │
| └─────┴───────┘
|
| set_tbl_width_chars(width: 'int | None') -> 'type[Config]'
| Set the maximum width of a table in characters.
|
| Parameters
| ----------
| width : int
| Maximum table width in characters; if n < 0 (eg: -1), display full width.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "id": ["SEQ1", "SEQ2"],
| ... "seq": ["ATGATAAAGGAG", "GCAACGCATATA"],
| ... }
| ... )
| >>> df
| shape: (2, 2)
| ┌──────┬──────────────┐
| │ id ┆ seq │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞══════╪══════════════╡
| │ SEQ1 ┆ ATGATAAAGGAG │
| │ SEQ2 ┆ GCAACGCATATA │
| └──────┴──────────────┘
| >>> pl.Config.set_tbl_width_chars(12) # doctest: +IGNORE_RESULT
| >>> df
| shape: (2, 2)
| ┌─────┬─────┐
| │ id ┆ seq │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞═════╪═════╡
| │ SEQ ┆ ATG │
| │ 1 ┆ ATA │
| │ ┆ AAG │
| │ ┆ GAG │
| │ SEQ ┆ GCA │
| │ 2 ┆ ACG │
| │ ┆ CAT │
| │ ┆ ATA │
| └─────┴─────┘
|
| set_thousands_separator(separator: 'str | bool | None' = None) -> 'type[Config]'
| Set the thousands grouping separator character.
|
| Parameters
| ----------
| separator : str, bool
| Set True to use the default "," (thousands) and "." (decimal) separators.
| Can also set a custom char, or set ``None`` to omit the separator.
|
| See Also
| --------
| set_decimal_separator : Set the decimal separator character.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "x": [1234567, -987654, 10101],
| ... "y": [1234.5, 100000.0, -7654321.25],
| ... }
| ... )
| >>> with pl.Config(
| ... tbl_cell_numeric_alignment="RIGHT",
| ... thousands_separator=True,
| ... float_precision=2,
| ... ):
| ... print(df)
| shape: (3, 2)
| ┌───────────┬───────────────┐
| │ x ┆ y │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═══════════╪═══════════════╡
| │ 1,234,567 ┆ 1,234.50 │
| │ -987,654 ┆ 100,000.00 │
| │ 10,101 ┆ -7,654,321.25 │
| └───────────┴───────────────┘
| >>> with pl.Config(
| ... tbl_cell_numeric_alignment="RIGHT",
| ... thousands_separator=".",
| ... decimal_separator=",",
| ... float_precision=2,
| ... ):
| ... print(df)
| shape: (3, 2)
| ┌───────────┬───────────────┐
| │ x ┆ y │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═══════════╪═══════════════╡
| │ 1.234.567 ┆ 1.234,50 │
| │ -987.654 ┆ 100.000,00 │
| │ 10.101 ┆ -7.654.321,25 │
| └───────────┴───────────────┘
|
| set_trim_decimal_zeros(active: 'bool | None' = True) -> 'type[Config]'
| Strip trailing zeros from Decimal data type values.
|
| Parameters
| ----------
| active : bool
| Enable stripping of trailing '0' characters from Decimal values.
|
| Examples
| --------
| >>> from decimal import Decimal as D
| >>> df = pl.DataFrame(
| ... data={"d": [D("1.01000"), D("-5.67890")]},
| ... schema={"d": pl.Decimal(scale=5)},
| ... )
| >>> with pl.Config(trim_decimal_zeros=False):
| ... print(df)
| shape: (2, 1)
| ┌───────────────┐
| │ d │
| │ --- │
| │ decimal[38,5] │
| ╞═══════════════╡
| │ 1.01000 │
| │ -5.67890 │
| └───────────────┘
| >>> with pl.Config(trim_decimal_zeros=True):
| ... print(df)
| shape: (2, 1)
| ┌───────────────┐
| │ d │
| │ --- │
| │ decimal[38,5] │
| ╞═══════════════╡
| │ 1.01 │
| │ -5.6789 │
| └───────────────┘
|
| set_verbose(active: 'bool | None' = True) -> 'type[Config]'
| Enable additional verbose/debug logging.
|
| Examples
| --------
| >>> pl.Config.set_verbose(True) # doctest: +SKIP
| >>> with pl.Config(verbose=True): # doctest: +SKIP
| ... do_polars_operations()
|
| state(*, if_set: 'bool' = False, env_only: 'bool' = False) -> 'dict[str, str | None]'
| Show the current state of all Config variables in the environment as a dict.
|
| Parameters
| ----------
| if_set
| By default this will show the state of all `Config` environment variables.
| change this to `True` to restrict the returned dictionary to include only
| those that have been set to a specific value.
| env_only
| Include only Config environment variables in the output; some options (such
| as "set_fmt_float") are set directly, not via an environment variable.
|
| Examples
| --------
| >>> set_state = pl.Config.state(if_set=True)
| >>> all_state = pl.Config.state()
|
| warn_unstable(active: 'bool | None' = True) -> 'type[Config]'
| Issue a warning when unstable functionality is used.
|
| Enabling this setting may help avoid functionality that is still evolving,
| potentially reducing maintenance burden from API changes and bugs.
|
| Examples
| --------
| >>> pl.Config.warn_unstable(True) # doctest: +SKIP
| >>> pl.col("a").qcut(5) # doctest: +SKIP
| UnstableWarning: `qcut` is considered unstable. It may be changed at any point without it being considered a breaking change.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_context_options': 'ConfigParameters | None', '_or...
|
| __hash__ = None
|
| ----------------------------------------------------------------------
| Methods inherited from contextlib.ContextDecorator:
|
| __call__(self, func)
| Call self as a function.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from contextlib.ContextDecorator:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
DATA
__all__ = ['Config']
__annotations__ = {'TableFormatNames': 'TypeAlias', '_POLARS_CFG_ENV_V...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\config.py
polars.contextlib
Python Library Documentation: module contextlib
NAME
contextlib - Utilities for with-statement contexts. See PEP 343.
MODULE REFERENCE
https://docs.python.org/3.12/library/contextlib.html
The following documentation is automatically generated from the Python
source files. It may be incomplete, incorrect or include features that
are considered implementation detail and may vary between Python
implementations. When in doubt, consult the module reference at the
location listed above.
CLASSES
abc.ABC(builtins.object)
AbstractAsyncContextManager
AsyncExitStack(_BaseExitStack, AbstractAsyncContextManager)
aclosing
AbstractContextManager
ExitStack(_BaseExitStack, AbstractContextManager)
chdir
closing
nullcontext(AbstractContextManager, AbstractAsyncContextManager)
suppress
builtins.object
ContextDecorator
_BaseExitStack(builtins.object)
AsyncExitStack(_BaseExitStack, AbstractAsyncContextManager)
ExitStack(_BaseExitStack, AbstractContextManager)
_RedirectStream(AbstractContextManager)
redirect_stderr
redirect_stdout
class AbstractAsyncContextManager(abc.ABC)
| An abstract base class for asynchronous context managers.
|
| Method resolution order:
| AbstractAsyncContextManager
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| async __aenter__(self)
| Return `self` upon entering the runtime context.
|
| async __aexit__(self, exc_type, exc_value, traceback)
| Raise any exception triggered within the runtime context.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'__aexit__'})
class AbstractContextManager(abc.ABC)
| An abstract base class for context managers.
|
| Method resolution order:
| AbstractContextManager
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
| Return `self` upon entering the runtime context.
|
| __exit__(self, exc_type, exc_value, traceback)
| Raise any exception triggered within the runtime context.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'__exit__'})
class AsyncExitStack(_BaseExitStack, AbstractAsyncContextManager)
| Async context manager for dynamic management of a stack of exit
| callbacks.
|
| For example:
| async with AsyncExitStack() as stack:
| connections = [await stack.enter_async_context(get_connection())
| for i in range(5)]
| # All opened connections will automatically be released at the
| # end of the async with statement, even if attempts to open a
| # connection later in the list raise an exception.
|
| Method resolution order:
| AsyncExitStack
| _BaseExitStack
| AbstractAsyncContextManager
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| async __aenter__(self)
| Return `self` upon entering the runtime context.
|
| async __aexit__(self, *exc_details)
| Raise any exception triggered within the runtime context.
|
| async aclose(self)
| Immediately unwind the context stack.
|
| async enter_async_context(self, cm)
| Enters the supplied async context manager.
|
| If successful, also pushes its __aexit__ method as a callback and
| returns the result of the __aenter__ method.
|
| push_async_callback(self, callback, /, *args, **kwds)
| Registers an arbitrary coroutine function and arguments.
|
| Cannot suppress exceptions.
|
| push_async_exit(self, exit)
| Registers a coroutine function with the standard __aexit__ method
| signature.
|
| Can suppress exceptions the same way __aexit__ method can.
| Also accepts any object with an __aexit__ method (registering a call
| to the method instead of the object itself).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExitStack:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| callback(self, callback, /, *args, **kwds)
| Registers an arbitrary callback and arguments.
|
| Cannot suppress exceptions.
|
| enter_context(self, cm)
| Enters the supplied context manager.
|
| If successful, also pushes its __exit__ method as a callback and
| returns the result of the __enter__ method.
|
| pop_all(self)
| Preserve the context stack by transferring it to a new instance.
|
| push(self, exit)
| Registers a callback with the standard __exit__ method signature.
|
| Can suppress exceptions the same way __exit__ method can.
| Also accepts any object with an __exit__ method (registering a call
| to the method instead of the object itself).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExitStack:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Class methods inherited from AbstractAsyncContextManager:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
class ContextDecorator(builtins.object)
| A base class or mixin that enables context managers to work as decorators.
|
| Methods defined here:
|
| __call__(self, func)
| Call self as a function.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
class ExitStack(_BaseExitStack, AbstractContextManager)
| Context manager for dynamic management of a stack of exit callbacks.
|
| For example:
| with ExitStack() as stack:
| files = [stack.enter_context(open(fname)) for fname in filenames]
| # All opened files will automatically be closed at the end of
| # the with statement, even if attempts to open files later
| # in the list raise an exception.
|
| Method resolution order:
| ExitStack
| _BaseExitStack
| AbstractContextManager
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
| Return `self` upon entering the runtime context.
|
| __exit__(self, *exc_details)
| Raise any exception triggered within the runtime context.
|
| close(self)
| Immediately unwind the context stack.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExitStack:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| callback(self, callback, /, *args, **kwds)
| Registers an arbitrary callback and arguments.
|
| Cannot suppress exceptions.
|
| enter_context(self, cm)
| Enters the supplied context manager.
|
| If successful, also pushes its __exit__ method as a callback and
| returns the result of the __enter__ method.
|
| pop_all(self)
| Preserve the context stack by transferring it to a new instance.
|
| push(self, exit)
| Registers a callback with the standard __exit__ method signature.
|
| Can suppress exceptions the same way __exit__ method can.
| Also accepts any object with an __exit__ method (registering a call
| to the method instead of the object itself).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExitStack:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Class methods inherited from AbstractContextManager:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
class aclosing(AbstractAsyncContextManager)
| aclosing(thing)
|
| Async context manager for safely finalizing an asynchronously cleaned-up
| resource such as an async generator, calling its ``aclose()`` method.
|
| Code like this:
|
| async with aclosing(<module>.fetch(<arguments>)) as agen:
| <block>
|
| is equivalent to this:
|
| agen = <module>.fetch(<arguments>)
| try:
| <block>
| finally:
| await agen.aclose()
|
| Method resolution order:
| aclosing
| AbstractAsyncContextManager
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| async __aenter__(self)
| Return `self` upon entering the runtime context.
|
| async __aexit__(self, *exc_info)
| Raise any exception triggered within the runtime context.
|
| __init__(self, thing)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Class methods inherited from AbstractAsyncContextManager:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from AbstractAsyncContextManager:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class chdir(AbstractContextManager)
| chdir(path)
|
| Non thread-safe context manager to change the current working directory.
|
| Method resolution order:
| chdir
| AbstractContextManager
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
| Return `self` upon entering the runtime context.
|
| __exit__(self, *excinfo)
| Raise any exception triggered within the runtime context.
|
| __init__(self, path)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Class methods inherited from AbstractContextManager:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from AbstractContextManager:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class closing(AbstractContextManager)
| closing(thing)
|
| Context to automatically close something at the end of a block.
|
| Code like this:
|
| with closing(<module>.open(<arguments>)) as f:
| <block>
|
| is equivalent to this:
|
| f = <module>.open(<arguments>)
| try:
| <block>
| finally:
| f.close()
|
| Method resolution order:
| closing
| AbstractContextManager
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
| Return `self` upon entering the runtime context.
|
| __exit__(self, *exc_info)
| Raise any exception triggered within the runtime context.
|
| __init__(self, thing)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Class methods inherited from AbstractContextManager:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from AbstractContextManager:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class nullcontext(AbstractContextManager, AbstractAsyncContextManager)
| nullcontext(enter_result=None)
|
| Context manager that does no additional processing.
|
| Used as a stand-in for a normal context manager, when a particular
| block of code is only sometimes used with a normal context manager:
|
| cm = optional_cm if condition else nullcontext()
| with cm:
| # Perform operation, using optional_cm if condition is True
|
| Method resolution order:
| nullcontext
| AbstractContextManager
| AbstractAsyncContextManager
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| async __aenter__(self)
| Return `self` upon entering the runtime context.
|
| async __aexit__(self, *excinfo)
| Raise any exception triggered within the runtime context.
|
| __enter__(self)
| Return `self` upon entering the runtime context.
|
| __exit__(self, *excinfo)
| Raise any exception triggered within the runtime context.
|
| __init__(self, enter_result=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Class methods inherited from AbstractContextManager:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from AbstractContextManager:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class redirect_stderr(_RedirectStream)
| redirect_stderr(new_target)
|
| Context manager for temporarily redirecting stderr to another file.
|
| Method resolution order:
| redirect_stderr
| _RedirectStream
| AbstractContextManager
| abc.ABC
| builtins.object
|
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Methods inherited from _RedirectStream:
|
| __enter__(self)
| Return `self` upon entering the runtime context.
|
| __exit__(self, exctype, excinst, exctb)
| Raise any exception triggered within the runtime context.
|
| __init__(self, new_target)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Class methods inherited from AbstractContextManager:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from AbstractContextManager:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class redirect_stdout(_RedirectStream)
| redirect_stdout(new_target)
|
| Context manager for temporarily redirecting stdout to another file.
|
| # How to send help() to stderr
| with redirect_stdout(sys.stderr):
| help(dir)
|
| # How to write help() to a file
| with open('help.txt', 'w') as f:
| with redirect_stdout(f):
| help(pow)
|
| Method resolution order:
| redirect_stdout
| _RedirectStream
| AbstractContextManager
| abc.ABC
| builtins.object
|
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Methods inherited from _RedirectStream:
|
| __enter__(self)
| Return `self` upon entering the runtime context.
|
| __exit__(self, exctype, excinst, exctb)
| Raise any exception triggered within the runtime context.
|
| __init__(self, new_target)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Class methods inherited from AbstractContextManager:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from AbstractContextManager:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class suppress(AbstractContextManager)
| suppress(*exceptions)
|
| Context manager to suppress specified exceptions
|
| After the exception is suppressed, execution proceeds with the next
| statement following the with statement.
|
| with suppress(FileNotFoundError):
| os.remove(somefile)
| # Execution still resumes here if the file was already removed
|
| Method resolution order:
| suppress
| AbstractContextManager
| abc.ABC
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
| Return `self` upon entering the runtime context.
|
| __exit__(self, exctype, excinst, exctb)
| Raise any exception triggered within the runtime context.
|
| __init__(self, *exceptions)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Class methods inherited from AbstractContextManager:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from AbstractContextManager:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
asynccontextmanager(func)
@asynccontextmanager decorator.
Typical usage:
@asynccontextmanager
async def some_async_generator(<arguments>):
<setup>
try:
yield <value>
finally:
<cleanup>
This makes this:
async with some_async_generator(<arguments>) as <variable>:
<body>
equivalent to this:
<setup>
try:
<variable> = <value>
<body>
finally:
<cleanup>
contextmanager(func)
@contextmanager decorator.
Typical usage:
@contextmanager
def some_generator(<arguments>):
<setup>
try:
yield <value>
finally:
<cleanup>
This makes this:
with some_generator(<arguments>) as <variable>:
<body>
equivalent to this:
<setup>
try:
<variable> = <value>
<body>
finally:
<cleanup>
DATA
__all__ = ['asynccontextmanager', 'contextmanager', 'closing', 'nullco...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\contextlib.py
polars.convert
Python Library Documentation: package polars.convert in polars
NAME
polars.convert
PACKAGE CONTENTS
general
normalize
FUNCTIONS
from_arrow(data: 'pa.Table | pa.Array | pa.ChunkedArray | pa.RecordBatch | Iterable[pa.RecordBatch | pa.Table] | ArrowArrayExportable | ArrowStreamExportable', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, rechunk: 'bool' = True) -> 'DataFrame | Series'
Create a DataFrame or Series from an Arrow Table or Array.
This operation will be zero copy for the most part. Types that are not
supported by Polars may be cast to the closest supported type.
Hint: You can also directly pass arrow tables to `pl.DataFrame()` / arrow
arrays to `pl.Series()` if the output type is known to avoid typing issues.
Parameters
----------
data : :class:`pyarrow.Table`, :class:`pyarrow.Array`, one or more :class:`pyarrow.RecordBatch`
Data representing an Arrow Table, Array, sequence of RecordBatches or Tables, or other
object that supports the Arrow PyCapsule interface.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
Schema entries are applied positionally, following the order of the
Python dictionary. The provided schema takes precedence over the schema
of the underlying Arrow data. As such, if the provided schema names do
not match the underlying Arrow data, the column names of the Arrow data
will be discarded in favor of the names set in this argument.
The number of names given in the schema must match the underlying data
dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the schema param will be overridden.
rechunk : bool, default True
Make sure that all data is in contiguous memory.
Returns
-------
DataFrame or Series
Examples
--------
Constructing a DataFrame from an Arrow Table:
>>> import pyarrow as pa
>>> data = pa.table({"a": [1, 2, 3], "b": [4, 5, 6]})
>>> pl.from_arrow(data)
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 4 │
│ 2 ┆ 5 │
│ 3 ┆ 6 │
└─────┴─────┘
Constructing a Series from an Arrow Array:
>>> import pyarrow as pa
>>> data = pa.array([1, 2, 3])
>>> pl.from_arrow(data, schema={"s": pl.Int32})
shape: (3,)
Series: 's' [i32]
[
1
2
3
]
from_dataframe(df: 'SupportsInterchange | ArrowArrayExportable | ArrowStreamExportable', *, allow_copy: 'bool | None' = None, rechunk: 'bool' = True) -> 'DataFrame'
Build a Polars DataFrame from any dataframe supporting the PyCapsule Interface.
.. versionchanged:: 1.23.0
`from_dataframe` uses the PyCapsule Interface instead of the Dataframe
Interchange Protocol for conversion, only using the latter as a fallback.
Parameters
----------
df
Object supporting the dataframe PyCapsule Interface.
allow_copy
Allow memory to be copied to perform the conversion. If set to False, may cause
conversions that are not zero-copy to fail.
.. deprecated: 1.23.0
`allow_copy` is deprecated and will be removed in a future version.
rechunk : bool, default True
Make sure that all data is in contiguous memory.
Notes
-----
- Details on the PyCapsule Interface:
https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html.
- Details on the Python dataframe interchange protocol:
https://data-apis.org/dataframe-protocol/latest/index.html.
Using a dedicated function like :func:`from_pandas` or :func:`from_arrow` is
a more efficient method of conversion.
Examples
--------
Convert a pandas dataframe to Polars.
>>> import pandas as pd
>>> df_pd = pd.DataFrame({"a": [1, 2], "b": [3.0, 4.0], "c": ["x", "y"]})
>>> pl.from_dataframe(df_pd)
shape: (2, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ f64 ┆ str │
╞═════╪═════╪═════╡
│ 1 ┆ 3.0 ┆ x │
│ 2 ┆ 4.0 ┆ y │
└─────┴─────┴─────┘
from_dict(data: 'Mapping[str, Sequence[object] | Mapping[str, Sequence[object]] | Series]', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True) -> 'DataFrame'
Construct a DataFrame from a dictionary of sequences.
This operation clones data, unless you pass a `{str: pl.Series,}` dict.
Parameters
----------
data : dict of sequences
Two-dimensional data represented as a dictionary. dict must contain
Sequences.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the columns param will be overridden.
strict : bool, default True
Throw an error if any `data` value does not exactly match the given or inferred
data type for that column. If set to `False`, values that do not match the data
type are cast to that data type or, if casting is not possible, set to null
instead.
Returns
-------
DataFrame
Examples
--------
>>> df = pl.from_dict({"a": [1, 2], "b": [3, 4]})
>>> df
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 3 │
│ 2 ┆ 4 │
└─────┴─────┘
from_dicts(data: 'Iterable[Mapping[str, Any]]', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, infer_schema_length: 'int | None' = 100) -> 'DataFrame'
Construct a DataFrame from a sequence of dictionaries. This operation clones data.
Parameters
----------
data
Sequence with dictionaries mapping column name to value
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If a list of column names is supplied that does NOT match the names in the
underlying data, the names given here will overwrite the actual fields in
the order that they appear - however, in this case it is typically clearer
to rename after loading the frame.
If you want to drop some of the fields found in the input dictionaries, a
*partial* schema can be declared, in which case omitted fields will not be
loaded. Similarly, you can extend the loaded frame with empty columns by
adding them to the schema.
schema_overrides : dict, default None
Support override of inferred types for one or more columns.
strict : bool, default True
Throw an error if any `data` value does not exactly match the given or inferred
data type for that column. If set to `False`, values that do not match the data
type are cast to that data type or, if casting is not possible, set to null
instead.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
Returns
-------
DataFrame
Examples
--------
>>> data = [{"a": 1, "b": 4}, {"a": 2, "b": 5}, {"a": 3, "b": 6}]
>>> df = pl.from_dicts(data)
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 4 │
│ 2 ┆ 5 │
│ 3 ┆ 6 │
└─────┴─────┘
Declaring a partial `schema` will drop the omitted columns.
>>> df = pl.from_dicts(data, schema={"a": pl.Int32})
>>> df
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i32 │
╞═════╡
│ 1 │
│ 2 │
│ 3 │
└─────┘
Can also use the `schema` param to extend the loaded columns with one
or more additional (empty) columns that are not present in the input dicts:
>>> pl.from_dicts(
... data,
... schema=["a", "b", "c", "d"],
... schema_overrides={"c": pl.Float64, "d": pl.String},
... )
shape: (3, 4)
┌─────┬─────┬──────┬──────┐
│ a ┆ b ┆ c ┆ d │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ f64 ┆ str │
╞═════╪═════╪══════╪══════╡
│ 1 ┆ 4 ┆ null ┆ null │
│ 2 ┆ 5 ┆ null ┆ null │
│ 3 ┆ 6 ┆ null ┆ null │
└─────┴─────┴──────┴──────┘
from_numpy(data: 'np.ndarray[Any, Any]', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, orient: 'Orientation | None' = None) -> 'DataFrame'
Construct a DataFrame from a NumPy ndarray. This operation clones data.
Note that this is slower than creating from columnar memory.
Parameters
----------
data : :class:`numpy.ndarray`
Two-dimensional data represented as a NumPy ndarray.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the columns param will be overridden.
orient : {None, 'col', 'row'}
Whether to interpret two-dimensional data as columns or as rows. If None,
the orientation is inferred by matching the columns and data dimensions. If
this does not yield conclusive results, column orientation is used.
Returns
-------
DataFrame
Examples
--------
>>> import numpy as np
>>> data = np.array([[1, 2, 3], [4, 5, 6]])
>>> df = pl.from_numpy(data, schema=["a", "b"], orient="col")
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 4 │
│ 2 ┆ 5 │
│ 3 ┆ 6 │
└─────┴─────┘
from_pandas(data: 'pd.DataFrame | pd.Series[Any] | pd.Index[Any] | pd.DatetimeIndex', *, schema_overrides: 'SchemaDict | None' = None, rechunk: 'bool' = True, nan_to_null: 'bool' = True, include_index: 'bool' = False) -> 'DataFrame | Series'
Construct a Polars DataFrame or Series from a pandas DataFrame, Series, or Index.
This operation may clone data. If you want to ensure that in-place modifications
of the output don't affect the input, you may want to consider one of the following:
- Enable `Copy-On-Write <https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html>`_
in pandas.
- Call :meth:`DataFrame.clone` on the output of `from_pandas`.
This requires that :mod:`pandas` and :mod:`pyarrow` are installed.
Parameters
----------
data : :class:`pandas.DataFrame` or :class:`pandas.Series` or :class:`pandas.Index`
Data represented as a pandas DataFrame, Series, or Index.
schema_overrides : dict, default None
Support override of inferred types for one or more columns.
rechunk : bool, default True
Make sure that all data is in contiguous memory.
nan_to_null : bool, default True
If data contains `NaN` values PyArrow will convert the `NaN` to `None`
include_index : bool, default False
Load any non-default pandas indexes as columns.
.. note::
If the input is a pandas ``DataFrame`` and has a nameless index
which just enumerates the rows, then it will not be included in the
result, regardless of this parameter. If you want to be sure to include it,
please call ``.reset_index()`` prior to calling this function.
Returns
-------
DataFrame
Examples
--------
Constructing a :class:`DataFrame` from a :class:`pandas.DataFrame`:
>>> import pandas as pd
>>> pd_df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns=["a", "b", "c"])
>>> df = pl.from_pandas(pd_df)
>>> df
shape: (2, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 2 ┆ 3 │
│ 4 ┆ 5 ┆ 6 │
└─────┴─────┴─────┘
Constructing a Series from a :class:`pandas.Series`:
>>> import pandas as pd
>>> pd_series = pd.Series([1, 2, 3], name="pd")
>>> df = pl.from_pandas(pd_series)
>>> df
shape: (3,)
Series: 'pd' [i64]
[
1
2
3
]
from_records(data: 'Sequence[Any]', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100) -> 'DataFrame'
Construct a DataFrame from a sequence of sequences. This operation clones data.
Note that this is slower than creating from columnar memory.
Parameters
----------
data : Sequence of sequences
Two-dimensional data represented as a sequence of sequences.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the columns param will be overridden.
strict : bool, default True
Throw an error if any `data` value does not exactly match the given or inferred
data type for that column. If set to `False`, values that do not match the data
type are cast to that data type or, if casting is not possible, set to null
instead.
orient : {None, 'col', 'row'}
Whether to interpret two-dimensional data as columns or as rows. If None,
the orientation is inferred by matching the columns and data dimensions. If
this does not yield conclusive results, column orientation is used.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
Returns
-------
DataFrame
Examples
--------
>>> data = [[1, 2, 3], [4, 5, 6]]
>>> df = pl.from_records(data, schema=["a", "b"])
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 4 │
│ 2 ┆ 5 │
│ 3 ┆ 6 │
└─────┴─────┘
from_repr(data: 'str') -> 'DataFrame | Series'
Construct a Polars DataFrame or Series from its string representation.
.. versionchanged:: 0.20.17
The `tbl` parameter was renamed to `data`.
Parameters
----------
data
A string containing a polars DataFrame or Series repr; does not need
to be trimmed of whitespace (or leading prompts) as the repr will be
found/extracted automatically.
Notes
-----
This function handles the UTF8_FULL (default) and UTF8_FULL_CONDENSED
DataFrame table formats (with or without rounded corners), as well as
ASCII_FULL and ASCII_FULL_CONDENSED. Truncated columns/rows are omitted,
wrapped headers are accounted for, and dtypes are automatically identified.
Currently compound/nested dtypes such as List and Struct are not supported;
neither are Object dtypes. The DuckDB table/relation repr is also compatible
with this function.
See Also
--------
polars.DataFrame.to_init_repr
polars.Series.to_init_repr
Examples
--------
From DataFrame table repr:
>>> df = pl.from_repr(
... '''
... Out[3]:
... shape: (1, 5)
... ┌───────────┬────────────┬───┬───────┬────────────────────────────────┐
... │ source_ac ┆ source_cha ┆ … ┆ ident ┆ timestamp │
... │ tor_id ┆ nnel_id ┆ ┆ --- ┆ --- │
... │ --- ┆ --- ┆ ┆ str ┆ datetime[μs, Asia/Tokyo] │
... │ i32 ┆ i64 ┆ ┆ ┆ │
... ╞═══════════╪════════════╪═══╪═══════╪════════════════════════════════╡
... │ 123456780 ┆ 9876543210 ┆ … ┆ a:b:c ┆ 2023-03-25 10:56:59.663053 JST │
... │ … ┆ … ┆ … ┆ … ┆ … │
... │ 803065983 ┆ 2055938745 ┆ … ┆ x:y:z ┆ 2023-03-25 12:38:18.050545 JST │
... └───────────┴────────────┴───┴───────┴────────────────────────────────┘
... '''
... )
>>> df
shape: (2, 4)
┌─────────────────┬───────────────────┬───────┬────────────────────────────────┐
│ source_actor_id ┆ source_channel_id ┆ ident ┆ timestamp │
│ --- ┆ --- ┆ --- ┆ --- │
│ i32 ┆ i64 ┆ str ┆ datetime[μs, Asia/Tokyo] │
╞═════════════════╪═══════════════════╪═══════╪════════════════════════════════╡
│ 123456780 ┆ 9876543210 ┆ a:b:c ┆ 2023-03-25 10:56:59.663053 JST │
│ 803065983 ┆ 2055938745 ┆ x:y:z ┆ 2023-03-25 12:38:18.050545 JST │
└─────────────────┴───────────────────┴───────┴────────────────────────────────┘
From Series repr:
>>> s = pl.from_repr(
... '''
... shape: (3,)
... Series: 's' [bool]
... [
... true
... false
... true
... ]
... '''
... )
>>> s.to_list()
[True, False, True]
from_torch(tensor: 'torch.Tensor', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, orient: 'Orientation | None' = None, force: 'bool' = False) -> 'DataFrame'
Construct a DataFrame from a PyTorch Tensor.
Parameters
----------
tensor : :class:`torch.Tensor`
A PyTorch `Tensor` object of one or more dimensions.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the columns param will be overridden.
orient : {None, 'col', 'row'}
Whether to interpret two-dimensional data as columns or as rows. If None,
the orientation is inferred by matching the columns and data dimensions. If
this does not yield conclusive results, column orientation is used.
force : bool
If False, the conversion is performed only if the Tensor is on CPU, does not
require grad, does not have its conjugate bit set, and is of a dtype (and
layout) that NumPy supports; this will typically be zero-copy. If True, it
is equivalent to calling `.detach().cpu().resolve_conj().resolve_neg()`
before passing the Tensor to Polars.
Returns
-------
DataFrame
Examples
--------
>>> import torch
>>> data = torch.tensor(
... [
... [1234.5, 200.0, 3000.5],
... [8000.0, 500.5, 6000.0],
... ]
... )
>>> df = pl.from_torch(
... data,
... schema=["colx", "coly", "colz"],
... schema_overrides={"colz": pl.Float64},
... )
>>> df
shape: (2, 3)
┌────────┬───────┬────────┐
│ colx ┆ coly ┆ colz │
│ --- ┆ --- ┆ --- │
│ f32 ┆ f32 ┆ f64 │
╞════════╪═══════╪════════╡
│ 1234.5 ┆ 200.0 ┆ 3000.5 │
│ 8000.0 ┆ 500.5 ┆ 6000.0 │
└────────┴───────┴────────┘
json_normalize(data: 'dict[Any, Any] | Sequence[dict[Any, Any] | Any]', *, separator: 'str' = '.', max_level: 'int | None' = None, schema: 'Schema | None' = None, strict: 'bool' = True, infer_schema_length: 'int | None' = 100, encoder: 'JSONEncoder | None' = None) -> 'DataFrame'
Normalize semi-structured deserialized JSON data into a flat table.
Dictionary objects that will not be unnested/normalized are encoded
as json string data. Unlike it pandas' counterpart, this function will
not encode dictionaries as objects at any level.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
data
Deserialized JSON objects.
separator
Nested records will generate names separated by sep. e.g.,
for `separator=".", {"foo": {"bar": 0}}` -> foo.bar.
max_level
Max number of levels(depth of dict) to normalize.
If None, normalizes all levels.
schema
Overwrite the `Schema` when the normalized data is passed to
the `DataFrame` constructor.
strict
Whether Polars should be strict when constructing the DataFrame.
infer_schema_length
Number of rows to take into consideration to determine the schema.
encoder
Custom JSON encoder function; if not given, `json.dumps` is used.
Examples
--------
>>> data = [
... {
... "id": 1,
... "name": "Cole Volk",
... "fitness": {"height": 180, "weight": 85},
... },
... {
... "id": 2,
... "name": "Faye Raker",
... "fitness": {"height": 155, "weight": 58},
... },
... {
... "name": "Mark Reg",
... "fitness": {"height": 170, "weight": 78},
... },
... ]
>>> pl.json_normalize(data, max_level=1)
shape: (3, 4)
┌──────┬────────────┬────────────────┬────────────────┐
│ id ┆ name ┆ fitness.height ┆ fitness.weight │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ i64 ┆ i64 │
╞══════╪════════════╪════════════════╪════════════════╡
│ 1 ┆ Cole Volk ┆ 180 ┆ 85 │
│ 2 ┆ Faye Raker ┆ 155 ┆ 58 │
│ null ┆ Mark Reg ┆ 170 ┆ 78 │
└──────┴────────────┴────────────────┴────────────────┘
Normalize to a specific depth, using a custom JSON encoder
(note that `orjson.dumps` encodes to bytes, not str).
>>> import orjson
>>> pl.json_normalize(data, max_level=0, encoder=orjson.dumps)
shape: (3, 3)
┌──────┬────────────┬───────────────────────────────┐
│ id ┆ name ┆ fitness │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ binary │
╞══════╪════════════╪═══════════════════════════════╡
│ 1 ┆ Cole Volk ┆ b"{"height":180,"weight":85}" │
│ 2 ┆ Faye Raker ┆ b"{"height":155,"weight":58}" │
│ null ┆ Mark Reg ┆ b"{"height":170,"weight":78}" │
└──────┴────────────┴───────────────────────────────┘
DATA
__all__ = ['from_arrow', 'from_dataframe', 'from_dict', 'from_dicts', ...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\convert\__init__.py
polars.corr
Python Library Documentation: function corr in module polars.functions.lazy
corr(a: 'IntoExpr', b: 'IntoExpr', *, method: 'CorrelationMethod' = 'pearson', ddof: 'int | None' = None, propagate_nans: 'bool' = False, eager: 'bool' = False) -> 'Expr | Series'
Compute the Pearson's or Spearman rank correlation between two columns.
Parameters
----------
a
Column name or Expression.
b
Column name or Expression.
ddof
Has no effect, do not use.
.. deprecated:: 1.17.0
method : {'pearson', 'spearman'}
Correlation method.
propagate_nans
If `True` any `NaN` encountered will lead to `NaN` in the output.
Defaults to `False` where `NaN` are regarded as larger than any finite number
and thus lead to the highest rank.
eager
Evaluate immediately and return a `Series`; this requires that at least one
of the given arguments is a `Series`. If set to `False` (default), return
an expression instead.
Examples
--------
Pearson's correlation:
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.corr("a", "b"))
shape: (1, 1)
┌──────────┐
│ a │
│ --- │
│ f64 │
╞══════════╡
│ 0.544705 │
└──────────┘
Spearman rank correlation:
>>> df.select(pl.corr("a", "b", method="spearman"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ f64 │
╞═════╡
│ 0.5 │
└─────┘
Eager evaluation:
>>> s1 = pl.Series("a", [1, 8, 3])
>>> s2 = pl.Series("b", [4, 5, 2])
>>> pl.corr(s1, s2, eager=True)
shape: (1,)
Series: 'a' [f64]
[
0.544705
]
>>> pl.corr(s1, s2, method="spearman", eager=True)
shape: (1,)
Series: 'a' [f64]
[
0.5
]
polars.count
Python Library Documentation: function count in module polars.functions.lazy
count(*columns: 'str') -> 'Expr'
Return the number of non-null values in the column.
This function is syntactic sugar for `col(columns).count()`.
Calling this function without any arguments returns the number of rows in the
context. **This way of using the function is deprecated.** Please use :func:`len`
instead.
Parameters
----------
*columns
One or more column names.
Returns
-------
Expr
Expression of data type :class:`UInt32`.
See Also
--------
Expr.count
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, None],
... "b": [3, None, None],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.count("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 2 │
└─────┘
Return the number of non-null values in multiple columns.
>>> df.select(pl.count("b", "c"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ u32 ┆ u32 │
╞═════╪═════╡
│ 1 ┆ 3 │
└─────┴─────┘
Return the number of rows in a context. **This way of using the function is
deprecated.** Please use :func:`len` instead.
>>> df.select(pl.count()) # doctest: +SKIP
shape: (1, 1)
┌───────┐
│ count │
│ --- │
│ u32 │
╞═══════╡
│ 3 │
└───────┘
polars.cov
Python Library Documentation: function cov in module polars.functions.lazy
cov(a: 'IntoExpr', b: 'IntoExpr', *, ddof: 'int' = 1, eager: 'bool' = False) -> 'Expr | Series'
Compute the covariance between two columns/ expressions.
Parameters
----------
a
Column name or Expression.
b
Column name or Expression.
ddof
"Delta Degrees of Freedom": the divisor used in the calculation is N - ddof,
where N represents the number of elements.
By default ddof is 1.
eager
Evaluate immediately and return a `Series`; this requires that at least one
of the given arguments is a `Series`. If set to `False` (default), return
an expression instead.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... },
... )
>>> df.select(
... x=pl.cov("a", "b"),
... y=pl.cov("a", "b", ddof=2),
... )
shape: (1, 2)
┌─────┬─────┐
│ x ┆ y │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞═════╪═════╡
│ 3.0 ┆ 6.0 │
└─────┴─────┘
Eager evaluation:
>>> s1 = pl.Series("a", [1, 8, 3])
>>> s2 = pl.Series("b", [4, 5, 2])
>>> pl.cov(s1, s2, eager=True)
shape: (1,)
Series: 'a' [f64]
[
3.0
]
polars.cum_count
Python Library Documentation: function cum_count in module polars.functions.lazy
cum_count(*columns: 'str', reverse: 'bool' = False) -> 'Expr'
Return the cumulative count of the non-null values in the column.
This function is syntactic sugar for `col(columns).cum_count()`.
Parameters
----------
*columns
Name(s) of the columns to use.
reverse
Reverse the operation.
Examples
--------
>>> df = pl.DataFrame({"a": [1, 2, None], "b": [3, None, None]})
>>> df.with_columns(
... ca=pl.cum_count("a"),
... cb=pl.cum_count("b"),
... )
shape: (3, 4)
┌──────┬──────┬─────┬─────┐
│ a ┆ b ┆ ca ┆ cb │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ u32 ┆ u32 │
╞══════╪══════╪═════╪═════╡
│ 1 ┆ 3 ┆ 1 ┆ 1 │
│ 2 ┆ null ┆ 2 ┆ 1 │
│ null ┆ null ┆ 2 ┆ 1 │
└──────┴──────┴─────┴─────┘
polars.cum_fold
Python Library Documentation: function cum_fold in module polars.functions.lazy
cum_fold(acc: 'IntoExpr', function: 'Callable[[Series, Series], Series]', exprs: 'Sequence[Expr | str] | Expr', *, returns_scalar: 'bool' = False, return_dtype: 'pl.DataTypeExpr | PolarsDataType | None' = None, include_init: 'bool' = False) -> 'Expr'
Cumulatively fold horizontally across columns with a left fold.
Every cumulative result is added as a separate field in a Struct column.
Parameters
----------
acc
Accumulator expression. This is the value that will be initialized when the fold
starts. For a sum this could for instance be lit(0).
function
Function to apply over the accumulator and the value.
Fn(acc, value) -> new_value
exprs
Expressions to aggregate over. May also be a wildcard expression.
returns_scalar
Whether or not `function` applied returns a scalar. This must be set correctly
by the user.
return_dtype
Output datatype.
If not set, the dtype will be inferred based on the dtype of the accumulator.
include_init
Include the initial accumulator state as struct field.
Notes
-----
If you simply want the first encountered expression as accumulator,
consider using :func:`cum_reduce`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [3, 4, 5],
... "c": [5, 6, 7],
... }
... )
>>> df.with_columns(
... pl.cum_fold(acc=pl.lit(1), function=lambda acc, x: acc + x, exprs=pl.all())
... )
shape: (3, 4)
┌─────┬─────┬─────┬───────────┐
│ a ┆ b ┆ c ┆ cum_fold │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ struct[3] │
╞═════╪═════╪═════╪═══════════╡
│ 1 ┆ 3 ┆ 5 ┆ {2,5,10} │
│ 2 ┆ 4 ┆ 6 ┆ {3,7,13} │
│ 3 ┆ 5 ┆ 7 ┆ {4,9,16} │
└─────┴─────┴─────┴───────────┘
polars.cum_reduce
Python Library Documentation: function cum_reduce in module polars.functions.lazy
cum_reduce(function: 'Callable[[Series, Series], Series]', exprs: 'Sequence[Expr | str] | Expr', *, returns_scalar: 'bool' = False, return_dtype: 'pl.DataTypeExpr | PolarsDataType | None' = None) -> 'Expr'
Cumulatively reduce horizontally across columns with a left fold.
Every cumulative result is added as a separate field in a Struct column.
Parameters
----------
function
Function to apply over the accumulator and the value.
Fn(acc, value) -> new_value
exprs
Expressions to aggregate over. May also be a wildcard expression.
returns_scalar
Whether or not `function` applied returns a scalar. This must be set correctly
by the user.
return_dtype
Output datatype.
If not set, the dtype will be inferred based on the dtype of the input
expressions.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [3, 4, 5],
... "c": [5, 6, 7],
... }
... )
>>> df.with_columns(pl.cum_reduce(function=lambda acc, x: acc + x, exprs=pl.all()))
shape: (3, 4)
┌─────┬─────┬─────┬────────────┐
│ a ┆ b ┆ c ┆ cum_reduce │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ struct[3] │
╞═════╪═════╪═════╪════════════╡
│ 1 ┆ 3 ┆ 5 ┆ {1,4,9} │
│ 2 ┆ 4 ┆ 6 ┆ {2,6,12} │
│ 3 ┆ 5 ┆ 7 ┆ {3,8,15} │
└─────┴─────┴─────┴────────────┘
polars.cum_sum
Python Library Documentation: function cum_sum in module polars.functions.aggregation.vertical
cum_sum(*names: 'str') -> 'Expr'
Cumulatively sum all values.
Syntactic sugar for `col(names).cum_sum()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
See Also
--------
cumsum_horizontal
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [4, 5, 6],
... }
... )
>>> df.select(pl.cum_sum("a"))
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 3 │
│ 6 │
└─────┘
polars.cum_sum_horizontal
Python Library Documentation: function cum_sum_horizontal in module polars.functions.aggregation.horizontal
cum_sum_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Cumulatively sum all values horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(pl.cum_sum_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬───────────┐
│ a ┆ b ┆ c ┆ cum_sum │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ struct[2] │
╞═════╪══════╪═════╪═══════════╡
│ 1 ┆ 4 ┆ x ┆ {1,5} │
│ 8 ┆ 5 ┆ y ┆ {8,13} │
│ 3 ┆ null ┆ z ┆ {3,null} │
└─────┴──────┴─────┴───────────┘
polars.dataframe
Python Library Documentation: package polars.dataframe in polars
NAME
polars.dataframe
PACKAGE CONTENTS
_html
frame
group_by
plotting
CLASSES
builtins.object
polars.dataframe.frame.DataFrame
class DataFrame(builtins.object)
| DataFrame(data: 'FrameInitTypes | None' = None, schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100, nan_to_null: 'bool' = False, height: 'int | None' = None) -> 'None'
|
| Two-dimensional data structure representing data as a table with rows and columns.
|
| Parameters
| ----------
| data : dict, Sequence, ndarray, Series, or pandas.DataFrame
| Two-dimensional data in various forms; dict input must contain Sequences,
| Generators, or a `range`. Sequence may contain Series or other Sequences.
| schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
| The schema of the resulting DataFrame. The schema may be declared in several
| ways:
|
| * As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
| * As a list of column names; in this case types are automatically inferred.
| * As a list of (name,type) pairs; this is equivalent to the dictionary form.
|
| The order of the schema determines the column order of the frame.
| When passing a dict, its insertion order is respected. To override specific
| column data types by name without changing column order, use
| ``schema_overrides`` instead.
|
| If you supply a list of column names that does not match the names in the
| underlying data, the names given here will overwrite them. The number
| of names given in the schema should match the underlying data dimensions.
|
| If set to `None` (default), the schema is inferred from the data.
| schema_overrides : dict, default None
| Support type specification or override of one or more columns; note that
| any dtypes inferred from the schema param will be overridden.
|
| The number of entries in the schema should match the underlying data
| dimensions, unless a sequence of dictionaries is being passed, in which case
| a *partial* schema can be declared to prevent specific fields from being loaded.
| strict : bool, default True
| Throw an error if any `data` value does not exactly match the given or inferred
| data type for that column. If set to `False`, values that do not match the data
| type are cast to that data type or, if casting is not possible, set to null
| instead.
| orient : {'col', 'row'}, default None
| Whether to interpret two-dimensional data as columns or as rows. If None,
| the orientation is inferred by matching the columns and data dimensions. If
| this does not yield conclusive results, column orientation is used.
| infer_schema_length : int or None
| The maximum number of rows to scan for schema inference. If set to `None`, the
| full data may be scanned *(this can be slow)*. This parameter only applies if
| the input data is a sequence or generator of rows; other input is read as-is.
| nan_to_null : bool, default False
| If the data comes from one or more numpy arrays, can optionally convert input
| data np.nan values to null instead. This is a no-op for all other input data.
| height : int or None, default None
| Allows constructing DataFrames with 0 width and a specified height. If
| passed with data, ensures the resulting DataFrame has this height.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Notes
| -----
| Polars explicitly does not support subclassing of its core data types. See
| the following GitHub issue for possible workarounds:
| https://github.com/pola-rs/polars/issues/2846#issuecomment-1711799869
|
| Examples
| --------
| Constructing a DataFrame from a dictionary:
|
| >>> data = {"a": [1, 2], "b": [3, 4]}
| >>> df = pl.DataFrame(data)
| >>> df
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
|
| Notice that the dtypes are automatically inferred as polars Int64:
|
| >>> df.dtypes
| [Int64, Int64]
|
| To specify a more detailed/specific frame schema you can supply the `schema`
| parameter with a dictionary of (name,dtype) pairs...
|
| >>> data = {"col1": [0, 2], "col2": [3, 7]}
| >>> df2 = pl.DataFrame(data, schema={"col1": pl.Float32, "col2": pl.Int64})
| >>> df2
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 0.0 ┆ 3 │
| │ 2.0 ┆ 7 │
| └──────┴──────┘
|
| ...a sequence of (name,dtype) pairs...
|
| >>> data = {"col1": [1, 2], "col2": [3, 4]}
| >>> df3 = pl.DataFrame(data, schema=[("col1", pl.Float32), ("col2", pl.Int64)])
| >>> df3
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 1.0 ┆ 3 │
| │ 2.0 ┆ 4 │
| └──────┴──────┘
|
| ...or a list of typed Series.
|
| >>> data = [
| ... pl.Series("col1", [1, 2], dtype=pl.Float32),
| ... pl.Series("col2", [3, 4], dtype=pl.Int64),
| ... ]
| >>> df4 = pl.DataFrame(data)
| >>> df4
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 1.0 ┆ 3 │
| │ 2.0 ┆ 4 │
| └──────┴──────┘
|
| Constructing a DataFrame from a numpy ndarray, specifying column names:
|
| >>> import numpy as np
| >>> data = np.array([(1, 2), (3, 4)], dtype=np.int64)
| >>> df5 = pl.DataFrame(data, schema=["a", "b"], orient="col")
| >>> df5
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
|
| Constructing a DataFrame from a list of lists, row orientation specified:
|
| >>> data = [[1, 2, 3], [4, 5, 6]]
| >>> df6 = pl.DataFrame(data, schema=["a", "b", "c"], orient="row")
| >>> df6
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| Methods defined here:
|
| __add__(self, other: 'DataFrame | Series | int | float | bool | str') -> 'DataFrame'
|
| __array__(self, dtype: 'np.dtype[Any] | None' = None, copy: 'bool | None' = None) -> 'np.ndarray[Any, Any]'
| Return a NumPy ndarray with the given data type.
|
| This method ensures a Polars DataFrame can be treated as a NumPy ndarray.
| It enables `np.asarray` and NumPy universal functions.
|
| See the NumPy documentation for more information:
| https://numpy.org/doc/stable/user/basics.interoperability.html#the-array-method
|
| __arrow_c_stream__(self, requested_schema: 'object | None' = None) -> 'object'
| Export a DataFrame via the Arrow PyCapsule Interface.
|
| https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html
|
| __bool__(self) -> 'NoReturn'
|
| __contains__(self, key: 'str') -> 'bool'
|
| __copy__(self) -> 'DataFrame'
|
| __dataframe__(self, nan_as_null: 'bool' = False, allow_copy: 'bool' = True) -> 'PolarsDataFrame'
| Convert to a dataframe object implementing the dataframe interchange protocol.
|
| .. deprecated:: 1.40.0
| Support for the Dataframe Interchange Protocol is deprecated.
|
| Parameters
| ----------
| nan_as_null
| Overwrite null values in the data with `NaN`.
|
| .. warning::
| This functionality has not been implemented and the parameter will be
| removed in a future version.
| Setting this to `True` will raise a `NotImplementedError`.
| allow_copy
| Allow memory to be copied to perform the conversion. If set to `False`,
| causes conversions that are not zero-copy to fail.
|
| Notes
| -----
| Details on the Python dataframe interchange protocol:
| https://data-apis.org/dataframe-protocol/latest/index.html
|
| Examples
| --------
| Convert a Polars DataFrame to a generic dataframe object and access some
| properties.
|
| >>> df = pl.DataFrame({"a": [1, 2], "b": [3.0, 4.0], "c": ["x", "y"]})
| >>> dfi = df.__dataframe__() # doctest: +SKIP
| >>> dfi.num_rows() # doctest: +SKIP
| 2
| >>> dfi.get_column(1).dtype # doctest: +SKIP
| (<DtypeKind.FLOAT: 2>, 64, 'g', '=')
|
| __deepcopy__(self, memo: 'None' = None) -> 'DataFrame'
|
| __eq__(self, other: 'object') -> 'DataFrame'
| Return self==value.
|
| __floordiv__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| __ge__(self, other: 'Any') -> 'DataFrame'
| Return self>=value.
|
| __getitem__(self, key: 'SingleIndexSelector | SingleColSelector | MultiColSelector | MultiIndexSelector | tuple[SingleIndexSelector, SingleColSelector] | tuple[SingleIndexSelector, MultiColSelector] | tuple[MultiIndexSelector, SingleColSelector] | tuple[MultiIndexSelector, MultiColSelector]') -> 'DataFrame | Series | Any'
| Get part of the DataFrame as a new DataFrame, Series, or scalar.
|
| Parameters
| ----------
| key
| Rows / columns to select. This is easiest to explain via example. Suppose
| we have a DataFrame with columns `'a'`, `'d'`, `'c'`, `'d'`. Here is what
| various types of `key` would do:
|
| - `df[0, 'a']` extracts the first element of column `'a'` and returns a
| scalar.
| - `df[0]` extracts the first row and returns a Dataframe.
| - `df['a']` extracts column `'a'` and returns a Series.
| - `df[0:2]` extracts the first two rows and returns a Dataframe.
| - `df[0:2, 'a']` extracts the first two rows from column `'a'` and returns
| a Series.
| - `df[0:2, 0]` extracts the first two rows from the first column and returns
| a Series.
| - `df[[0, 1], [0, 1, 2]]` extracts the first two rows and the first three
| columns and returns a Dataframe.
| - `df[0: 2, ['a', 'c']]` extracts the first two rows from columns `'a'` and
| `'c'` and returns a Dataframe.
| - `df[:, 0: 2]` extracts all rows from the first two columns and returns a
| Dataframe.
| - `df[:, 'a': 'c']` extracts all rows and all columns positioned between
| `'a'` and `'c'` *inclusive* and returns a Dataframe. In our example,
| that would extract columns `'a'`, `'d'`, and `'c'`.
|
| Returns
| -------
| DataFrame, Series, or scalar, depending on `key`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"a": [1, 2, 3], "d": [4, 5, 6], "c": [1, 3, 2], "b": [7, 8, 9]}
| ... )
| >>> df[0]
| shape: (1, 4)
| ┌─────┬─────┬─────┬─────┐
| │ a ┆ d ┆ c ┆ b │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╡
| │ 1 ┆ 4 ┆ 1 ┆ 7 │
| └─────┴─────┴─────┴─────┘
| >>> df[0, "a"]
| 1
| >>> df["a"]
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
| >>> df[0:2]
| shape: (2, 4)
| ┌─────┬─────┬─────┬─────┐
| │ a ┆ d ┆ c ┆ b │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╡
| │ 1 ┆ 4 ┆ 1 ┆ 7 │
| │ 2 ┆ 5 ┆ 3 ┆ 8 │
| └─────┴─────┴─────┴─────┘
| >>> df[0:2, "a"]
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 2
| ]
| >>> df[0:2, 0]
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 2
| ]
| >>> df[[0, 1], [0, 1, 2]]
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ d ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 4 ┆ 1 │
| │ 2 ┆ 5 ┆ 3 │
| └─────┴─────┴─────┘
| >>> df[0:2, ["a", "c"]]
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ c │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 3 │
| └─────┴─────┘
| >>> df[:, 0:2]
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ d │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| │ 3 ┆ 6 │
| └─────┴─────┘
| >>> df[:, "a":"c"]
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ d ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 4 ┆ 1 │
| │ 2 ┆ 5 ┆ 3 │
| │ 3 ┆ 6 ┆ 2 │
| └─────┴─────┴─────┘
|
| __getstate__(self) -> 'bytes'
| Helper for pickle.
|
| __gt__(self, other: 'Any') -> 'DataFrame'
| Return self>value.
|
| __init__(self, data: 'FrameInitTypes | None' = None, schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100, nan_to_null: 'bool' = False, height: 'int | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> 'Iterator[Series]'
|
| __le__(self, other: 'Any') -> 'DataFrame'
| Return self<=value.
|
| __len__(self) -> 'int'
|
| __lt__(self, other: 'Any') -> 'DataFrame'
| Return self<value.
|
| __mod__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| __mul__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| __ne__(self, other: 'object') -> 'DataFrame'
| Return self!=value.
|
| __radd__(self, other: 'DataFrame | Series | int | float | bool | str') -> 'DataFrame'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __reversed__(self) -> 'Iterator[Series]'
|
| __rmul__(self, other: 'int | float') -> 'DataFrame'
|
| __setitem__(self, key: 'str | Sequence[str] | tuple[Any, str | int]', value: 'Any') -> 'None'
| Modify DataFrame elements in place, using assignment syntax.
|
| Parameters
| ----------
| key : str | Sequence[int] | Sequence[str] | tuple[Any, str | int]
| Specifies the location(s) within the DataFrame to assign new values.
| The behavior varies based on the type of `key`:
|
| - Str: `df["a"] = value`:
| Not supported. Raises a `TypeError`. Use `df.with_columns(...)`
| to add or modify columns.
|
| - Sequence[str]: `df[["a", "b"]] = value`:
| Assigns multiple columns at once. `value` must be a 2D array-like
| structure with the same number of columns as the list
| of column names provided.
|
| - tuple[Any, str | int]: `df[row_idx, "a"] = value`:
| Assigns a new value to a specific element in the DataFrame, where
| `row_idx` specifies the row and `"a"` specifies the column.
|
| - `df[row_idx, col_idx] = value`:
| Similar to the above, but `col_idx` is the integer index of the column.
|
| value : Any
| The new value(s) to assign. The expected structure of `value` depends on the
| form of `key`:
|
| - For multiple column assignment (`df[["a", "b"]] = value`), `value` should
| be a 2D array-like object with shape (n_rows, n_columns).
|
| - For single element assignment (`df[row_idx, "a"] = value`), `value` should
| be a scalar.
|
| Raises
| ------
| TypeError
| If an unsupported assignment is attempted, such as assigning a Series
| directly to a column using `df["a"] = series`.
|
| ValueError
| If the shape of `value` does not match the expected shape based on `key`.
|
| Examples
| --------
| Sequence[str] : `df[["a", "b"]] = value`:
|
| >>> import numpy as np
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
| >>> df[["a", "b"]] = np.array([[10, 40], [20, 50], [30, 60]])
| >>> df
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 10 ┆ 40 │
| │ 20 ┆ 50 │
| │ 30 ┆ 60 │
| └─────┴─────┘
|
| tuple[Any, str | int] : `df[row_idx, "a"] = value`:
|
| >>> df[1, "a"] = 100
| >>> df
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 10 ┆ 40 │
| │ 100 ┆ 50 │
| │ 30 ┆ 60 │
| └─────┴─────┘
|
| `df[row_idx, col_idx] = value`:
|
| >>> df[0, 1] = 30
| >>> df
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 10 ┆ 30 │
| │ 100 ┆ 50 │
| │ 30 ┆ 60 │
| └─────┴─────┘
|
| __setstate__(self, state: 'bytes') -> 'None'
|
| __str__(self) -> 'str'
| Return str(self).
|
| __sub__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| __truediv__(self, other: 'DataFrame | Series | int | float') -> 'DataFrame'
|
| approx_n_unique(self) -> 'DataFrame'
| Approximate count of unique values.
|
| .. deprecated:: 0.20.11
| Use the `select(pl.all().approx_n_unique())` method instead.
|
| This is done using the HyperLogLog++ algorithm for cardinality estimation.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> df.approx_n_unique() # doctest: +SKIP
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| bottom_k(self, k: 'int', *, by: 'IntoExpr | Iterable[IntoExpr]', reverse: 'bool | Sequence[bool]' = False) -> 'DataFrame'
| Return the `k` smallest rows.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| k
| Number of rows to return.
| by
| Column(s) used to determine the bottom rows.
| Accepts expression input. Strings are parsed as column names.
| reverse
| Consider the `k` largest elements of the `by` column(s) (instead of the `k`
| smallest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [2, 1, 1, 3, 2, 1],
| ... }
| ... )
|
| Get the rows which contain the 4 smallest values in column b.
|
| >>> df.bottom_k(4, by="b")
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 1 │
| │ a ┆ 1 │
| │ c ┆ 1 │
| │ a ┆ 2 │
| └─────┴─────┘
|
| Get the rows which contain the 4 smallest values when sorting on column a and b.
|
| >>> df.bottom_k(4, by=["a", "b"])
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ a ┆ 2 │
| │ b ┆ 1 │
| │ b ┆ 2 │
| └─────┴─────┘
|
| cast(self, dtypes: 'Mapping[ColumnNameOrSelector | PolarsDataType, PolarsDataType | PythonDataType] | PolarsDataType | Schema', *, strict: 'bool' = True) -> 'DataFrame'
| Cast DataFrame column(s) to the specified dtype(s).
|
| Parameters
| ----------
| dtypes
| Mapping of column names (or selector) to dtypes, or a single dtype
| to which all columns will be cast.
| strict
| Raise if cast is invalid on rows after predicates are pushed down.
| If `False`, invalid casts will produce null values.
|
| Examples
| --------
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": [date(2020, 1, 2), date(2021, 3, 4), date(2022, 5, 6)],
| ... }
| ... )
|
| Cast specific frame columns to the specified dtypes:
|
| >>> df.cast({"foo": pl.Float32, "bar": pl.UInt8})
| shape: (3, 3)
| ┌─────┬─────┬────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f32 ┆ u8 ┆ date │
| ╞═════╪═════╪════════════╡
| │ 1.0 ┆ 6 ┆ 2020-01-02 │
| │ 2.0 ┆ 7 ┆ 2021-03-04 │
| │ 3.0 ┆ 8 ┆ 2022-05-06 │
| └─────┴─────┴────────────┘
|
| Cast all frame columns matching one dtype (or dtype group) to another dtype:
|
| >>> df.cast({pl.Date: pl.Datetime})
| shape: (3, 3)
| ┌─────┬─────┬─────────────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ datetime[μs] │
| ╞═════╪═════╪═════════════════════╡
| │ 1 ┆ 6.0 ┆ 2020-01-02 00:00:00 │
| │ 2 ┆ 7.0 ┆ 2021-03-04 00:00:00 │
| │ 3 ┆ 8.0 ┆ 2022-05-06 00:00:00 │
| └─────┴─────┴─────────────────────┘
|
| Use selectors to define the columns being cast:
|
| >>> import polars.selectors as cs
| >>> df.cast({cs.numeric(): pl.UInt32, cs.temporal(): pl.String})
| shape: (3, 3)
| ┌─────┬─────┬────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ str │
| ╞═════╪═════╪════════════╡
| │ 1 ┆ 6 ┆ 2020-01-02 │
| │ 2 ┆ 7 ┆ 2021-03-04 │
| │ 3 ┆ 8 ┆ 2022-05-06 │
| └─────┴─────┴────────────┘
|
| Cast all frame columns to the specified dtype:
|
| >>> df.cast(pl.String).to_dict(as_series=False)
| {'foo': ['1', '2', '3'],
| 'bar': ['6.0', '7.0', '8.0'],
| 'ham': ['2020-01-02', '2021-03-04', '2022-05-06']}
|
| clear(self, n: 'int' = 0) -> 'DataFrame'
| Create an empty (n=0) or `n`-row null-filled (n>0) copy of the DataFrame.
|
| Returns a `n`-row null-filled DataFrame with an identical schema.
| `n` can be greater than the current number of rows in the DataFrame.
|
| Parameters
| ----------
| n
| Number of (null-filled) rows to return in the cleared frame.
|
| See Also
| --------
| clone : Cheap deepcopy/clone.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> df.clear()
| shape: (0, 3)
| ┌─────┬─────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞═════╪═════╪══════╡
| └─────┴─────┴──────┘
|
| >>> df.clear(n=2)
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ null ┆ null ┆ null │
| └──────┴──────┴──────┘
|
| clone(self) -> 'DataFrame'
| Create a copy of this DataFrame.
|
| This is a cheap operation that does not copy data.
|
| See Also
| --------
| clear : Create an empty copy of the current DataFrame, with identical
| schema but no data.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> df.clone()
| shape: (4, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true │
| │ 2 ┆ 4.0 ┆ true │
| │ 3 ┆ 10.0 ┆ false │
| │ 4 ┆ 13.0 ┆ true │
| └─────┴──────┴───────┘
|
| collect_schema(self) -> 'Schema'
| Get an ordered mapping of column names to their data type.
|
| This is an alias for the :attr:`schema` property.
|
| See Also
| --------
| schema
|
| Notes
| -----
| This method is included to facilitate writing code that is generic for both
| DataFrame and LazyFrame.
|
| Examples
| --------
| Determine the schema.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.collect_schema()
| Schema({'foo': Int64, 'bar': Float64, 'ham': String})
|
| Access various properties of the schema using the :class:`Schema` object.
|
| >>> schema = df.collect_schema()
| >>> schema["bar"]
| Float64
| >>> schema.names()
| ['foo', 'bar', 'ham']
| >>> schema.dtypes()
| [Int64, Float64, String]
| >>> schema.len()
| 3
|
| corr(self, *, label: 'str | None' = None, **kwargs: 'Any') -> 'DataFrame'
| Return pairwise Pearson product-moment correlation coefficients between columns.
|
| See numpy `corrcoef` for more information:
| https://numpy.org/doc/stable/reference/generated/numpy.corrcoef.html
|
| Parameters
| ----------
| label
| If given, a new column that contains the labels (column names)
| associated with each row is added, with this name.
| **kwargs
| Keyword arguments that are passed to `numpy.corrcoef`.
|
| Notes
| -----
| This functionality requires `numpy` to be installed.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [3, 2, 1], "ham": [7, 8, 9]})
| >>> df.corr()
| shape: (3, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪══════╡
| │ 1.0 ┆ -1.0 ┆ 1.0 │
| │ -1.0 ┆ 1.0 ┆ -1.0 │
| │ 1.0 ┆ -1.0 ┆ 1.0 │
| └──────┴──────┴──────┘
| >>> df.corr(label="cols")
| shape: (3, 4)
| ┌──────┬──────┬──────┬──────┐
| │ cols ┆ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪══════╪══════╡
| │ foo ┆ 1.0 ┆ -1.0 ┆ 1.0 │
| │ bar ┆ -1.0 ┆ 1.0 ┆ -1.0 │
| │ ham ┆ 1.0 ┆ -1.0 ┆ 1.0 │
| └──────┴──────┴──────┴──────┘
|
| count(self) -> 'DataFrame'
| Return the number of non-null elements for each column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"a": [1, 2, 3, 4], "b": [1, 2, 1, None], "c": [None, None, None, None]}
| ... )
| >>> df.count()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 4 ┆ 3 ┆ 0 │
| └─────┴─────┴─────┘
|
| describe(self, percentiles: 'Sequence[float] | float | None' = (0.25, 0.5, 0.75), *, interpolation: 'QuantileMethod' = 'nearest') -> 'DataFrame'
| Summary statistics for a DataFrame.
|
| Parameters
| ----------
| percentiles
| One or more percentiles to include in the summary statistics.
| All values must be in the range `[0, 1]`.
|
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method used when calculating percentiles.
|
| Notes
| -----
| The median is included by default as the 50% percentile.
|
| Warnings
| --------
| We do not guarantee the output of `describe` to be stable. It will show
| statistics that we deem informative, and may be updated in the future.
| Using `describe` programmatically (versus interactive exploration) is
| not recommended for this reason.
|
| See Also
| --------
| glimpse
|
| Examples
| --------
| >>> from datetime import date, time
| >>> df = pl.DataFrame(
| ... {
| ... "float": [1.0, 2.8, 3.0],
| ... "int": [40, 50, None],
| ... "bool": [True, False, True],
| ... "str": ["zz", "xx", "yy"],
| ... "date": [date(2020, 1, 1), date(2021, 7, 5), date(2022, 12, 31)],
| ... "time": [time(10, 20, 30), time(14, 45, 50), time(23, 15, 10)],
| ... }
| ... )
|
| Show default frame statistics:
|
| >>> df.describe()
| shape: (9, 7)
| ┌────────────┬──────────┬──────────┬──────────┬──────┬─────────────────────┬──────────┐
| │ statistic ┆ float ┆ int ┆ bool ┆ str ┆ date ┆ time │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞════════════╪══════════╪══════════╪══════════╪══════╪═════════════════════╪══════════╡
| │ count ┆ 3.0 ┆ 2.0 ┆ 3.0 ┆ 3 ┆ 3 ┆ 3 │
| │ null_count ┆ 0.0 ┆ 1.0 ┆ 0.0 ┆ 0 ┆ 0 ┆ 0 │
| │ mean ┆ 2.266667 ┆ 45.0 ┆ 0.666667 ┆ null ┆ 2021-07-02 16:00:00 ┆ 16:07:10 │
| │ std ┆ 1.101514 ┆ 7.071068 ┆ null ┆ null ┆ null ┆ null │
| │ min ┆ 1.0 ┆ 40.0 ┆ 0.0 ┆ xx ┆ 2020-01-01 ┆ 10:20:30 │
| │ 25% ┆ 2.8 ┆ 40.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 50% ┆ 2.8 ┆ 50.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 75% ┆ 3.0 ┆ 50.0 ┆ null ┆ null ┆ 2022-12-31 ┆ 23:15:10 │
| │ max ┆ 3.0 ┆ 50.0 ┆ 1.0 ┆ zz ┆ 2022-12-31 ┆ 23:15:10 │
| └────────────┴──────────┴──────────┴──────────┴──────┴─────────────────────┴──────────┘
|
| Customize which percentiles are displayed, applying linear interpolation:
|
| >>> with pl.Config(tbl_rows=12):
| ... df.describe(
| ... percentiles=[0.1, 0.3, 0.5, 0.7, 0.9],
| ... interpolation="linear",
| ... )
| shape: (11, 7)
| ┌────────────┬──────────┬──────────┬──────────┬──────┬─────────────────────┬──────────┐
| │ statistic ┆ float ┆ int ┆ bool ┆ str ┆ date ┆ time │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞════════════╪══════════╪══════════╪══════════╪══════╪═════════════════════╪══════════╡
| │ count ┆ 3.0 ┆ 2.0 ┆ 3.0 ┆ 3 ┆ 3 ┆ 3 │
| │ null_count ┆ 0.0 ┆ 1.0 ┆ 0.0 ┆ 0 ┆ 0 ┆ 0 │
| │ mean ┆ 2.266667 ┆ 45.0 ┆ 0.666667 ┆ null ┆ 2021-07-02 16:00:00 ┆ 16:07:10 │
| │ std ┆ 1.101514 ┆ 7.071068 ┆ null ┆ null ┆ null ┆ null │
| │ min ┆ 1.0 ┆ 40.0 ┆ 0.0 ┆ xx ┆ 2020-01-01 ┆ 10:20:30 │
| │ 10% ┆ 1.36 ┆ 41.0 ┆ null ┆ null ┆ 2020-04-20 ┆ 11:13:34 │
| │ 30% ┆ 2.08 ┆ 43.0 ┆ null ┆ null ┆ 2020-11-26 ┆ 12:59:42 │
| │ 50% ┆ 2.8 ┆ 45.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 70% ┆ 2.88 ┆ 47.0 ┆ null ┆ null ┆ 2022-02-07 ┆ 18:09:34 │
| │ 90% ┆ 2.96 ┆ 49.0 ┆ null ┆ null ┆ 2022-09-13 ┆ 21:33:18 │
| │ max ┆ 3.0 ┆ 50.0 ┆ 1.0 ┆ zz ┆ 2022-12-31 ┆ 23:15:10 │
| └────────────┴──────────┴──────────┴──────────┴──────┴─────────────────────┴──────────┘
|
| drop(self, *columns: 'ColumnNameOrSelector | Iterable[ColumnNameOrSelector]', strict: 'bool' = True) -> 'DataFrame'
| Remove columns from the dataframe.
|
| Parameters
| ----------
| *columns
| Names of the columns that should be removed from the dataframe.
| Accepts column selector input.
| strict
| Validate that all column names exist in the current schema,
| and throw an exception if any do not.
|
| Examples
| --------
| Drop a single column by passing the name of that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.drop("ham")
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 6.0 │
| │ 2 ┆ 7.0 │
| │ 3 ┆ 8.0 │
| └─────┴─────┘
|
| Drop multiple columns by passing a list of column names.
|
| >>> df.drop(["bar", "ham"])
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Drop multiple columns by passing a selector.
|
| >>> import polars.selectors as cs
| >>> df.drop(cs.numeric())
| shape: (3, 1)
| ┌─────┐
| │ ham │
| │ --- │
| │ str │
| ╞═════╡
| │ a │
| │ b │
| │ c │
| └─────┘
|
| Use positional arguments to drop multiple columns.
|
| >>> df.drop("foo", "ham")
| shape: (3, 1)
| ┌─────┐
| │ bar │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 6.0 │
| │ 7.0 │
| │ 8.0 │
| └─────┘
|
| drop_in_place(self, name: 'str') -> 'Series'
| Drop a single column in-place and return the dropped column.
|
| Parameters
| ----------
| name
| Name of the column to drop.
|
| Returns
| -------
| Series
| The dropped column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.drop_in_place("ham")
| shape: (3,)
| Series: 'ham' [str]
| [
| "a"
| "b"
| "c"
| ]
|
| drop_nans(self, subset: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None) -> 'DataFrame'
| Drop all rows that contain one or more NaN values.
|
| The original order of the remaining rows is preserved.
|
| Parameters
| ----------
| subset
| Column name(s) for which NaN values are considered; if set to `None`
| (default), use all columns (note that only floating-point columns
| can contain NaNs).
|
| See Also
| --------
| drop_nulls
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To drop null values, use :func:`drop_nulls`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [-20.5, float("nan"), 80.0],
| ... "bar": [float("nan"), 110.0, 25.5],
| ... "ham": ["xxx", "yyy", None],
| ... }
| ... )
|
| The default behavior of this method is to drop rows where any single
| value in the row is NaN:
|
| >>> df.drop_nans()
| shape: (1, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════╪══════╪══════╡
| │ 80.0 ┆ 25.5 ┆ null │
| └──────┴──────┴──────┘
|
| This behaviour can be constrained to consider only a subset of columns, as
| defined by name, or with a selector. For example, dropping rows only if
| there is a NaN in the "bar" column:
|
| >>> df.drop_nans(subset=["bar"])
| shape: (2, 3)
| ┌──────┬───────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════╪═══════╪══════╡
| │ NaN ┆ 110.0 ┆ yyy │
| │ 80.0 ┆ 25.5 ┆ null │
| └──────┴───────┴──────┘
|
| Dropping a row only if *all* values are NaN requires a different formulation:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [float("nan"), float("nan"), float("nan"), float("nan")],
| ... "b": [10.0, 2.5, float("nan"), 5.25],
| ... "c": [65.75, float("nan"), float("nan"), 10.5],
| ... }
| ... )
| >>> df.filter(~pl.all_horizontal(pl.all().is_nan()))
| shape: (3, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞═════╪══════╪═══════╡
| │ NaN ┆ 10.0 ┆ 65.75 │
| │ NaN ┆ 2.5 ┆ NaN │
| │ NaN ┆ 5.25 ┆ 10.5 │
| └─────┴──────┴───────┘
|
| drop_nulls(self, subset: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None) -> 'DataFrame'
| Drop all rows that contain one or more null values.
|
| The original order of the remaining rows is preserved.
|
| Parameters
| ----------
| subset
| Column name(s) for which null values are considered.
| If set to `None` (default), use all columns.
|
| See Also
| --------
| drop_nans
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To drop NaN values, use :func:`drop_nans`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, None, 8],
| ... "ham": ["a", "b", None],
| ... }
| ... )
|
| The default behavior of this method is to drop rows where any single
| value of the row is null.
|
| >>> df.drop_nulls()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| This behaviour can be constrained to consider only a subset of columns, as
| defined by name or with a selector. For example, dropping rows if there is
| a null in any of the integer columns:
|
| >>> import polars.selectors as cs
| >>> df.drop_nulls(subset=cs.integer())
| shape: (2, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 6 ┆ a │
| │ 3 ┆ 8 ┆ null │
| └─────┴─────┴──────┘
|
| Below are some additional examples that show how to drop null
| values based on other conditions.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, None, None, None],
| ... "b": [1, 2, None, 1],
| ... "c": [1, None, None, 1],
| ... }
| ... )
| >>> df
| shape: (4, 3)
| ┌──────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ null ┆ i64 ┆ i64 │
| ╞══════╪══════╪══════╡
| │ null ┆ 1 ┆ 1 │
| │ null ┆ 2 ┆ null │
| │ null ┆ null ┆ null │
| │ null ┆ 1 ┆ 1 │
| └──────┴──────┴──────┘
|
| Drop a row only if all values are null:
|
| >>> df.filter(~pl.all_horizontal(pl.all().is_null()))
| shape: (3, 3)
| ┌──────┬─────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ null ┆ i64 ┆ i64 │
| ╞══════╪═════╪══════╡
| │ null ┆ 1 ┆ 1 │
| │ null ┆ 2 ┆ null │
| │ null ┆ 1 ┆ 1 │
| └──────┴─────┴──────┘
|
| Drop a column if all values are null:
|
| >>> df[[s.name for s in df if not (s.null_count() == df.height)]]
| shape: (4, 2)
| ┌──────┬──────┐
| │ b ┆ c │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ 1 ┆ 1 │
| │ 2 ┆ null │
| │ null ┆ null │
| │ 1 ┆ 1 │
| └──────┴──────┘
|
| equals(self, other: 'DataFrame', *, null_equal: 'bool' = True) -> 'bool'
| Check whether the DataFrame is equal to another DataFrame.
|
| Parameters
| ----------
| other
| DataFrame to compare with.
| null_equal
| Consider null values as equal.
|
| See Also
| --------
| polars.testing.assert_frame_equal
|
| Examples
| --------
| >>> df1 = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df2 = pl.DataFrame(
| ... {
| ... "foo": [3, 2, 1],
| ... "bar": [8.0, 7.0, 6.0],
| ... "ham": ["c", "b", "a"],
| ... }
| ... )
| >>> df1.equals(df1)
| True
| >>> df1.equals(df2)
| False
|
| estimated_size(self, unit: 'SizeUnit' = 'b') -> 'int | float'
| Return an estimation of the total (heap) allocated size of the `DataFrame`.
|
| Estimated size is given in the specified unit (bytes by default).
|
| This estimation is the sum of the size of its buffers, validity, including
| nested arrays. Multiple arrays may share buffers and bitmaps. Therefore, the
| size of 2 arrays is not the sum of the sizes computed from this function. In
| particular, [`StructArray`]'s size is an upper bound.
|
| When an array is sliced, its allocated size remains constant because the buffer
| unchanged. However, this function will yield a smaller number. This is because
| this function returns the visible size of the buffer, not its total capacity.
|
| FFI buffers are included in this estimation.
|
| Notes
| -----
| For data with Object dtype, the estimated size only reports the pointer
| size, which is a huge underestimation.
|
| Parameters
| ----------
| unit : {'b', 'kb', 'mb', 'gb', 'tb'}
| Scale the returned size to the given unit.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "x": list(reversed(range(1_000_000))),
| ... "y": [v / 1000 for v in range(1_000_000)],
| ... "z": [str(v) for v in range(1_000_000)],
| ... },
| ... schema=[("x", pl.UInt32), ("y", pl.Float64), ("z", pl.String)],
| ... )
| >>> df.estimated_size()
| 17888890
| >>> df.estimated_size("mb")
| 17.0601749420166
|
| explode(self, columns: 'ColumnNameOrSelector | Iterable[ColumnNameOrSelector]', *more_columns: 'ColumnNameOrSelector', empty_as_null: 'bool' = <object object at 0x00000280D5528F20>, keep_nulls: 'bool' = True) -> 'DataFrame'
| Explode the dataframe to long format by exploding the given columns.
|
| Parameters
| ----------
| columns
| Column names, expressions, or a selector defining them. The underlying
| columns being exploded must be of the `List` or `Array` data type.
| *more_columns
| Additional names of columns to explode, specified as positional arguments.
| empty_as_null
| Explode an empty list/array into a `null`.
| keep_nulls
| Explode a `null` list/array into a `null`.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "letters": ["a", "a", "b", "c"],
| ... "numbers": [[1], [2, 3], [4, 5], [6, 7, 8]],
| ... }
| ... )
| >>> df
| shape: (4, 2)
| ┌─────────┬───────────┐
| │ letters ┆ numbers │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═════════╪═══════════╡
| │ a ┆ [1] │
| │ a ┆ [2, 3] │
| │ b ┆ [4, 5] │
| │ c ┆ [6, 7, 8] │
| └─────────┴───────────┘
| >>> df.explode("numbers", empty_as_null=False)
| shape: (8, 2)
| ┌─────────┬─────────┐
| │ letters ┆ numbers │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════════╡
| │ a ┆ 1 │
| │ a ┆ 2 │
| │ a ┆ 3 │
| │ b ┆ 4 │
| │ b ┆ 5 │
| │ c ┆ 6 │
| │ c ┆ 7 │
| │ c ┆ 8 │
| └─────────┴─────────┘
|
| extend(self, other: 'DataFrame') -> 'DataFrame'
| Extend the memory backed by this `DataFrame` with the values from `other`.
|
| Different from `vstack` which adds the chunks from `other` to the chunks of
| this `DataFrame`, `extend` appends the data from `other` to the underlying
| memory locations and thus may cause a reallocation.
|
| If this does not cause a reallocation, the resulting data structure will not
| have any extra chunks and thus will yield faster queries.
|
| Prefer `extend` over `vstack` when you want to do a query after a single
| append. For instance, during online operations where you add `n` rows and rerun
| a query.
|
| Prefer `vstack` over `extend` when you want to append many times before
| doing a query. For instance, when you read in multiple files and want to store
| them in a single `DataFrame`. In the latter case, finish the sequence of
| `vstack` operations with a `rechunk`.
|
| Parameters
| ----------
| other
| DataFrame to vertically add.
|
| Warnings
| --------
| This method modifies the dataframe in-place. The dataframe is returned for
| convenience only.
|
| See Also
| --------
| vstack
|
| Examples
| --------
| >>> df1 = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df2 = pl.DataFrame({"foo": [10, 20, 30], "bar": [40, 50, 60]})
| >>> df1.extend(df2)
| shape: (6, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| │ 3 ┆ 6 │
| │ 10 ┆ 40 │
| │ 20 ┆ 50 │
| │ 30 ┆ 60 │
| └─────┴─────┘
|
| fill_nan(self, value: 'Expr | int | float | None') -> 'DataFrame'
| Fill floating point NaN values by an Expression evaluation.
|
| Parameters
| ----------
| value
| Value used to fill NaN values.
|
| Returns
| -------
| DataFrame
| DataFrame with NaN values replaced by the given value.
|
| See Also
| --------
| fill_null
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To fill null values, use :func:`fill_null`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1.5, 2, float("nan"), 4],
| ... "b": [0.5, 4, float("nan"), 13],
| ... }
| ... )
| >>> df.fill_nan(99)
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪══════╡
| │ 1.5 ┆ 0.5 │
| │ 2.0 ┆ 4.0 │
| │ 99.0 ┆ 99.0 │
| │ 4.0 ┆ 13.0 │
| └──────┴──────┘
|
| fill_null(self, value: 'Any | Expr | None' = None, strategy: 'FillNullStrategy | None' = None, limit: 'int | None' = None, *, matches_supertype: 'bool' = True) -> 'DataFrame'
| Fill null values using the specified value or strategy.
|
| Parameters
| ----------
| value
| Value used to fill null values.
| strategy : {None, 'forward', 'backward', 'min', 'max', 'mean', 'zero', 'one'}
| Strategy used to fill null values.
| limit
| Number of consecutive null values to fill when using the 'forward' or
| 'backward' strategy.
| matches_supertype
| Fill all matching supertype of the fill `value`.
|
| Returns
| -------
| DataFrame
| DataFrame with None values replaced by the filling strategy.
|
| See Also
| --------
| fill_nan
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To fill NaN values, use :func:`fill_nan`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 4],
| ... "b": [0.5, 4, None, 13],
| ... }
| ... )
| >>> df.fill_null(99)
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 99 ┆ 99.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
| >>> df.fill_null(strategy="forward")
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 2 ┆ 4.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| >>> df.fill_null(strategy="max")
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 4 ┆ 13.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| >>> df.fill_null(strategy="zero")
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 0 ┆ 0.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| filter(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]', **constraints: 'Any') -> 'DataFrame'
| Filter rows, retaining those that match the given predicate expression(s).
|
| The original order of the remaining rows is preserved.
|
| Only rows where the predicate resolves as True are retained; when the
| predicate result is False (or null), the row is discarded.
|
| Parameters
| ----------
| predicates
| Expression(s) that evaluate to a boolean Series.
| constraints
| Column filters; use `name = value` to filter columns by the supplied value.
| Each constraint will behave the same as `pl.col(name).eq(value)`, and
| be implicitly joined with the other filter conditions using `&`.
|
| Notes
| -----
| If you are transitioning from Pandas, and performing filter operations based on
| the comparison of two or more columns, please note that in Polars any comparison
| involving `null` values will result in a `null` result, *not* boolean True or
| False. As a result, these rows will not be retained. Ensure that null values
| are handled appropriately to avoid unexpected behaviour (see examples below).
|
| See Also
| --------
| remove
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, None, 4, None, 0],
| ... "bar": [6, 7, 8, None, None, 9, 0],
| ... "ham": ["a", "b", "c", None, "d", "e", "f"],
| ... }
| ... )
|
| Filter rows matching a condition:
|
| >>> df.filter(pl.col("foo") > 1)
| shape: (3, 3)
| ┌─────┬──────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪══════╪═════╡
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ null ┆ d │
| └─────┴──────┴─────┘
|
| Filter on multiple conditions, combined with and/or operators:
|
| >>> df.filter(
| ... (pl.col("foo") < 3) & (pl.col("ham") == "a"),
| ... )
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| >>> df.filter(
| ... (pl.col("foo") == 1) | (pl.col("ham") == "c"),
| ... )
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Provide multiple filters using `*args` syntax:
|
| >>> df.filter(
| ... pl.col("foo") <= 2,
| ... ~pl.col("ham").is_in(["b", "c"]),
| ... )
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 0 ┆ 0 ┆ f │
| └─────┴─────┴─────┘
|
| Provide multiple filters using `**kwargs` syntax:
|
| >>> df.filter(foo=2, ham="b")
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ 7 ┆ b │
| └─────┴─────┴─────┘
|
| Filter by comparing two columns against each other:
|
| >>> df.filter(
| ... pl.col("foo") == pl.col("bar"),
| ... )
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 0 ┆ 0 ┆ f │
| └─────┴─────┴─────┘
|
| >>> df.filter(
| ... pl.col("foo") != pl.col("bar"),
| ... )
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Notice how the row with `None` values is filtered out. In order to keep the
| same behavior as pandas, use:
|
| >>> df.filter(
| ... pl.col("foo").ne_missing(pl.col("bar")),
| ... )
| shape: (5, 3)
| ┌──────┬──────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ null ┆ d │
| │ null ┆ 9 ┆ e │
| └──────┴──────┴─────┘
|
| fold(self, operation: 'Callable[[Series, Series], Series]') -> 'Series'
| Apply a horizontal reduction on a DataFrame.
|
| This can be used to effectively determine aggregations on a row level, and can
| be applied to any DataType that can be supercast (cast to a similar parent
| type).
|
| An example of the supercast rules when applying an arithmetic operation on two
| DataTypes are for instance:
|
| - Int8 + String = String
| - Float32 + Int64 = Float32
| - Float32 + Float64 = Float64
|
| Examples
| --------
| A horizontal sum operation:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [2, 1, 3],
| ... "b": [1, 2, 3],
| ... "c": [1.0, 2.0, 3.0],
| ... }
| ... )
| >>> df.fold(lambda s1, s2: s1 + s2)
| shape: (3,)
| Series: 'a' [f64]
| [
| 4.0
| 5.0
| 9.0
| ]
|
| A horizontal minimum operation:
|
| >>> df = pl.DataFrame({"a": [2, 1, 3], "b": [1, 2, 3], "c": [1.0, 2.0, 3.0]})
| >>> df.fold(lambda s1, s2: s1.zip_with(s1 < s2, s2))
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.0
| 1.0
| 3.0
| ]
|
| A horizontal string concatenation:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["foo", "bar", None],
| ... "b": [1, 2, 3],
| ... "c": [1.0, 2.0, 3.0],
| ... }
| ... )
| >>> df.fold(lambda s1, s2: s1 + s2)
| shape: (3,)
| Series: 'a' [str]
| [
| "foo11.0"
| "bar22.0"
| null
| ]
|
| A horizontal boolean or, similar to a row-wise .any():
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [False, False, True],
| ... "b": [False, True, False],
| ... }
| ... )
| >>> df.fold(lambda s1, s2: s1 | s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| true
| true
| ]
|
| Parameters
| ----------
| operation
| function that takes two `Series` and returns a `Series`.
|
| gather(self, indices: 'int | Sequence[int] | IntoExpr | Series | np.ndarray[Any, Any]', *, null_on_oob: 'bool' = False) -> 'DataFrame'
| Selects rows from this DataFrame at the given indices.
|
| .. warning::
| This functionality is experimental. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| indices
| The indices of the rows to select.
|
| null_on_oob
| If true when an index is out-of-bounds a null row will be generated
| instead of raising an error.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [2, 1, 0], "s": ["foo", "bar", "baz"]})
| >>> df.gather([2, 0, 0])
| shape: (3, 2)
| ┌─────┬─────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 0 ┆ baz │
| │ 2 ┆ foo │
| │ 2 ┆ foo │
| └─────┴─────┘
|
| >>> df.gather([0, 10, 1], null_on_oob=True)
| shape: (3, 2)
| ┌──────┬──────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞══════╪══════╡
| │ 2 ┆ foo │
| │ null ┆ null │
| │ 1 ┆ bar │
| └──────┴──────┘
|
| gather_every(self, n: 'int', offset: 'int' = 0) -> 'DataFrame'
| Take every nth row in the DataFrame and return as a new DataFrame.
|
| Parameters
| ----------
| n
| Gather every *n*-th row.
| offset
| Starting index.
|
| Examples
| --------
| >>> s = pl.DataFrame({"a": [1, 2, 3, 4], "b": [5, 6, 7, 8]})
| >>> s.gather_every(2)
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 5 │
| │ 3 ┆ 7 │
| └─────┴─────┘
|
| >>> s.gather_every(2, offset=1)
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 6 │
| │ 4 ┆ 8 │
| └─────┴─────┘
|
| get_column(self, name: 'str', *, default: 'Any | NoDefault' = <no_default>) -> 'Series | Any'
| Get a single column by name.
|
| Parameters
| ----------
| name
| String name of the column to retrieve.
| default
| Value to return if the column does not exist; if not explicitly set and
| the column is not present a `ColumnNotFoundError` exception is raised.
|
| Returns
| -------
| Series (or arbitrary default value, if specified).
|
| See Also
| --------
| to_series
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df.get_column("foo")
| shape: (3,)
| Series: 'foo' [i64]
| [
| 1
| 2
| 3
| ]
|
| Missing column handling; can optionally provide an arbitrary default value
| to the method (otherwise a `ColumnNotFoundError` exception is raised).
|
| >>> df.get_column("baz", default=pl.Series("baz", ["?", "?", "?"]))
| shape: (3,)
| Series: 'baz' [str]
| [
| "?"
| "?"
| "?"
| ]
| >>> res = df.get_column("baz", default=None)
| >>> res is None
| True
|
| get_column_index(self, name: 'str') -> 'int'
| Find the index of a column by name.
|
| Parameters
| ----------
| name
| Name of the column to find.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"foo": [1, 2, 3], "bar": [6, 7, 8], "ham": ["a", "b", "c"]}
| ... )
| >>> df.get_column_index("ham")
| 2
| >>> df.get_column_index("sandwich") # doctest: +SKIP
| ColumnNotFoundError: sandwich
|
| get_columns(self) -> 'list[Series]'
| Get the DataFrame as a List of Series.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df.get_columns()
| [shape: (3,)
| Series: 'foo' [i64]
| [
| 1
| 2
| 3
| ], shape: (3,)
| Series: 'bar' [i64]
| [
| 4
| 5
| 6
| ]]
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> df.get_columns()
| [shape: (4,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| ], shape: (4,)
| Series: 'b' [f64]
| [
| 0.5
| 4.0
| 10.0
| 13.0
| ], shape: (4,)
| Series: 'c' [bool]
| [
| true
| true
| false
| true
| ]]
|
| glimpse(self, *, max_items_per_column: 'int' = 10, max_colname_length: 'int' = 50, return_type: "Literal['frame', 'self', 'string'] | None" = None) -> 'str | DataFrame | None'
| Return a dense preview of the DataFrame.
|
| The formatting shows one line per column so that wide dataframes display
| cleanly. Each line shows the column name, the data type, and the first
| few values.
|
| .. versionchanged:: 1.35.0
| The `return_as_string` parameter was renamed `return_type` and now accepts
| string values `'string'` and `'frame'` instead of boolean True or False.
|
| Parameters
| ----------
| max_items_per_column
| Maximum number of items to show per column.
| max_colname_length
| Maximum length of the displayed column names; values that exceed
| this value are truncated with a trailing ellipsis.
| return_type
| Modify the return format:
|
| - `None` (default): Print the glimpse output to stdout, returning `None`.
| - `"self"`: Print the glimpse output to stdout, returning the *original* frame.
| - `"frame"`: Return the glimpse output as a new DataFrame.
| - `"string"`: Return the glimpse output as a string.
|
| See Also
| --------
| describe, head, tail
|
| Examples
| --------
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1.0, 2.8, 3.0],
| ... "b": [4, 5, None],
| ... "c": [True, False, True],
| ... "d": [None, "b", "c"],
| ... "e": ["usd", "eur", None],
| ... "f": [date(2020, 1, 1), date(2021, 1, 2), date(2022, 1, 1)],
| ... }
| ... )
|
| Print glimpse-formatted output to stdout, returning `None`:
|
| >>> res = df.glimpse()
| Rows: 3
| Columns: 6
| $ a <f64> 1.0, 2.8, 3.0
| $ b <i64> 4, 5, null
| $ c <bool> True, False, True
| $ d <str> null, 'b', 'c'
| $ e <str> 'usd', 'eur', null
| $ f <date> 2020-01-01, 2021-01-02, 2022-01-01
| >>> res is None
| True
|
| Return the glimpse output as a string:
|
| >>> res = df.glimpse(return_type="string")
| >>> isinstance(res, str)
| True
|
| Return the glimpse output as a DataFrame:
|
| >>> df.glimpse(return_type="frame")
| shape: (6, 3)
| ┌────────┬───────┬─────────────────────────────────┐
| │ column ┆ dtype ┆ values │
| │ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ list[str] │
| ╞════════╪═══════╪═════════════════════════════════╡
| │ a ┆ f64 ┆ ["1.0", "2.8", "3.0"] │
| │ b ┆ i64 ┆ ["4", "5", null] │
| │ c ┆ bool ┆ ["True", "False", "True"] │
| │ d ┆ str ┆ [null, "'b'", "'c'"] │
| │ e ┆ str ┆ ["'usd'", "'eur'", null] │
| │ f ┆ date ┆ ["2020-01-01", "2021-01-02", "… │
| └────────┴───────┴─────────────────────────────────┘
|
| Print glimpse-formatted output to stdout, returning the *original* frame:
|
| >>> res = df.glimpse(return_type="self")
| Rows: 3
| Columns: 6
| $ a <f64> 1.0, 2.8, 3.0
| $ b <i64> 4, 5, null
| $ c <bool> True, False, True
| $ d <str> null, 'b', 'c'
| $ e <str> 'usd', 'eur', null
| $ f <date> 2020-01-01, 2021-01-02, 2022-01-01
| >>> res
| shape: (3, 6)
| ┌─────┬──────┬───────┬──────┬──────┬────────────┐
| │ a ┆ b ┆ c ┆ d ┆ e ┆ f │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ i64 ┆ bool ┆ str ┆ str ┆ date │
| ╞═════╪══════╪═══════╪══════╪══════╪════════════╡
| │ 1.0 ┆ 4 ┆ true ┆ null ┆ usd ┆ 2020-01-01 │
| │ 2.8 ┆ 5 ┆ false ┆ b ┆ eur ┆ 2021-01-02 │
| │ 3.0 ┆ null ┆ true ┆ c ┆ null ┆ 2022-01-01 │
| └─────┴──────┴───────┴──────┴──────┴────────────┘
|
| group_by(self, *by: 'IntoExpr | Iterable[IntoExpr]', maintain_order: 'bool' = False, **named_by: 'IntoExpr') -> 'GroupBy'
| Start a group by operation.
|
| Parameters
| ----------
| *by
| Column(s) to group by. Accepts expression input. Strings are parsed as
| column names.
| maintain_order
| Ensure that the order of the groups is consistent with the input data.
| This is slower than a default group by.
| Settings this to `True` blocks the possibility
| to run on the streaming engine.
|
| .. note::
| Within each group, the order of rows is always preserved, regardless
| of this argument.
| **named_by
| Additional columns to group by, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| GroupBy
| Object which can be used to perform aggregations.
|
| Examples
| --------
| Group by one column and call `agg` to compute the grouped sum of another
| column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "c"],
| ... "b": [1, 2, 1, 3, 3],
| ... "c": [5, 4, 3, 2, 1],
| ... }
| ... )
| >>> df.group_by("a").agg(pl.col("b").sum()) # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 2 │
| │ b ┆ 5 │
| │ c ┆ 3 │
| └─────┴─────┘
|
| Set `maintain_order=True` to ensure the order of the groups is consistent with
| the input.
|
| >>> df.group_by("a", maintain_order=True).agg(pl.col("c"))
| shape: (3, 2)
| ┌─────┬───────────┐
| │ a ┆ c │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═════╪═══════════╡
| │ a ┆ [5, 3] │
| │ b ┆ [4, 2] │
| │ c ┆ [1] │
| └─────┴───────────┘
|
| Group by multiple columns by passing a list of column names.
|
| >>> df.group_by(["a", "b"]).agg(pl.max("c")) # doctest: +IGNORE_RESULT
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘
|
| Or use positional arguments to group by multiple columns in the same way.
| Expressions are also accepted.
|
| >>> df.group_by("a", pl.col("b") // 2).agg(pl.col("c").mean()) # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ f64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 0 ┆ 4.0 │
| │ b ┆ 1 ┆ 3.0 │
| │ c ┆ 1 ┆ 1.0 │
| └─────┴─────┴─────┘
|
| The `GroupBy` object returned by this method is iterable, returning the name
| and data of each group.
|
| >>> for name, data in df.group_by("a"): # doctest: +SKIP
| ... print(name)
| ... print(data)
| ('a',)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ a ┆ 1 ┆ 3 │
| └─────┴─────┴─────┘
| ('b',)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| └─────┴─────┴─────┘
| ('c',)
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘
|
| group_by_dynamic(self, index_column: 'IntoExpr', *, every: 'str | timedelta', period: 'str | timedelta | None' = None, offset: 'str | timedelta | None' = None, include_boundaries: 'bool' = False, closed: 'ClosedInterval' = 'left', label: 'Label' = 'left', group_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, start_by: 'StartBy' = 'window') -> 'DynamicGroupBy'
| Group based on a time value (or index value of type Int32, Int64).
|
| Time windows are calculated and rows are assigned to windows. Different from a
| normal group by is that a row can be member of multiple groups.
| By default, the windows look like:
|
| - [start, start + period)
| - [start + every, start + every + period)
| - [start + 2*every, start + 2*every + period)
| - ...
|
| where `start` is determined by `start_by`, `offset`, `every`, and the earliest
| datapoint. See the `start_by` argument description for details.
|
| .. warning::
| The index column must be sorted in ascending order. If `group_by` is passed, then
| the index column must be sorted in ascending order within each group.
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order (or, if `group_by` is specified,
| then it must be sorted in ascending order within each group).
|
| In case of a dynamic group by on indices, dtype needs to be one of
| {Int32, Int64}. Note that Int32 gets temporarily cast to Int64, so if
| performance matters use an Int64 column.
| every
| interval of the window
| period
| length of the window, if None it will equal 'every'
| offset
| offset of the window, does not take effect if `start_by` is 'datapoint'.
| Defaults to zero.
| include_boundaries
| Add the lower and upper bound of the window to the "_lower_boundary" and
| "_upper_boundary" columns. This will impact performance because it's harder to
| parallelize
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
| label : {'left', 'right', 'datapoint'}
| Define which label to use for the window:
|
| - 'left': lower boundary of the window
| - 'right': upper boundary of the window
| - 'datapoint': the first value of the index column in the given window.
| If you don't need the label to be at one of the boundaries, choose this
| option for maximum performance
| group_by
| Also group by this column/these columns
| start_by : {'window', 'datapoint', 'monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday', 'sunday'}
| The strategy to determine the start of the first window by.
|
| * 'window': Start by taking the earliest timestamp, truncating it with
| `every`, and then adding `offset`.
| Note that weekly windows start on Monday.
| * 'datapoint': Start from the first encountered data point.
| * a day of the week (only takes effect if `every` contains `'w'`):
|
| * 'monday': Start the window on the Monday before the first data point.
| * 'tuesday': Start the window on the Tuesday before the first data point.
| * ...
| * 'sunday': Start the window on the Sunday before the first data point.
|
| The resulting window is then shifted back until the earliest datapoint
| is in or in front of it.
|
| Returns
| -------
| DynamicGroupBy
| Object you can call `.agg` on to aggregate by groups, the result
| of which will be sorted by `index_column` (but note that if `group_by` columns are
| passed, it will only be sorted within each group).
|
| See Also
| --------
| rolling
|
| Notes
| -----
| 1) If you're coming from pandas, then
|
| .. code-block:: python
|
| # polars
| df.group_by_dynamic("ts", every="1d").agg(pl.col("value").sum())
|
| is equivalent to
|
| .. code-block:: python
|
| # pandas
| df.set_index("ts").resample("D")["value"].sum().reset_index()
|
| though note that, unlike pandas, polars doesn't add extra rows for empty
| windows. If you need `index_column` to be evenly spaced, then please combine
| with :func:`DataFrame.upsample`.
|
| 2) The `every`, `period` and `offset` arguments are created with
| the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them (except in `every`):
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| In case of a group_by_dynamic on an integer column, the windows are defined by:
|
| - "1i" # length 1
| - "10i" # length 10
|
| Examples
| --------
| >>> from datetime import datetime
| >>> df = pl.DataFrame(
| ... {
| ... "time": pl.datetime_range(
| ... start=datetime(2021, 12, 16),
| ... end=datetime(2021, 12, 16, 3),
| ... interval="30m",
| ... eager=True,
| ... ),
| ... "n": range(7),
| ... }
| ... )
| >>> df
| shape: (7, 2)
| ┌─────────────────────┬─────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ i64 │
| ╞═════════════════════╪═════╡
| │ 2021-12-16 00:00:00 ┆ 0 │
| │ 2021-12-16 00:30:00 ┆ 1 │
| │ 2021-12-16 01:00:00 ┆ 2 │
| │ 2021-12-16 01:30:00 ┆ 3 │
| │ 2021-12-16 02:00:00 ┆ 4 │
| │ 2021-12-16 02:30:00 ┆ 5 │
| │ 2021-12-16 03:00:00 ┆ 6 │
| └─────────────────────┴─────┘
|
| Group by windows of 1 hour.
|
| >>> df.group_by_dynamic("time", every="1h", closed="right").agg(pl.col("n"))
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-15 23:00:00 ┆ [0] │
| │ 2021-12-16 00:00:00 ┆ [1, 2] │
| │ 2021-12-16 01:00:00 ┆ [3, 4] │
| │ 2021-12-16 02:00:00 ┆ [5, 6] │
| └─────────────────────┴───────────┘
|
| The window boundaries can also be added to the aggregation result
|
| >>> df.group_by_dynamic(
| ... "time", every="1h", include_boundaries=True, closed="right"
| ... ).agg(pl.col("n").mean())
| shape: (4, 4)
| ┌─────────────────────┬─────────────────────┬─────────────────────┬─────┐
| │ _lower_boundary ┆ _upper_boundary ┆ time ┆ n │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ f64 │
| ╞═════════════════════╪═════════════════════╪═════════════════════╪═════╡
| │ 2021-12-15 23:00:00 ┆ 2021-12-16 00:00:00 ┆ 2021-12-15 23:00:00 ┆ 0.0 │
| │ 2021-12-16 00:00:00 ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 00:00:00 ┆ 1.5 │
| │ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ 3.5 │
| │ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ 5.5 │
| └─────────────────────┴─────────────────────┴─────────────────────┴─────┘
|
| When closed="left", the window excludes the right end of interval:
| [lower_bound, upper_bound)
|
| >>> df.group_by_dynamic("time", every="1h", closed="left").agg(pl.col("n"))
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-16 00:00:00 ┆ [0, 1] │
| │ 2021-12-16 01:00:00 ┆ [2, 3] │
| │ 2021-12-16 02:00:00 ┆ [4, 5] │
| │ 2021-12-16 03:00:00 ┆ [6] │
| └─────────────────────┴───────────┘
|
| When closed="both" the time values at the window boundaries belong to 2 groups.
|
| >>> df.group_by_dynamic("time", every="1h", closed="both").agg(pl.col("n"))
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-16 00:00:00 ┆ [0, 1, 2] │
| │ 2021-12-16 01:00:00 ┆ [2, 3, 4] │
| │ 2021-12-16 02:00:00 ┆ [4, 5, 6] │
| │ 2021-12-16 03:00:00 ┆ [6] │
| └─────────────────────┴───────────┘
|
| Dynamic group bys can also be combined with grouping on normal keys
|
| >>> df = df.with_columns(groups=pl.Series(["a", "a", "a", "b", "b", "a", "a"]))
| >>> df
| shape: (7, 3)
| ┌─────────────────────┬─────┬────────┐
| │ time ┆ n ┆ groups │
| │ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ str │
| ╞═════════════════════╪═════╪════════╡
| │ 2021-12-16 00:00:00 ┆ 0 ┆ a │
| │ 2021-12-16 00:30:00 ┆ 1 ┆ a │
| │ 2021-12-16 01:00:00 ┆ 2 ┆ a │
| │ 2021-12-16 01:30:00 ┆ 3 ┆ b │
| │ 2021-12-16 02:00:00 ┆ 4 ┆ b │
| │ 2021-12-16 02:30:00 ┆ 5 ┆ a │
| │ 2021-12-16 03:00:00 ┆ 6 ┆ a │
| └─────────────────────┴─────┴────────┘
| >>> df.group_by_dynamic(
| ... "time",
| ... every="1h",
| ... closed="both",
| ... group_by="groups",
| ... include_boundaries=True,
| ... ).agg(pl.col("n"))
| shape: (6, 5)
| ┌────────┬─────────────────────┬─────────────────────┬─────────────────────┬───────────┐
| │ groups ┆ _lower_boundary ┆ _upper_boundary ┆ time ┆ n │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ list[i64] │
| ╞════════╪═════════════════════╪═════════════════════╪═════════════════════╪═══════════╡
| │ a ┆ 2021-12-16 00:00:00 ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 00:00:00 ┆ [0, 1, 2] │
| │ a ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ [2] │
| │ a ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ [5, 6] │
| │ a ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 04:00:00 ┆ 2021-12-16 03:00:00 ┆ [6] │
| │ b ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ [3, 4] │
| │ b ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ [4] │
| └────────┴─────────────────────┴─────────────────────┴─────────────────────┴───────────┘
|
| Dynamic group by on an index column
|
| >>> df = pl.DataFrame(
| ... {
| ... "idx": pl.int_range(0, 6, eager=True),
| ... "A": ["A", "A", "B", "B", "B", "C"],
| ... }
| ... )
| >>> (
| ... df.group_by_dynamic(
| ... "idx",
| ... every="2i",
| ... period="3i",
| ... include_boundaries=True,
| ... closed="right",
| ... ).agg(pl.col("A").alias("A_agg_list"))
| ... )
| shape: (4, 4)
| ┌─────────────────┬─────────────────┬─────┬─────────────────┐
| │ _lower_boundary ┆ _upper_boundary ┆ idx ┆ A_agg_list │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ list[str] │
| ╞═════════════════╪═════════════════╪═════╪═════════════════╡
| │ -2 ┆ 1 ┆ -2 ┆ ["A", "A"] │
| │ 0 ┆ 3 ┆ 0 ┆ ["A", "B", "B"] │
| │ 2 ┆ 5 ┆ 2 ┆ ["B", "B", "C"] │
| │ 4 ┆ 7 ┆ 4 ┆ ["C"] │
| └─────────────────┴─────────────────┴─────┴─────────────────┘
|
| hash_rows(self, seed: 'int' = 0, seed_1: 'int | None' = None, seed_2: 'int | None' = None, seed_3: 'int | None' = None) -> 'Series'
| Hash and combine the rows in this DataFrame.
|
| The hash value is of type `UInt64`.
|
| Parameters
| ----------
| seed
| Random seed parameter. Defaults to 0.
| seed_1
| Random seed parameter. Defaults to `seed` if not set.
| seed_2
| Random seed parameter. Defaults to `seed` if not set.
| seed_3
| Random seed parameter. Defaults to `seed` if not set.
|
| Notes
| -----
| This implementation of `hash_rows` does not guarantee stable results
| across different Polars versions. Its stability is only guaranteed within a
| single version.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, None, 3, 4],
| ... "ham": ["a", "b", None, "d"],
| ... }
| ... )
| >>> df.hash_rows(seed=42) # doctest: +IGNORE_RESULT
| shape: (4,)
| Series: '' [u64]
| [
| 10783150408545073287
| 1438741209321515184
| 10047419486152048166
| 2047317070637311557
| ]
|
| head(self, n: 'int' = 5) -> 'DataFrame'
| Get the first `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return. If a negative value is passed, return all rows
| except the last `abs(n)`.
|
| See Also
| --------
| tail, glimpse, slice
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> df.head(3)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Pass a negative value to get all rows `except` the last `abs(n)`.
|
| >>> df.head(-3)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| └─────┴─────┴─────┘
|
| hstack(self, columns: 'list[Series] | DataFrame', *, in_place: 'bool' = False) -> 'DataFrame'
| Return a new DataFrame grown horizontally by stacking multiple Series to it.
|
| Parameters
| ----------
| columns
| Series to stack.
| in_place
| Modify in place.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> x = pl.Series("apple", [10, 20, 30])
| >>> df.hstack([x])
| shape: (3, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6 ┆ a ┆ 10 │
| │ 2 ┆ 7 ┆ b ┆ 20 │
| │ 3 ┆ 8 ┆ c ┆ 30 │
| └─────┴─────┴─────┴───────┘
|
| insert_column(self, index: 'int', column: 'IntoExprColumn') -> 'DataFrame'
| Insert a Series (or expression) at a certain column index.
|
| This operation is in place.
|
| Parameters
| ----------
| index
| Index at which to insert the new column.
| column
| `Series` or expression to insert.
|
| Examples
| --------
| Insert a new Series column at the given index:
|
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> s = pl.Series("baz", [97, 98, 99])
| >>> df.insert_column(1, s)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ baz ┆ bar │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 97 ┆ 4 │
| │ 2 ┆ 98 ┆ 5 │
| │ 3 ┆ 99 ┆ 6 │
| └─────┴─────┴─────┘
|
| Insert a new expression column at the given index:
|
| >>> df = pl.DataFrame(
| ... {"a": [2, 4, 2], "b": [0.5, 4, 10], "c": ["xx", "yy", "zz"]}
| ... )
| >>> expr = (pl.col("b") / pl.col("a")).alias("b_div_a")
| >>> df.insert_column(2, expr)
| shape: (3, 4)
| ┌─────┬──────┬─────────┬─────┐
| │ a ┆ b ┆ b_div_a ┆ c │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 ┆ str │
| ╞═════╪══════╪═════════╪═════╡
| │ 2 ┆ 0.5 ┆ 0.25 ┆ xx │
| │ 4 ┆ 4.0 ┆ 1.0 ┆ yy │
| │ 2 ┆ 10.0 ┆ 5.0 ┆ zz │
| └─────┴──────┴─────────┴─────┘
|
| interpolate(self) -> 'DataFrame'
| Interpolate intermediate values. The interpolation method is linear.
|
| Nulls at the beginning and end of the series remain null.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, None, 9, 10],
| ... "bar": [6, 7, 9, None],
| ... "baz": [1, None, None, 9],
| ... }
| ... )
| >>> df.interpolate()
| shape: (4, 3)
| ┌──────┬──────┬──────────┐
| │ foo ┆ bar ┆ baz │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪══════════╡
| │ 1.0 ┆ 6.0 ┆ 1.0 │
| │ 5.0 ┆ 7.0 ┆ 3.666667 │
| │ 9.0 ┆ 9.0 ┆ 6.333333 │
| │ 10.0 ┆ null ┆ 9.0 │
| └──────┴──────┴──────────┘
|
| is_duplicated(self) -> 'Series'
| Get a mask of all duplicated rows in this DataFrame.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 1],
| ... "b": ["x", "y", "z", "x"],
| ... }
| ... )
| >>> df.is_duplicated()
| shape: (4,)
| Series: '' [bool]
| [
| true
| false
| false
| true
| ]
|
| This mask can be used to visualize the duplicated lines like this:
|
| >>> df.filter(df.is_duplicated())
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ x │
| │ 1 ┆ x │
| └─────┴─────┘
|
| is_empty(self) -> 'bool'
| Returns `True` if the DataFrame contains no rows.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df.is_empty()
| False
| >>> df.filter(pl.col("foo") > 99).is_empty()
| True
|
| is_sorted(self, by: 'str | Iterable[str]', *more_by: 'str', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False) -> 'bool'
| Check whether the DataFrame is sorted by the given columns.
|
| Parameters
| ----------
| by
| Column name(s) to check.
| *more_by
| Additional column names.
| descending
| Sort in descending order. When sorting by multiple columns, can be
| specified per column by passing a sequence of booleans.
| nulls_last
| Place null values last. When sorting by multiple columns, can be
| specified per column by passing a sequence of booleans.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [5, 4, 3]})
| >>> df.is_sorted("a")
| True
| >>> df.is_sorted("b", descending=True)
| True
| >>> df.is_sorted("a", "b")
| True
|
| is_unique(self) -> 'Series'
| Get a mask of all unique rows in this DataFrame.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 1],
| ... "b": ["x", "y", "z", "x"],
| ... }
| ... )
| >>> df.is_unique()
| shape: (4,)
| Series: '' [bool]
| [
| false
| true
| true
| false
| ]
|
| This mask can be used to visualize the unique lines like this:
|
| >>> df.filter(df.is_unique())
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 2 ┆ y │
| │ 3 ┆ z │
| └─────┴─────┘
|
| item(self, row: 'int | None' = None, column: 'int | str | None' = None) -> 'Any'
| Return the DataFrame as a scalar, or return the element at the given row/column.
|
| Parameters
| ----------
| row
| Optional row index.
| column
| Optional column index or name.
|
| See Also
| --------
| row : Get the values of a single row, either by index or by predicate.
|
| Notes
| -----
| If row/col not provided, this is equivalent to `df[0,0]`, with a check that
| the shape is (1,1). With row/col, this is equivalent to `df[row,col]`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
| >>> df.select((pl.col("a") * pl.col("b")).sum()).item()
| 32
| >>> df.item(1, 1)
| 5
| >>> df.item(2, "b")
| 6
|
| iter_columns(self) -> 'Iterator[Series]'
| Returns an iterator over the columns of this DataFrame.
|
| Yields
| ------
| Series
|
| Notes
| -----
| Consider whether you can use :func:`all` instead.
| If you can, it will be more efficient.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> [s.name for s in df.iter_columns()]
| ['a', 'b']
|
| If you're using this to modify a dataframe's columns, e.g.
|
| >>> # Do NOT do this
| >>> pl.DataFrame(column * 2 for column in df.iter_columns())
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 4 │
| │ 6 ┆ 8 │
| │ 10 ┆ 12 │
| └─────┴─────┘
|
| then consider whether you can use :func:`all` instead:
|
| >>> df.select(pl.all() * 2)
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 4 │
| │ 6 ┆ 8 │
| │ 10 ┆ 12 │
| └─────┴─────┘
|
| iter_rows(self, *, named: 'bool' = False, buffer_size: 'int' = 512) -> 'Iterator[tuple[Any, ...]] | Iterator[dict[str, Any]]'
| Returns an iterator over the DataFrame of rows of python-native values.
|
| Parameters
| ----------
| named
| Return dictionaries instead of tuples. The dictionaries are a mapping of
| column name to row value. This is more expensive than returning a regular
| tuple, but allows for accessing values by column name.
| buffer_size
| Determines the number of rows that are buffered internally while iterating
| over the data; you should only modify this in very specific cases where the
| default value is determined not to be a good fit to your access pattern, as
| the speedup from using the buffer is significant (~2-4x). Setting this
| value to zero disables row buffering (not recommended).
|
| Notes
| -----
| If you have `ns`-precision temporal values you should be aware that Python
| natively only supports up to `μs`-precision; `ns`-precision values will be
| truncated to microseconds on conversion to Python. If this matters to your
| use-case you should export to a different format (such as Arrow or NumPy).
|
| Warnings
| --------
| Row iteration is not optimal as the underlying data is stored in columnar form;
| where possible, prefer export via one of the dedicated export/output methods
| that deals with columnar data.
|
| Yields
| ------
| iterator of tuples (default) or dictionaries (if named) of python row values
|
| See Also
| --------
| rows : Materialises all frame data as a list of rows (potentially expensive).
| rows_by_key : Materialises frame data as a key-indexed dictionary.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> [row[0] for row in df.iter_rows()]
| [1, 3, 5]
| >>> [row["b"] for row in df.iter_rows(named=True)]
| [2, 4, 6]
|
| iter_slices(self, n_rows: 'int' = 10000) -> 'Iterator[DataFrame]'
| Returns a non-copying iterator of slices over the underlying DataFrame.
|
| Parameters
| ----------
| n_rows
| Determines the number of rows contained in each DataFrame slice.
|
| Examples
| --------
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... data={
| ... "a": range(17_500),
| ... "b": date(2023, 1, 1),
| ... "c": "klmnoopqrstuvwxyz",
| ... },
| ... schema_overrides={"a": pl.Int32},
| ... )
| >>> for idx, frame in enumerate(df.iter_slices()):
| ... print(f"{type(frame).__name__}:[{idx}]:{len(frame)}")
| DataFrame:[0]:10000
| DataFrame:[1]:7500
|
| Using `iter_slices` is an efficient way to chunk-iterate over DataFrames and
| any supported frame export/conversion types; for example, as RecordBatches:
|
| >>> for frame in df.iter_slices(n_rows=15_000):
| ... record_batch = frame.to_arrow().to_batches()[0]
| ... print(f"{record_batch.schema}\n<< {len(record_batch)}")
| a: int32
| b: date32[day]
| c: large_string
| << 15000
| a: int32
| b: date32[day]
| c: large_string
| << 2500
|
| See Also
| --------
| iter_rows : Row iterator over frame data (does not materialise all rows).
| partition_by : Split into multiple DataFrames, partitioned by groups.
|
| join(self, other: 'DataFrame', on: 'str | Expr | Sequence[str | Expr] | None' = None, how: 'JoinStrategy' = 'inner', *, left_on: 'str | Expr | Sequence[str | Expr] | None' = None, right_on: 'str | Expr | Sequence[str | Expr] | None' = None, suffix: 'str' = '_right', validate: 'JoinValidation' = 'm:m', nulls_equal: 'bool' = False, coalesce: 'bool | None' = None, maintain_order: 'MaintainOrderJoin | None' = None, build_side: 'JoinBuildSide' = 'auto') -> 'DataFrame'
| Join in SQL-like fashion.
|
| .. versionchanged:: 1.24
| The `join_nulls` parameter was renamed `nulls_equal`.
|
| Parameters
| ----------
| other
| DataFrame to join with.
| on
| Name(s) of the join columns in both DataFrames. If set, `left_on` and
| `right_on` should be None. This should not be specified if `how='cross'`.
| how : {'inner', 'left', 'right', 'full', 'semi', 'anti', 'cross'}
| Join strategy.
|
| .. list-table ::
| :header-rows: 0
|
| * - **inner**
| - *(Default)* Returns rows that have matching values in both tables.
| * - **left**
| - Returns all rows from the left table, and the matched rows from
| the right table.
| * - **right**
| - Returns all rows from the right table, and the matched rows from
| the left table.
| * - **full**
| - Returns all rows from both tables, joining matching rows and
| filling non-matches with null values.
| * - **cross**
| - Returns the Cartesian product of rows from both tables
| * - **semi**
| - Returns rows from the left table that have a match in the right
| table. Does not return columns from the right table.
| * - **anti**
| - Returns rows from the left table that have no match in the right
| table. Does not return columns from the right table.
|
| left_on
| Name(s) of the left join column(s).
| right_on
| Name(s) of the right join column(s).
| suffix
| Suffix to append to columns with a duplicate name.
| validate: {'m:m', 'm:1', '1:m', '1:1'}
| Checks if join is of specified type.
|
| .. list-table ::
| :header-rows: 0
|
| * - **m:m**
| - *(Default)* Many-to-many (default). Does not result in checks.
| * - **1:1**
| - One-to-one. Checks if join keys are unique in both left and
| right datasets.
| * - **1:m**
| - One-to-many. Checks if join keys are unique in left dataset.
| * - **m:1**
| - Many-to-one. Check if join keys are unique in right dataset.
|
| .. note::
| This is currently not supported by the streaming engine.
|
| nulls_equal
| Join on null values. By default null values will never produce matches.
| coalesce
| Coalescing behavior (merging of join columns).
|
| .. list-table ::
| :header-rows: 0
|
| * - **None**
| - *(Default)* Coalesce unless `how='full'` is specified.
| * - **True**
| - Always coalesce join columns.
| * - **False**
| - Never coalesce join columns.
|
| .. note::
| Joining on any other expressions than `col`
| will turn off coalescing.
| maintain_order : {'none', 'left', 'right', 'left_right', 'right_left'}
| Which DataFrame row order to preserve, if any.
| Do not rely on any observed ordering without explicitly setting this
| parameter, as your code may break in a future release.
| Not specifying any ordering can improve performance.
|
| .. list-table ::
| :header-rows: 0
|
| * - **none**
| - *(Default)* No specific ordering is desired. The ordering might
| differ across Polars versions or even between different runs.
| * - **left**
| - Preserves the order of the left DataFrame.
| * - **right**
| - Preserves the order of the right DataFrame.
| * - **left_right**
| - First preserves the order of the left DataFrame, then the right.
| * - **right_left**
| - First preserves the order of the right DataFrame, then the left.
| build_side: {'auto', 'prefer_left', 'prefer_right', 'force_left', 'force_right'}
| Which side of the join will be used as the build side. This side will be
| likely be held in memory as a hash table. Note that unless a `force_`
| variant is chosen, the chosen side might differ across Polars versions or
| even between different runs.
|
| .. list-table ::
| :header-rows: 0
|
| * - **auto**
| - *(Default)* Let Polars figure out the build side.
| * - **prefer_left**
| - Unless there's a very good reason to believe that the right side is
| smaller, use the left side.
| * - **prefer_right**
| - Unless there's a very good reason to believe that the left side is
| smaller, use the right side.
| * - **force_left**
| - Always use the left side.
| * - **force_right**
| - Always use the right side.
|
| .. warning::
| This functionality is considered **experimental**. It may be removed or
| changed at any point without it being considered a breaking change.
|
|
| See Also
| --------
| join_asof
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> other_df = pl.DataFrame(
| ... {
| ... "apple": ["x", "y", "z"],
| ... "ham": ["a", "b", "d"],
| ... }
| ... )
| >>> df.join(other_df, on="ham")
| shape: (2, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| └─────┴─────┴─────┴───────┘
|
| >>> df.join(other_df, on="ham", how="full")
| shape: (4, 5)
| ┌──────┬──────┬──────┬───────┬───────────┐
| │ foo ┆ bar ┆ ham ┆ apple ┆ ham_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞══════╪══════╪══════╪═══════╪═══════════╡
| │ 1 ┆ 6.0 ┆ a ┆ x ┆ a │
| │ 2 ┆ 7.0 ┆ b ┆ y ┆ b │
| │ null ┆ null ┆ null ┆ z ┆ d │
| │ 3 ┆ 8.0 ┆ c ┆ null ┆ null │
| └──────┴──────┴──────┴───────┴───────────┘
|
| >>> df.join(other_df, on="ham", how="full", coalesce=True)
| shape: (4, 4)
| ┌──────┬──────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞══════╪══════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| │ null ┆ null ┆ d ┆ z │
| │ 3 ┆ 8.0 ┆ c ┆ null │
| └──────┴──────┴─────┴───────┘
|
| >>> df.join(other_df, on="ham", how="left")
| shape: (3, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| │ 3 ┆ 8.0 ┆ c ┆ null │
| └─────┴─────┴─────┴───────┘
|
| >>> df.join(other_df, on="ham", how="semi")
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 7.0 ┆ b │
| └─────┴─────┴─────┘
|
| >>> df.join(other_df, on="ham", how="anti")
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| >>> df.join(other_df, how="cross")
| shape: (9, 5)
| ┌─────┬─────┬─────┬───────┬───────────┐
| │ foo ┆ bar ┆ ham ┆ apple ┆ ham_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╪═══════════╡
| │ 1 ┆ 6.0 ┆ a ┆ x ┆ a │
| │ 1 ┆ 6.0 ┆ a ┆ y ┆ b │
| │ 1 ┆ 6.0 ┆ a ┆ z ┆ d │
| │ 2 ┆ 7.0 ┆ b ┆ x ┆ a │
| │ 2 ┆ 7.0 ┆ b ┆ y ┆ b │
| │ 2 ┆ 7.0 ┆ b ┆ z ┆ d │
| │ 3 ┆ 8.0 ┆ c ┆ x ┆ a │
| │ 3 ┆ 8.0 ┆ c ┆ y ┆ b │
| │ 3 ┆ 8.0 ┆ c ┆ z ┆ d │
| └─────┴─────┴─────┴───────┴───────────┘
|
| join_asof(self, other: 'DataFrame', *, left_on: 'str | None | Expr' = None, right_on: 'str | None | Expr' = None, on: 'str | None | Expr' = None, by_left: 'str | Sequence[str] | None' = None, by_right: 'str | Sequence[str] | None' = None, by: 'str | Sequence[str] | None' = None, strategy: 'AsofJoinStrategy' = 'backward', suffix: 'str' = '_right', tolerance: 'str | int | float | timedelta | None' = None, allow_parallel: 'bool' = True, force_parallel: 'bool' = False, coalesce: 'bool' = True, allow_exact_matches: 'bool' = True, check_sortedness: 'bool' = True) -> 'DataFrame'
| Perform an asof join.
|
| This is similar to a left-join except that we match on nearest key rather than
| equal keys.
|
| Both DataFrames must be sorted by the `on` key (within each `by` group, if
| specified).
|
| For each row in the left DataFrame:
|
| - A "backward" search selects the last row in the right DataFrame whose
| 'on' key is less than or equal to the left's key.
|
| - A "forward" search selects the first row in the right DataFrame whose
| 'on' key is greater than or equal to the left's key.
|
| - A "nearest" search selects the last row in the right DataFrame whose value
| is nearest to the left's key. String keys are not currently supported for a
| nearest search.
|
| The default is "backward".
|
| Parameters
| ----------
| other
| Lazy DataFrame to join with.
| left_on
| Join column of the left DataFrame.
| right_on
| Join column of the right DataFrame.
| on
| Join column of both DataFrames. If set, `left_on` and `right_on` should be
| None.
| by_left
| Join on these columns before doing asof join
| by_right
| Join on these columns before doing asof join
| by
| Join on these columns before doing asof join
| strategy : {'backward', 'forward', 'nearest'}
| Join strategy.
| suffix
| Suffix to append to columns with a duplicate name.
| tolerance
| Numeric tolerance. By setting this the join will only be done if the near
| keys are within this distance. If an asof join is done on columns of dtype
| "Date", "Datetime", "Duration" or "Time", use either a datetime.timedelta
| object or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of
| ambiguity, we follow RFC-5545 and preserve the DST fold of the original
| datetime). Similarly for "calendar week", "calendar month",
| "calendar quarter", and "calendar year".
|
| allow_parallel
| Allow the physical plan to optionally evaluate the computation of both
| DataFrames up to the join in parallel.
| force_parallel
| Force the physical plan to evaluate the computation of both DataFrames up to
| the join in parallel.
| coalesce
| Coalescing behavior (merging of `on` / `left_on` / `right_on` columns):
|
| - *True*: Always coalesce join columns.
| - *False*: Never coalesce join columns.
|
| Note that joining on any other expressions than `col`
| will turn off coalescing.
| allow_exact_matches
| Whether exact matches are valid join predicates.
|
| - If True, allow matching with the same ``on`` value
| (i.e. less-than-or-equal-to / greater-than-or-equal-to)
| - If False, don't match the same ``on`` value
| (i.e., strictly less-than / strictly greater-than).
| check_sortedness
| Check the sortedness of the asof keys. If the keys are not sorted Polars
| will error. Currently, the `in-memory` engine cannot check the sortedness
| if 'by' groups are provided. The `streaming` engine will only check the
| sortedness of the rows it processes.
|
| Examples
| --------
| >>> from datetime import date
| >>> gdp = pl.DataFrame(
| ... {
| ... "date": pl.date_range(
| ... date(2016, 1, 1),
| ... date(2020, 1, 1),
| ... "1y",
| ... eager=True,
| ... ),
| ... "gdp": [4164, 4411, 4566, 4696, 4827],
| ... }
| ... )
| >>> gdp
| shape: (5, 2)
| ┌────────────┬──────┐
| │ date ┆ gdp │
| │ --- ┆ --- │
| │ date ┆ i64 │
| ╞════════════╪══════╡
| │ 2016-01-01 ┆ 4164 │
| │ 2017-01-01 ┆ 4411 │
| │ 2018-01-01 ┆ 4566 │
| │ 2019-01-01 ┆ 4696 │
| │ 2020-01-01 ┆ 4827 │
| └────────────┴──────┘
|
| >>> population = pl.DataFrame(
| ... {
| ... "date": [date(2016, 3, 1), date(2018, 8, 1), date(2019, 1, 1)],
| ... "population": [82.19, 82.66, 83.12],
| ... }
| ... ).sort("date")
| >>> population
| shape: (3, 2)
| ┌────────────┬────────────┐
| │ date ┆ population │
| │ --- ┆ --- │
| │ date ┆ f64 │
| ╞════════════╪════════════╡
| │ 2016-03-01 ┆ 82.19 │
| │ 2018-08-01 ┆ 82.66 │
| │ 2019-01-01 ┆ 83.12 │
| └────────────┴────────────┘
|
| Note how the dates don't quite match. If we join them using `join_asof` and
| `strategy='backward'`, then each date from `population` which doesn't have an
| exact match is matched with the closest earlier date from `gdp`:
|
| >>> population.join_asof(gdp, on="date", strategy="backward")
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 4566 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2016-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2018-01-01` from `gdp`.
|
| You can verify this by passing `coalesce=False`:
|
| >>> population.join_asof(gdp, on="date", strategy="backward", coalesce=False)
| shape: (3, 4)
| ┌────────────┬────────────┬────────────┬──────┐
| │ date ┆ population ┆ date_right ┆ gdp │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ date ┆ i64 │
| ╞════════════╪════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 2016-01-01 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 2018-01-01 ┆ 4566 │
| │ 2019-01-01 ┆ 83.12 ┆ 2019-01-01 ┆ 4696 │
| └────────────┴────────────┴────────────┴──────┘
|
| If we instead use `strategy='forward'`, then each date from `population` which
| doesn't have an exact match is matched with the closest later date from `gdp`:
|
| >>> population.join_asof(gdp, on="date", strategy="forward")
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4411 │
| │ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2017-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2019-01-01` from `gdp`.
|
| Finally, `strategy='nearest'` gives us a mix of the two results above, as each
| date from `population` which doesn't have an exact match is matched with the
| closest date from `gdp`, regardless of whether it's earlier or later:
|
| >>> population.join_asof(gdp, on="date", strategy="nearest")
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2016-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2019-01-01` from `gdp`.
|
| The `by` argument allows joining on another column first, before the asof join.
| In this example we join by `country` first, then asof join by date, as above.
|
| >>> gdp_dates = pl.date_range( # fmt: skip
| ... date(2016, 1, 1), date(2020, 1, 1), "1y", eager=True
| ... )
| >>> gdp2 = pl.DataFrame(
| ... {
| ... "country": ["Germany"] * 5 + ["Netherlands"] * 5,
| ... "date": pl.concat([gdp_dates, gdp_dates]),
| ... "gdp": [4164, 4411, 4566, 4696, 4827, 784, 833, 914, 910, 909],
| ... }
| ... ).sort("country", "date")
| >>>
| >>> gdp2
| shape: (10, 3)
| ┌─────────────┬────────────┬──────┐
| │ country ┆ date ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 │
| ╞═════════════╪════════════╪══════╡
| │ Germany ┆ 2016-01-01 ┆ 4164 │
| │ Germany ┆ 2017-01-01 ┆ 4411 │
| │ Germany ┆ 2018-01-01 ┆ 4566 │
| │ Germany ┆ 2019-01-01 ┆ 4696 │
| │ Germany ┆ 2020-01-01 ┆ 4827 │
| │ Netherlands ┆ 2016-01-01 ┆ 784 │
| │ Netherlands ┆ 2017-01-01 ┆ 833 │
| │ Netherlands ┆ 2018-01-01 ┆ 914 │
| │ Netherlands ┆ 2019-01-01 ┆ 910 │
| │ Netherlands ┆ 2020-01-01 ┆ 909 │
| └─────────────┴────────────┴──────┘
| >>> pop2 = pl.DataFrame(
| ... {
| ... "country": ["Germany"] * 3 + ["Netherlands"] * 3,
| ... "date": [
| ... date(2016, 3, 1),
| ... date(2018, 8, 1),
| ... date(2019, 1, 1),
| ... date(2016, 3, 1),
| ... date(2018, 8, 1),
| ... date(2019, 1, 1),
| ... ],
| ... "population": [82.19, 82.66, 83.12, 17.11, 17.32, 17.40],
| ... }
| ... ).sort("country", "date")
| >>>
| >>> pop2
| shape: (6, 3)
| ┌─────────────┬────────────┬────────────┐
| │ country ┆ date ┆ population │
| │ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ f64 │
| ╞═════════════╪════════════╪════════════╡
| │ Germany ┆ 2016-03-01 ┆ 82.19 │
| │ Germany ┆ 2018-08-01 ┆ 82.66 │
| │ Germany ┆ 2019-01-01 ┆ 83.12 │
| │ Netherlands ┆ 2016-03-01 ┆ 17.11 │
| │ Netherlands ┆ 2018-08-01 ┆ 17.32 │
| │ Netherlands ┆ 2019-01-01 ┆ 17.4 │
| └─────────────┴────────────┴────────────┘
| >>> pop2.join_asof(gdp2, by="country", on="date", strategy="nearest")
| shape: (6, 4)
| ┌─────────────┬────────────┬────────────┬──────┐
| │ country ┆ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ f64 ┆ i64 │
| ╞═════════════╪════════════╪════════════╪══════╡
| │ Germany ┆ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ Germany ┆ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ Germany ┆ 2019-01-01 ┆ 83.12 ┆ 4696 │
| │ Netherlands ┆ 2016-03-01 ┆ 17.11 ┆ 784 │
| │ Netherlands ┆ 2018-08-01 ┆ 17.32 ┆ 910 │
| │ Netherlands ┆ 2019-01-01 ┆ 17.4 ┆ 910 │
| └─────────────┴────────────┴────────────┴──────┘
|
| join_where(self, other: 'DataFrame', *predicates: 'Expr | Iterable[Expr]', suffix: 'str' = '_right') -> 'DataFrame'
| Perform a join based on one or multiple (in)equality predicates.
|
| This performs an inner join, so only rows where all predicates are true
| are included in the result, and a row from either DataFrame may be included
| multiple times in the result.
|
| .. note::
| The row order of the input DataFrames is not preserved.
|
| .. warning::
| This functionality is experimental. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| other
| DataFrame to join with.
| *predicates
| (In)Equality condition to join the two tables on.
| When a column name occurs in both tables, the proper suffix must
| be applied in the predicate.
| suffix
| Suffix to append to columns with a duplicate name.
|
| Examples
| --------
| Join two dataframes together based on two predicates which get AND-ed together.
|
| >>> east = pl.DataFrame(
| ... {
| ... "id": [100, 101, 102],
| ... "dur": [120, 140, 160],
| ... "rev": [12, 14, 16],
| ... "cores": [2, 8, 4],
| ... }
| ... )
| >>> west = pl.DataFrame(
| ... {
| ... "t_id": [404, 498, 676, 742],
| ... "time": [90, 130, 150, 170],
| ... "cost": [9, 13, 15, 16],
| ... "cores": [4, 2, 1, 4],
| ... }
| ... )
| >>> east.join_where(
| ... west,
| ... pl.col("dur") < pl.col("time"),
| ... pl.col("rev") < pl.col("cost"),
| ... )
| shape: (5, 8)
| ┌─────┬─────┬─────┬───────┬──────┬──────┬──────┬─────────────┐
| │ id ┆ dur ┆ rev ┆ cores ┆ t_id ┆ time ┆ cost ┆ cores_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╪══════╪══════╪══════╪═════════════╡
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 498 ┆ 130 ┆ 13 ┆ 2 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| └─────┴─────┴─────┴───────┴──────┴──────┴──────┴─────────────┘
|
| To OR them together, use a single expression and the `|` operator.
|
| >>> east.join_where(
| ... west,
| ... (pl.col("dur") < pl.col("time")) | (pl.col("rev") < pl.col("cost")),
| ... )
| shape: (6, 8)
| ┌─────┬─────┬─────┬───────┬──────┬──────┬──────┬─────────────┐
| │ id ┆ dur ┆ rev ┆ cores ┆ t_id ┆ time ┆ cost ┆ cores_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╪══════╪══════╪══════╪═════════════╡
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 498 ┆ 130 ┆ 13 ┆ 2 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 102 ┆ 160 ┆ 16 ┆ 4 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| └─────┴─────┴─────┴───────┴──────┴──────┴──────┴─────────────┘
|
| lazy(self) -> 'LazyFrame'
| Start a lazy query from this point. This returns a `LazyFrame` object.
|
| Operations on a `LazyFrame` are not executed until this is triggered
| by calling one of:
|
| * :meth:`.collect() <polars.LazyFrame.collect>`
| (run on all data)
| * :meth:`.explain() <polars.LazyFrame.explain>`
| (print the query plan)
| * :meth:`.show_graph() <polars.LazyFrame.show_graph>`
| (show the query plan as graphviz graph)
| * :meth:`.collect_schema() <polars.LazyFrame.collect_schema>`
| (return the final frame schema)
|
| Lazy operations are recommended because they allow for query optimization and
| additional parallelism.
|
| Returns
| -------
| LazyFrame
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> df.lazy()
| <LazyFrame at ...>
|
| limit(self, n: 'int' = 5) -> 'DataFrame'
| Get the first `n` rows.
|
| Alias for :func:`DataFrame.head`.
|
| Parameters
| ----------
| n
| Number of rows to return. If a negative value is passed, return all rows
| except the last `abs(n)`.
|
| See Also
| --------
| head
|
| Examples
| --------
| Get the first 3 rows of a DataFrame.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> df.limit(3)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| map_columns(self, column_names: 'str | Sequence[str] | pl.Selector', function: 'Callable[Concatenate[Series, P], Series]', *args: 'P.args', **kwargs: 'P.kwargs') -> 'DataFrame'
| Apply eager functions to columns of a DataFrame.
|
| Users should always prefer :meth:`with_columns` unless they are using
| expressions that are only possible on `Series` and not on `Expr`. This is almost
| never the case, except for a very select few functions that cannot know the
| output datatype without looking at the data.
|
| Parameters
| ----------
| column_names
| The columns to apply the UDF to.
| function
| Callable; will receive a column series as the first parameter,
| followed by any given args/kwargs.
| *args
| Arguments to pass to the UDF.
| **kwargs
| Keyword arguments to pass to the UDF.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4], "b": ["10", "20", "30", "40"]})
| >>> df.map_columns("a", lambda s: s.shrink_dtype())
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i8 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ 10 │
| │ 2 ┆ 20 │
| │ 3 ┆ 30 │
| │ 4 ┆ 40 │
| └─────┴─────┘
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ['{"x":"a"}', None, '{"x":"b"}', None],
| ... "b": ['{"a":1, "b": true}', None, '{"a":2, "b": false}', None],
| ... }
| ... )
| >>> df.map_columns(["a", "b"], lambda s: s.str.json_decode())
| shape: (4, 2)
| ┌───────────┬───────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ struct[1] ┆ struct[2] │
| ╞═══════════╪═══════════╡
| │ {"a"} ┆ {1,true} │
| │ null ┆ null │
| │ {"b"} ┆ {2,false} │
| │ null ┆ null │
| └───────────┴───────────┘
| >>> import polars.selectors as cs
| >>> df.map_columns(cs.all(), lambda s: s.str.json_decode())
| shape: (4, 2)
| ┌───────────┬───────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ struct[1] ┆ struct[2] │
| ╞═══════════╪═══════════╡
| │ {"a"} ┆ {1,true} │
| │ null ┆ null │
| │ {"b"} ┆ {2,false} │
| │ null ┆ null │
| └───────────┴───────────┘
|
| See Also
| --------
| with_columns
|
| map_rows(self, function: 'Callable[[tuple[Any, ...]], Any]', return_dtype: 'PolarsDataType | None' = None, *, inference_size: 'int' = 256) -> 'DataFrame'
| Apply a custom/user-defined function (UDF) over the rows of the DataFrame.
|
| .. warning::
| This method is much slower than the native expressions API.
| Only use it if you cannot implement your logic otherwise.
|
| The UDF will receive each row as a tuple of values: `udf(row)`.
|
| Implementing logic using a Python function is almost always *significantly*
| slower and more memory intensive than implementing the same logic using
| the native expression API because:
|
| - The native expression engine runs in Rust; UDFs run in Python.
| - Use of Python UDFs forces the DataFrame to be materialized in memory.
| - Polars-native expressions can be parallelised (UDFs typically cannot).
| - Polars-native expressions can be logically optimised (UDFs cannot).
|
| Wherever possible you should strongly prefer the native expression API
| to achieve the best performance.
|
| Parameters
| ----------
| function
| Custom function or lambda.
| return_dtype
| Output type of the operation. If none given, Polars tries to infer the type.
| inference_size
| Only used in the case when the custom function returns rows.
| This uses the first `n` rows to determine the output schema.
|
| Notes
| -----
| * The frame-level `map_rows` cannot track column names (as the UDF is a
| black-box that may arbitrarily drop, rearrange, transform, or add new
| columns); if you want to apply a UDF such that column names are preserved,
| you should use the expression-level `map_elements` syntax instead.
|
| * If your function is expensive and you don't want it to be called more than
| once for a given input, consider applying an `@lru_cache` decorator to it.
| If your data is suitable you may achieve *significant* speedups.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [-1, 5, 8]})
|
| Return a DataFrame by mapping each row to a tuple:
|
| >>> df.map_rows(lambda t: (t[0] * 2, t[1] * 3))
| shape: (3, 2)
| ┌──────────┬──────────┐
| │ column_0 ┆ column_1 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════════╪══════════╡
| │ 2 ┆ -3 │
| │ 4 ┆ 15 │
| │ 6 ┆ 24 │
| └──────────┴──────────┘
|
| However, it is much better to implement this with a native expression:
|
| >>> df.select(
| ... pl.col("foo") * 2,
| ... pl.col("bar") * 3,
| ... ) # doctest: +IGNORE_RESULT
|
| Return a DataFrame with a single column by mapping each row to a scalar:
|
| >>> df.map_rows(lambda t: t[0] * 2 + t[1])
| shape: (3, 1)
| ┌─────┐
| │ map │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 9 │
| │ 14 │
| └─────┘
|
| In this case it is better to use the following native expression:
|
| >>> df.select(pl.col("foo") * 2 + pl.col("bar")) # doctest: +IGNORE_RESULT
|
| match_to_schema(self, schema: 'SchemaDict | Schema', *, missing_columns: "Literal['insert', 'raise'] | Mapping[str, Literal['insert', 'raise'] | Expr]" = 'raise', missing_struct_fields: "Literal['insert', 'raise'] | Mapping[str, Literal['insert', 'raise']]" = 'raise', extra_columns: "Literal['ignore', 'raise']" = 'raise', extra_struct_fields: "Literal['ignore', 'raise'] | Mapping[str, Literal['ignore', 'raise']]" = 'raise', integer_cast: "Literal['upcast', 'forbid'] | Mapping[str, Literal['upcast', 'forbid']]" = 'forbid', float_cast: "Literal['upcast', 'forbid'] | Mapping[str, Literal['upcast', 'forbid']]" = 'forbid') -> 'DataFrame'
| Match or evolve the schema of a LazyFrame into a specific schema.
|
| By default, match_to_schema returns an error if the input schema does not
| exactly match the target schema. It also allows columns to be freely reordered,
| with additional coercion rules available through optional parameters.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| schema
| Target schema to match or evolve to.
| missing_columns
| Raise of insert missing columns from the input with respect to the `schema`.
|
| This can also be an expression per column with what to insert if it is
| missing.
| missing_struct_fields
| Raise of insert missing struct fields from the input with respect to the
| `schema`.
| extra_columns
| Raise of ignore extra columns from the input with respect to the `schema`.
| extra_struct_fields
| Raise of ignore extra struct fields from the input with respect to the
| `schema`.
| integer_cast
| Forbid of upcast for integer columns from the input to the respective column
| in `schema`.
| float_cast
| Forbid of upcast for float columns from the input to the respective column
| in `schema`.
|
| Examples
| --------
| Ensuring the schema matches
|
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": ["A", "B", "C"]})
| >>> df.match_to_schema({"a": pl.Int64, "b": pl.String})
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ A │
| │ 2 ┆ B │
| │ 3 ┆ C │
| └─────┴─────┘
| >>> df.match_to_schema({"a": pl.Int64}) # doctest: +SKIP
| polars.exceptions.SchemaError: extra columns in `match_to_schema`: "b"
|
| Adding missing columns
|
| >>> (
| ... pl.DataFrame({"a": [1, 2, 3]}).match_to_schema(
| ... {"a": pl.Int64, "b": pl.String},
| ... missing_columns="insert",
| ... )
| ... )
| shape: (3, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪══════╡
| │ 1 ┆ null │
| │ 2 ┆ null │
| │ 3 ┆ null │
| └─────┴──────┘
| >>> (
| ... pl.DataFrame({"a": [1, 2, 3]}).match_to_schema(
| ... {"a": pl.Int64, "b": pl.String},
| ... missing_columns={"b": pl.col.a.cast(pl.String)},
| ... )
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| └─────┴─────┘
|
| Removing extra columns
|
| >>> (
| ... pl.DataFrame({"a": [1, 2, 3], "b": ["A", "B", "C"]}).match_to_schema(
| ... {"a": pl.Int64},
| ... extra_columns="ignore",
| ... )
| ... )
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Upcasting integers and floats
|
| >>> (
| ... pl.DataFrame(
| ... {"a": [1, 2, 3], "b": [1.0, 2.0, 3.0]},
| ... schema={"a": pl.Int32, "b": pl.Float32},
| ... ).match_to_schema(
| ... {"a": pl.Int64, "b": pl.Float64},
| ... integer_cast="upcast",
| ... float_cast="upcast",
| ... )
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 1.0 │
| │ 2 ┆ 2.0 │
| │ 3 ┆ 3.0 │
| └─────┴─────┘
|
| max(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their maximum value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.max()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| max_horizontal(self) -> 'Series'
| Get the maximum value horizontally across columns.
|
| Returns
| -------
| Series
| A Series named `"max"`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4.0, 5.0, 6.0],
| ... }
| ... )
| >>> df.max_horizontal()
| shape: (3,)
| Series: 'max' [f64]
| [
| 4.0
| 5.0
| 6.0
| ]
|
| mean(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their mean value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... "spam": [True, False, None],
| ... }
| ... )
| >>> df.mean()
| shape: (1, 4)
| ┌─────┬─────┬──────┬──────┐
| │ foo ┆ bar ┆ ham ┆ spam │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str ┆ f64 │
| ╞═════╪═════╪══════╪══════╡
| │ 2.0 ┆ 7.0 ┆ null ┆ 0.5 │
| └─────┴─────┴──────┴──────┘
|
| mean_horizontal(self, *, ignore_nulls: 'bool' = True) -> 'Series'
| Take the mean of all values horizontally across columns.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default).
| If set to `False`, any null value in the input will lead to a null output.
|
| Returns
| -------
| Series
| A Series named `"mean"`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4.0, 5.0, 6.0],
| ... }
| ... )
| >>> df.mean_horizontal()
| shape: (3,)
| Series: 'mean' [f64]
| [
| 2.5
| 3.5
| 4.5
| ]
|
| median(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their median value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.median()
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 2.0 ┆ 7.0 ┆ null │
| └─────┴─────┴──────┘
|
| melt(self, id_vars: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, value_vars: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, variable_name: 'str | None' = None, value_name: 'str | None' = None) -> 'DataFrame'
| Unpivot a DataFrame from wide to long format.
|
| Optionally leaves identifiers set.
|
| This function is useful to massage a DataFrame into a format where one or more
| columns are identifier variables (id_vars) while all other columns, considered
| measured variables (value_vars), are "unpivoted" to the row axis leaving just
| two non-identifier columns, 'variable' and 'value'.
|
| .. deprecated:: 1.0.0
| Use the :meth:`.unpivot` method instead.
|
| Parameters
| ----------
| id_vars
| Column(s) or selector(s) to use as identifier variables.
| value_vars
| Column(s) or selector(s) to use as values variables; if `value_vars`
| is empty all columns that are not in `id_vars` will be used.
| variable_name
| Name to give to the `variable` column. Defaults to "variable"
| value_name
| Name to give to the `value` column. Defaults to "value"
|
| merge_sorted(self, other: 'DataFrame', key: 'str | Sequence[str]', *, maintain_order: 'bool' = False) -> 'DataFrame'
| Take two sorted DataFrames and merge them by the sorted key.
|
| The output of this operation will also be sorted.
| It is the callers responsibility that the frames
| are sorted in ascending order by the key(s), with null
| keys at the start, otherwise the order of the output
| will not make sense.
|
| The schemas of both DataFrames must be equal.
|
| Parameters
| ----------
| other
| Other DataFrame that must be merged
| key
| Key column(s) that the frames are sorted by. A single column name or a
| sequence of column names can be passed. When multiple keys are given the
| frames are merged as if sorted by those keys in order.
| maintain_order
| If ``True``, the output is guaranteed to have left-biased ordering
| for equal keys: rows from the left frame appear before rows from
| the right frame when their keys are equal.
|
| Examples
| --------
| >>> df0 = pl.DataFrame(
| ... {"name": ["steve", "elise", "bob"], "age": [42, 44, 18]}
| ... ).sort("age")
| >>> df0
| shape: (3, 2)
| ┌───────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═════╡
| │ bob ┆ 18 │
| │ steve ┆ 42 │
| │ elise ┆ 44 │
| └───────┴─────┘
| >>> df1 = pl.DataFrame(
| ... {"name": ["anna", "megan", "steve", "thomas"], "age": [21, 33, 42, 20]}
| ... ).sort("age")
| >>> df1
| shape: (4, 2)
| ┌────────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞════════╪═════╡
| │ thomas ┆ 20 │
| │ anna ┆ 21 │
| │ megan ┆ 33 │
| │ steve ┆ 42 │
| └────────┴─────┘
| >>> df0.merge_sorted(df1, key="age")
| shape: (7, 2)
| ┌────────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞════════╪═════╡
| │ bob ┆ 18 │
| │ thomas ┆ 20 │
| │ anna ┆ 21 │
| │ megan ┆ 33 │
| │ steve ┆ 42 │
| │ steve ┆ 42 │
| │ elise ┆ 44 │
| └────────┴─────┘
|
| Multiple keys can be passed to merge frames sorted by a composite key. The
| frames are merged as if sorted by ``key_1``, then ``key_2``.
|
| >>> df0 = pl.DataFrame({"key_1": [1, 1, 3], "key_2": [1, 4, 2]})
| >>> df1 = pl.DataFrame({"key_1": [1, 2, 3], "key_2": [2, 1, 1]})
| >>> df0.merge_sorted(df1, key=["key_1", "key_2"])
| shape: (6, 2)
| ┌───────┬───────┐
| │ key_1 ┆ key_2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪═══════╡
| │ 1 ┆ 1 │
| │ 1 ┆ 2 │
| │ 1 ┆ 4 │
| │ 2 ┆ 1 │
| │ 3 ┆ 1 │
| │ 3 ┆ 2 │
| └───────┴───────┘
|
| Notes
| -----
| Unless ``maintain_order=True``, no guarantee is given over the output
| row order when the key is equal between the both dataframes.
|
| The key(s) must be sorted in ascending order.
|
| min(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their minimum value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.min()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| min_horizontal(self) -> 'Series'
| Get the minimum value horizontally across columns.
|
| Returns
| -------
| Series
| A Series named `"min"`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4.0, 5.0, 6.0],
| ... }
| ... )
| >>> df.min_horizontal()
| shape: (3,)
| Series: 'min' [f64]
| [
| 1.0
| 2.0
| 3.0
| ]
|
| n_chunks(self, strategy: "Literal['first', 'all']" = 'first') -> 'int | list[int]'
| Get number of chunks used by the ChunkedArrays of this DataFrame.
|
| Parameters
| ----------
| strategy : {'first', 'all'}
| Return the number of chunks of the 'first' column,
| or 'all' columns in this DataFrame.
|
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> df.n_chunks()
| 1
| >>> df.n_chunks(strategy="all")
| [1, 1, 1]
|
| n_unique(self, subset: 'str | Expr | Sequence[str | Expr] | None' = None) -> 'int'
| Return the number of unique rows, or the number of unique row-subsets.
|
| Parameters
| ----------
| subset
| One or more columns/expressions that define what to count;
| omit to return the count of unique rows.
|
| Notes
| -----
| This method operates at the `DataFrame` level; to operate on subsets at the
| expression level you can make use of struct-packing instead, for example:
|
| >>> expr_unique_subset = pl.struct("a", "b").n_unique()
|
| If instead you want to count the number of unique values per-column, you can
| also use expression-level syntax to return a new frame containing that result:
|
| >>> df = pl.DataFrame(
| ... [[1, 2, 3], [1, 2, 4]], schema=["a", "b", "c"], orient="row"
| ... )
| >>> df_nunique = df.select(pl.all().n_unique())
|
| In aggregate context there is also an equivalent method for returning the
| unique values per-group:
|
| >>> df_agg_nunique = df.group_by("a").n_unique()
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 1, 2, 3, 4, 5],
| ... "b": [0.5, 0.5, 1.0, 2.0, 3.0, 3.0],
| ... "c": [True, True, True, False, True, True],
| ... }
| ... )
| >>> df.n_unique()
| 5
|
| Simple columns subset.
|
| >>> df.n_unique(subset=["b", "c"])
| 4
|
| Expression subset.
|
| >>> df.n_unique(
| ... subset=[
| ... (pl.col("a") // 2),
| ... (pl.col("c") | (pl.col("b") >= 2)),
| ... ],
| ... )
| 3
|
| null_count(self) -> 'DataFrame'
| Create a new DataFrame that shows the null counts per column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, None, 3],
| ... "bar": [6, 7, None],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.null_count()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 1 ┆ 0 │
| └─────┴─────┴─────┘
|
| partition_by(self, by: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector]', *more_by: 'ColumnNameOrSelector', maintain_order: 'bool' = True, include_key: 'bool' = True, as_dict: 'bool' = False) -> 'list[DataFrame] | dict[tuple[Any, ...], DataFrame]'
| Group by the given columns and return the groups as separate dataframes.
|
| Parameters
| ----------
| by
| Column name(s) or selector(s) to group by.
| *more_by
| Additional names of columns to group by, specified as positional arguments.
| maintain_order
| Ensure that the order of the groups is consistent with the input data.
| This is slower than a default partition by operation.
| include_key
| Include the columns used to partition the DataFrame in the output.
| as_dict
| Return a dictionary instead of a list. The dictionary keys are tuples of
| the distinct group values that identify each group.
|
| Examples
| --------
| Pass a single column name to partition by that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "c"],
| ... "b": [1, 2, 1, 3, 3],
| ... "c": [5, 4, 3, 2, 1],
| ... }
| ... )
| >>> df.partition_by("a") # doctest: +IGNORE_RESULT
| [shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ a ┆ 1 ┆ 3 │
| └─────┴─────┴─────┘,
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| └─────┴─────┴─────┘,
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘]
|
| Partition by multiple columns by either passing a list of column names, or by
| specifying each column name as a positional argument.
|
| >>> df.partition_by("a", "b") # doctest: +IGNORE_RESULT
| [shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ a ┆ 1 ┆ 3 │
| └─────┴─────┴─────┘,
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 2 ┆ 4 │
| └─────┴─────┴─────┘,
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 3 ┆ 2 │
| └─────┴─────┴─────┘,
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘]
|
| Return the partitions as a dictionary by specifying `as_dict=True`.
|
| >>> import polars.selectors as cs
| >>> df.partition_by(cs.string(), as_dict=True) # doctest: +IGNORE_RESULT
| {('a',): shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ a ┆ 1 ┆ 3 │
| └─────┴─────┴─────┘,
| ('b',): shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| └─────┴─────┴─────┘,
| ('c',): shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘}
|
| pipe(self, function: 'Callable[Concatenate[DataFrame, P], T]', *args: 'P.args', **kwargs: 'P.kwargs') -> 'T'
| Offers a structured way to apply a sequence of user-defined functions (UDFs).
|
| Parameters
| ----------
| function
| Callable; will receive the frame as the first parameter,
| followed by any given args/kwargs.
| *args
| Arguments to pass to the UDF.
| **kwargs
| Keyword arguments to pass to the UDF.
|
| Notes
| -----
| It is recommended to use LazyFrame when piping operations, in order
| to fully take advantage of query optimization and parallelization.
| See :meth:`df.lazy() <polars.DataFrame.lazy>`.
|
| Examples
| --------
| >>> def cast_str_to_int(data, col_name):
| ... return data.with_columns(pl.col(col_name).cast(pl.Int64))
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4], "b": ["10", "20", "30", "40"]})
| >>> df.pipe(cast_str_to_int, col_name="b")
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 10 │
| │ 2 ┆ 20 │
| │ 3 ┆ 30 │
| │ 4 ┆ 40 │
| └─────┴─────┘
|
| >>> df = pl.DataFrame({"b": [1, 2], "a": [3, 4]})
| >>> df
| shape: (2, 2)
| ┌─────┬─────┐
| │ b ┆ a │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
| >>> df.pipe(lambda tdf: tdf.select(sorted(tdf.columns)))
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 1 │
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| pivot(self, on: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector]', on_columns: 'Sequence[Any] | pl.Series | pl.DataFrame | None' = None, *, index: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, values: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, aggregate_function: 'PivotAgg | Expr | None' = None, maintain_order: 'bool' = True, sort_columns: 'bool' = False, separator: 'str' = '_', column_naming: "Literal['auto', 'combine']" = 'auto') -> 'DataFrame'
| Create a spreadsheet-style pivot table as a DataFrame.
|
| Only available in eager mode. See "Examples" section below for how to do a
| "lazy pivot" if you know the unique column values in advance.
|
| .. versionchanged:: 1.0.0
| The `columns` parameter was renamed `on`.
|
| Parameters
| ----------
| on
| The column(s) whose values will be used as the new columns of the output
| DataFrame.
| on_columns
| What value combinations will be considered for the output table.
| index
| The column(s) that remain from the input to the output. The output DataFrame will have one row
| for each unique combination of the `index`'s values.
| If None, all remaining columns not specified on `on` and `values` will be used. At least one
| of `index` and `values` must be specified.
| values
| The existing column(s) of values which will be moved under the new columns from index. If an
| aggregation is specified, these are the values on which the aggregation will be computed.
| If None, all remaining columns not specified on `on` and `index` will be used.
| At least one of `index` and `values` must be specified.
| aggregate_function
| Choose from:
|
| - None: no aggregation takes place, will raise error if multiple values are in group.
| - A predefined aggregate function string, one of
| {'min', 'max', 'first', 'last', 'sum', 'mean', 'median', 'len'}
| - An expression to do the aggregation. The expression can only access data from the respective
| 'values' columns as generated by pivot, through `pl.element()`.
| maintain_order
| Ensure the values of `index` are sorted by discovery order.
| sort_columns
| Sort the transposed columns by name. Default is by order of discovery.
| separator
| Used as separator/delimiter in generated column names in case of multiple
| `values` columns.
| column_naming : {'auto', 'combine'}
| How resulting column names will be constructed.
|
| * 'auto': The default; combine with separator if there are multiple
| `values` columns, otherwise just use the `on_columns` names.
| * 'combine': Always combine the `values` columns' names with
| the `on_columns` names.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Notes
| -----
| In some other frameworks, you might know this operation as `pivot_wider`.
|
| Examples
| --------
| You can use `pivot` to reshape a dataframe from "long" to "wide" format.
|
| For example, suppose we have a dataframe of test scores achieved by some
| students, where each row represents a distinct test.
|
| >>> df = pl.DataFrame(
| ... {
| ... "name": ["Cady", "Cady", "Karen", "Karen"],
| ... "subject": ["maths", "physics", "maths", "physics"],
| ... "test_1": [98, 99, 61, 58],
| ... "test_2": [100, 100, 60, 60],
| ... }
| ... )
| >>> df
| shape: (4, 4)
| ┌───────┬─────────┬────────┬────────┐
| │ name ┆ subject ┆ test_1 ┆ test_2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 ┆ i64 │
| ╞═══════╪═════════╪════════╪════════╡
| │ Cady ┆ maths ┆ 98 ┆ 100 │
| │ Cady ┆ physics ┆ 99 ┆ 100 │
| │ Karen ┆ maths ┆ 61 ┆ 60 │
| │ Karen ┆ physics ┆ 58 ┆ 60 │
| └───────┴─────────┴────────┴────────┘
|
| Using `pivot`, we can reshape so we have one row per student, with different
| subjects as columns, and their `test_1` scores as values:
|
| >>> df.pivot("subject", index="name", values="test_1")
| shape: (2, 3)
| ┌───────┬───────┬─────────┐
| │ name ┆ maths ┆ physics │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════╪═══════╪═════════╡
| │ Cady ┆ 98 ┆ 99 │
| │ Karen ┆ 61 ┆ 58 │
| └───────┴───────┴─────────┘
|
| If you want to only pivot over a limited set of `subject` values or already
| know the `subject` values ahead of time, you can provide these using the
| `on_columns` argument.
|
| >>> df.pivot(
| ... "subject",
| ... on_columns=["maths", "physics"],
| ... index="name",
| ... values="test_1",
| ... )
| shape: (2, 3)
| ┌───────┬───────┬─────────┐
| │ name ┆ maths ┆ physics │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════╪═══════╪═════════╡
| │ Cady ┆ 98 ┆ 99 │
| │ Karen ┆ 61 ┆ 58 │
| └───────┴───────┴─────────┘
|
| You can use selectors too - here we include all test scores in the pivoted table:
|
| >>> import polars.selectors as cs
| >>> df.pivot("subject", values=cs.starts_with("test"))
| shape: (2, 5)
| ┌───────┬──────────────┬────────────────┬──────────────┬────────────────┐
| │ name ┆ test_1_maths ┆ test_1_physics ┆ test_2_maths ┆ test_2_physics │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═══════╪══════════════╪════════════════╪══════════════╪════════════════╡
| │ Cady ┆ 98 ┆ 99 ┆ 100 ┆ 100 │
| │ Karen ┆ 61 ┆ 58 ┆ 60 ┆ 60 │
| └───────┴──────────────┴────────────────┴──────────────┴────────────────┘
|
| If you end up with multiple values per cell, you can specify how to aggregate
| them with `aggregate_function`:
|
| >>> df = pl.DataFrame(
| ... {
| ... "ix": [1, 1, 2, 2, 1, 2],
| ... "col": ["a", "a", "a", "a", "b", "b"],
| ... "foo": [0, 1, 2, 2, 7, 1],
| ... "bar": [0, 2, 0, 0, 9, 4],
| ... }
| ... )
| >>> df.pivot("col", index="ix", aggregate_function="sum")
| shape: (2, 5)
| ┌─────┬───────┬───────┬───────┬───────┐
| │ ix ┆ foo_a ┆ foo_b ┆ bar_a ┆ bar_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═══════╪═══════╪═══════╪═══════╡
| │ 1 ┆ 1 ┆ 7 ┆ 2 ┆ 9 │
| │ 2 ┆ 4 ┆ 1 ┆ 0 ┆ 4 │
| └─────┴───────┴───────┴───────┴───────┘
|
| You can also pass a custom aggregation function using
| :meth:`polars.element`:
|
| >>> df = pl.DataFrame(
| ... {
| ... "col1": ["a", "a", "a", "b", "b", "b"],
| ... "col2": ["x", "x", "x", "x", "y", "y"],
| ... "col3": [6, 7, 3, 2, 5, 7],
| ... }
| ... )
| >>> df.pivot(
| ... "col2",
| ... index="col1",
| ... values="col3",
| ... aggregate_function=pl.element().tanh().mean(),
| ... )
| shape: (2, 3)
| ┌──────┬──────────┬──────────┐
| │ col1 ┆ x ┆ y │
| │ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 │
| ╞══════╪══════════╪══════════╡
| │ a ┆ 0.998347 ┆ null │
| │ b ┆ 0.964028 ┆ 0.999954 │
| └──────┴──────────┴──────────┘
|
| See Also
| --------
| LazyFrame.pivot
|
| product(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their product values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": [0.5, 4, 10],
| ... "c": [True, True, False],
| ... }
| ... )
|
| >>> df.product()
| shape: (1, 3)
| ┌─────┬──────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ i64 │
| ╞═════╪══════╪═════╡
| │ 6 ┆ 20.0 ┆ 0 │
| └─────┴──────┴─────┘
|
| quantile(self, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest') -> 'DataFrame'
| Aggregate the columns of this DataFrame to their quantile value.
|
| Parameters
| ----------
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.quantile(0.5, "nearest")
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 2.0 ┆ 7.0 ┆ null │
| └─────┴─────┴──────┘
|
| rechunk(self) -> 'DataFrame'
| Rechunk the data in this DataFrame to a contiguous allocation.
|
| This will make sure all subsequent operations have optimal and predictable
| performance.
|
| remove(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]', **constraints: 'Any') -> 'DataFrame'
| Remove rows, dropping those that match the given predicate expression(s).
|
| The original order of the remaining rows is preserved.
|
| Rows where the filter predicate does not evaluate to True are retained
| (this includes rows where the predicate evaluates as `null`).
|
| Parameters
| ----------
| predicates
| Expression(s) that evaluate to a boolean Series. When multiple predicates
| are provided they are combined using `&` (logical AND), so a row is only
| removed when *every* predicate evaluates to True for that row.
| constraints
| Column filters; use `name = value` to filter columns using the supplied
| value. Each constraint behaves the same as `pl.col(name).eq(value)`,
| and is implicitly joined with the other filter conditions using `&`.
|
| Notes
| -----
| If you are transitioning from Pandas, and performing filter operations based on
| the comparison of two or more columns, please note that in Polars any comparison
| involving `null` values will result in a `null` result, *not* boolean True or
| False. As a result, these rows will not be removed. Ensure that null values
| are handled appropriately to avoid unexpected behaviour (see examples below).
|
| See Also
| --------
| filter
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [2, 3, None, 4, 0],
| ... "bar": [5, 6, None, None, 0],
| ... "ham": ["a", "b", None, "c", "d"],
| ... }
| ... )
|
| Remove rows matching a condition:
|
| >>> df.remove(pl.col("bar") >= 5)
| shape: (3, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| │ 0 ┆ 0 ┆ d │
| └──────┴──────┴──────┘
|
| Discard rows based on multiple conditions, combined with and/or operators:
|
| >>> df.remove(
| ... (pl.col("foo") >= 0) & (pl.col("bar") >= 0),
| ... )
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| >>> df.remove(
| ... (pl.col("foo") >= 0) | (pl.col("bar") >= 0),
| ... )
| shape: (1, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| └──────┴──────┴──────┘
|
| Provide multiple constraints using `*args` syntax:
|
| >>> df.remove(
| ... pl.col("ham").is_not_null(),
| ... pl.col("bar") >= 0,
| ... )
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| Provide constraints(s) using `**kwargs` syntax:
|
| >>> df.remove(foo=0, bar=0)
| shape: (4, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ 2 ┆ 5 ┆ a │
| │ 3 ┆ 6 ┆ b │
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| Remove rows by comparing two columns against each other:
|
| >>> df.remove(
| ... pl.col("foo").ne_missing(pl.col("bar")),
| ... )
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 0 ┆ 0 ┆ d │
| └──────┴──────┴──────┘
|
| rename(self, mapping: 'Mapping[str, str] | Callable[[str], str]', *, strict: 'bool' = True) -> 'DataFrame'
| Rename column names.
|
| Parameters
| ----------
| mapping
| Key value pairs that map from old name to new name, or a function
| that takes the old name as input and returns the new name.
| strict
| Validate that all column names exist in the current schema,
| and throw an exception if any do not. (Note that this parameter
| is a no-op when passing a function to `mapping`).
|
| See Also
| --------
| Expr.name.replace
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"foo": [1, 2, 3], "bar": [6, 7, 8], "ham": ["a", "b", "c"]}
| ... )
| >>> df.rename({"foo": "apple"})
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ apple ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═══════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └───────┴─────┴─────┘
| >>> df.rename(lambda column_name: "c" + column_name[1:])
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ coo ┆ car ┆ cam │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| replace_column(self, index: 'int', column: 'Series') -> 'DataFrame'
| Replace a column at an index location.
|
| This operation is in place.
|
| Parameters
| ----------
| index
| Column index.
| column
| Series that will replace the column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> s = pl.Series("apple", [10, 20, 30])
| >>> df.replace_column(0, s)
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ apple ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═══════╪═════╪═════╡
| │ 10 ┆ 6 ┆ a │
| │ 20 ┆ 7 ┆ b │
| │ 30 ┆ 8 ┆ c │
| └───────┴─────┴─────┘
|
| reverse(self) -> 'DataFrame'
| Reverse the DataFrame.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "key": ["a", "b", "c"],
| ... "val": [1, 2, 3],
| ... }
| ... )
| >>> df.reverse()
| shape: (3, 2)
| ┌─────┬─────┐
| │ key ┆ val │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ c ┆ 3 │
| │ b ┆ 2 │
| │ a ┆ 1 │
| └─────┴─────┘
|
| rolling(self, index_column: 'IntoExpr', *, period: 'str | timedelta', offset: 'str | timedelta | None' = None, closed: 'ClosedInterval' = 'right', group_by: 'IntoExpr | Iterable[IntoExpr] | None' = None) -> 'RollingGroupBy'
| Create rolling groups based on a temporal or integer column.
|
| Different from a `group_by_dynamic` the windows are now determined by the
| individual values and are not of constant intervals. For constant intervals use
| :func:`DataFrame.group_by_dynamic`.
|
| If you have a time series `<t_0, t_1, ..., t_n>`, then by default the
| windows created will be
|
| * (t_0 - period, t_0]
| * (t_1 - period, t_1]
| * ...
| * (t_n - period, t_n]
|
| whereas if you pass a non-default `offset`, then the windows will be
|
| * (t_0 + offset, t_0 + offset + period]
| * (t_1 + offset, t_1 + offset + period]
| * ...
| * (t_n + offset, t_n + offset + period]
|
| The `period` and `offset` arguments are created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order (or, if `group_by` is
| specified, then it must be sorted in ascending order within each group).
|
| In case of a rolling operation on indices, dtype needs to be one of
| {UInt32, UInt64, Int32, Int64}. Note that the first three get temporarily
| cast to Int64, so if performance matters use an Int64 column.
| period
| Length of the window - must be non-negative.
| offset
| Offset of the window. Default is `-period`.
| closed : {'right', 'left', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
| group_by
| Also group by this column/these columns
|
| Returns
| -------
| RollingGroupBy
| Object you can call `.agg` on to aggregate by groups, the result
| of which will be sorted by `index_column` (but note that if `group_by`
| columns are passed, it will only be sorted within each group).
|
| See Also
| --------
| group_by_dynamic
|
| Examples
| --------
| >>> dates = [
| ... "2020-01-01 13:45:48",
| ... "2020-01-01 16:42:13",
| ... "2020-01-01 16:45:09",
| ... "2020-01-02 18:12:48",
| ... "2020-01-03 19:45:32",
| ... "2020-01-08 23:16:43",
| ... ]
| >>> df = pl.DataFrame({"dt": dates, "a": [3, 7, 5, 9, 2, 1]}).with_columns(
| ... pl.col("dt").str.strptime(pl.Datetime).set_sorted()
| ... )
| >>> out = df.rolling(index_column="dt", period="2d").agg(
| ... [
| ... pl.sum("a").alias("sum_a"),
| ... pl.min("a").alias("min_a"),
| ... pl.max("a").alias("max_a"),
| ... ]
| ... )
| >>> assert out["sum_a"].to_list() == [3, 10, 15, 24, 11, 1]
| >>> assert out["max_a"].to_list() == [3, 7, 7, 9, 9, 1]
| >>> assert out["min_a"].to_list() == [3, 3, 3, 3, 2, 1]
| >>> out
| shape: (6, 4)
| ┌─────────────────────┬───────┬───────┬───────┐
| │ dt ┆ sum_a ┆ min_a ┆ max_a │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ i64 ┆ i64 │
| ╞═════════════════════╪═══════╪═══════╪═══════╡
| │ 2020-01-01 13:45:48 ┆ 3 ┆ 3 ┆ 3 │
| │ 2020-01-01 16:42:13 ┆ 10 ┆ 3 ┆ 7 │
| │ 2020-01-01 16:45:09 ┆ 15 ┆ 3 ┆ 7 │
| │ 2020-01-02 18:12:48 ┆ 24 ┆ 3 ┆ 9 │
| │ 2020-01-03 19:45:32 ┆ 11 ┆ 2 ┆ 9 │
| │ 2020-01-08 23:16:43 ┆ 1 ┆ 1 ┆ 1 │
| └─────────────────────┴───────┴───────┴───────┘
|
| If you use an index count in `period` or `offset`, then it's based on the
| values in `index_column`:
|
| >>> df = pl.DataFrame({"int": [0, 4, 5, 6, 8], "value": [1, 4, 2, 4, 1]})
| >>> df.rolling("int", period="3i").agg(pl.col("int").alias("aggregated"))
| shape: (5, 2)
| ┌─────┬────────────┐
| │ int ┆ aggregated │
| │ --- ┆ --- │
| │ i64 ┆ list[i64] │
| ╞═════╪════════════╡
| │ 0 ┆ [0] │
| │ 4 ┆ [4] │
| │ 5 ┆ [4, 5] │
| │ 6 ┆ [4, 5, 6] │
| │ 8 ┆ [6, 8] │
| └─────┴────────────┘
|
| If you want the index count to be based on row number, then you may want to
| combine `rolling` with :meth:`.with_row_index`.
|
| row(self, index: 'int | None' = None, *, by_predicate: 'Expr | None' = None, named: 'bool' = False) -> 'tuple[Any, ...] | dict[str, Any]'
| Get the values of a single row, either by index or by predicate.
|
| Parameters
| ----------
| index
| Row index.
| by_predicate
| Select the row according to a given expression/predicate.
| named
| Return a dictionary instead of a tuple. The dictionary is a mapping of
| column name to row value. This is more expensive than returning a regular
| tuple, but allows for accessing values by column name.
|
| Returns
| -------
| tuple (default) or dictionary of row values
|
| Notes
| -----
| The `index` and `by_predicate` params are mutually exclusive. Additionally,
| to ensure clarity, the `by_predicate` parameter must be supplied by keyword.
|
| When using `by_predicate` it is an error condition if anything other than
| one row is returned; more than one row raises `TooManyRowsReturnedError`, and
| zero rows will raise `NoRowsReturnedError` (both inherit from `RowsError`).
|
| Warnings
| --------
| You should NEVER use this method to iterate over a DataFrame; if you require
| row-iteration you should strongly prefer use of `iter_rows()` instead.
|
| See Also
| --------
| iter_rows : Row iterator over frame data (does not materialise all rows).
| rows : Materialise all frame data as a list of rows (potentially expensive).
| item: Return dataframe element as a scalar.
|
| Examples
| --------
| Specify an index to return the row at the given index as a tuple.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.row(2)
| (3, 8, 'c')
|
| Specify `named=True` to get a dictionary instead with a mapping of column
| names to row values.
|
| >>> df.row(2, named=True)
| {'foo': 3, 'bar': 8, 'ham': 'c'}
|
| Use `by_predicate` to return the row that matches the given predicate.
|
| >>> df.row(by_predicate=(pl.col("ham") == "b"))
| (2, 7, 'b')
|
| rows(self, *, named: 'bool' = False) -> 'list[tuple[Any, ...]] | list[dict[str, Any]]'
| Returns all data in the DataFrame as a list of rows of python-native values.
|
| By default, each row is returned as a tuple of values given in the same order
| as the frame columns. Setting `named=True` will return rows of dictionaries
| instead.
|
| Parameters
| ----------
| named
| Return dictionaries instead of tuples. The dictionaries are a mapping of
| column name to row value. This is more expensive than returning a regular
| tuple, but allows for accessing values by column name.
|
| Notes
| -----
| If you have `ns`-precision temporal values you should be aware that Python
| natively only supports up to `μs`-precision; `ns`-precision values will be
| truncated to microseconds on conversion to Python. If this matters to your
| use-case you should export to a different format (such as Arrow or NumPy).
|
| Warnings
| --------
| Row-iteration is not optimal as the underlying data is stored in columnar form;
| where possible, prefer export via one of the dedicated export/output methods.
| You should also consider using `iter_rows` instead, to avoid materialising all
| the data at once; there is little performance difference between the two, but
| peak memory can be reduced if processing rows in batches.
|
| Returns
| -------
| list of row value tuples (default), or list of dictionaries (if `named=True`).
|
| See Also
| --------
| iter_rows : Row iterator over frame data (does not materialise all rows).
| rows_by_key : Materialises frame data as a key-indexed dictionary.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "x": ["a", "b", "b", "a"],
| ... "y": [1, 2, 3, 4],
| ... "z": [0, 3, 6, 9],
| ... }
| ... )
| >>> df.rows()
| [('a', 1, 0), ('b', 2, 3), ('b', 3, 6), ('a', 4, 9)]
| >>> df.rows(named=True)
| [{'x': 'a', 'y': 1, 'z': 0},
| {'x': 'b', 'y': 2, 'z': 3},
| {'x': 'b', 'y': 3, 'z': 6},
| {'x': 'a', 'y': 4, 'z': 9}]
|
| rows_by_key(self, key: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector]', *, named: 'bool' = False, include_key: 'bool' = False, unique: 'bool' = False) -> 'dict[Any, Any]'
| Returns all data as a dictionary of python-native values keyed by some column.
|
| This method is like `rows`, but instead of returning rows in a flat list, rows
| are grouped by the values in the `key` column(s) and returned as a dictionary.
|
| Note that this method should not be used in place of native operations, due to
| the high cost of materializing all frame data out into a dictionary; it should
| be used only when you need to move the values out into a Python data structure
| or other object that cannot operate directly with Polars/Arrow.
|
| Parameters
| ----------
| key
| The column(s) to use as the key for the returned dictionary. If multiple
| columns are specified, the key will be a tuple of those values, otherwise
| it will be a string.
| named
| Return dictionary rows instead of tuples, mapping column name to row value.
| include_key
| Include key values inline with the associated data (by default the key
| values are omitted as a memory/performance optimisation, as they can be
| reoconstructed from the key).
| unique
| Indicate that the key is unique; this will result in a 1:1 mapping from
| key to a single associated row. Note that if the key is *not* actually
| unique the last row with the given key will be returned.
|
| Notes
| -----
| If you have `ns`-precision temporal values you should be aware that Python
| natively only supports up to `μs`-precision; `ns`-precision values will be
| truncated to microseconds on conversion to Python. If this matters to your
| use-case you should export to a different format (such as Arrow or NumPy).
|
| See Also
| --------
| rows : Materialize all frame data as a list of rows (potentially expensive).
| iter_rows : Row iterator over frame data (does not materialize all rows).
| to_dict : Convert DataFrame to a dictionary mapping column name to values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "w": ["a", "b", "b", "a"],
| ... "x": ["q", "q", "q", "k"],
| ... "y": [1.0, 2.5, 3.0, 4.5],
| ... "z": [9, 8, 7, 6],
| ... }
| ... )
|
| Group rows by the given key column(s):
|
| >>> df.rows_by_key(key=["w"])
| defaultdict(<class 'list'>,
| {'a': [('q', 1.0, 9), ('k', 4.5, 6)],
| 'b': [('q', 2.5, 8), ('q', 3.0, 7)]})
|
| Return the same row groupings as dictionaries:
|
| >>> df.rows_by_key(key=["w"], named=True)
| defaultdict(<class 'list'>,
| {'a': [{'x': 'q', 'y': 1.0, 'z': 9},
| {'x': 'k', 'y': 4.5, 'z': 6}],
| 'b': [{'x': 'q', 'y': 2.5, 'z': 8},
| {'x': 'q', 'y': 3.0, 'z': 7}]})
|
| Return row groupings, assuming keys are unique:
|
| >>> df.rows_by_key(key=["z"], unique=True)
| {9: ('a', 'q', 1.0),
| 8: ('b', 'q', 2.5),
| 7: ('b', 'q', 3.0),
| 6: ('a', 'k', 4.5)}
|
| Return row groupings as dictionaries, assuming keys are unique:
|
| >>> df.rows_by_key(key=["z"], named=True, unique=True)
| {9: {'w': 'a', 'x': 'q', 'y': 1.0},
| 8: {'w': 'b', 'x': 'q', 'y': 2.5},
| 7: {'w': 'b', 'x': 'q', 'y': 3.0},
| 6: {'w': 'a', 'x': 'k', 'y': 4.5}}
|
| Return dictionary rows grouped by a compound key, including key values:
|
| >>> df.rows_by_key(key=["w", "x"], named=True, include_key=True)
| defaultdict(<class 'list'>,
| {('a', 'q'): [{'w': 'a', 'x': 'q', 'y': 1.0, 'z': 9}],
| ('b', 'q'): [{'w': 'b', 'x': 'q', 'y': 2.5, 'z': 8},
| {'w': 'b', 'x': 'q', 'y': 3.0, 'z': 7}],
| ('a', 'k'): [{'w': 'a', 'x': 'k', 'y': 4.5, 'z': 6}]})
|
| sample(self, n: 'int | Series | None' = None, *, fraction: 'float | Series | None' = None, with_replacement: 'bool' = False, shuffle: 'bool | None' = None, seed: 'int | None' = None) -> 'DataFrame'
| Sample from this DataFrame.
|
| Parameters
| ----------
| n
| Number of items to return. Cannot be used with `fraction`. Defaults to 1 if
| `fraction` is None.
| fraction
| Fraction of items to return. Cannot be used with `n`.
| with_replacement
| Allow values to be sampled more than once.
| shuffle
| Determines the order of the sampled rows.
| If True, sampled rows are explicitly shuffled.
| If False, the relative order of the sampled rows is preserved.
| (i.e. they appear in the same order as the original DataFrame).
| If None (default), no ordering guarantee; uses the most performant
| algorithm.
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated for each time the sample is called.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.sample(n=2, shuffle=False, seed=0) # doctest: +IGNORE_RESULT
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8 ┆ c │
| │ 2 ┆ 7 ┆ b │
| └─────┴─────┴─────┘
|
| select(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'DataFrame'
| Select columns from this DataFrame.
|
| Parameters
| ----------
| *exprs
| Column(s) to select, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names,
| other non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to select, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Examples
| --------
| Pass the name of a column to select that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.select("foo")
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Multiple columns can be selected by passing a list of column names.
|
| >>> df.select(["foo", "bar"])
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 6 │
| │ 2 ┆ 7 │
| │ 3 ┆ 8 │
| └─────┴─────┘
|
| Multiple columns can also be selected using positional arguments instead of a
| list. Expressions are also accepted.
|
| >>> df.select(pl.col("foo"), pl.col("bar") + 1)
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| └─────┴─────┘
|
| Use keyword arguments to easily name your expression inputs.
|
| >>> df.select(threshold=pl.when(pl.col("foo") > 2).then(10).otherwise(0))
| shape: (3, 1)
| ┌───────────┐
| │ threshold │
| │ --- │
| │ i32 │
| ╞═══════════╡
| │ 0 │
| │ 0 │
| │ 10 │
| └───────────┘
|
| select_seq(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'DataFrame'
| Select columns from this DataFrame.
|
| This will run all expression sequentially instead of in parallel.
| Use this when the work per expression is cheap.
|
| Parameters
| ----------
| *exprs
| Column(s) to select, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names,
| other non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to select, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| See Also
| --------
| select
|
| serialize(self, file: 'IOBase | str | Path | None' = None, *, format: 'SerializationFormat' = 'binary') -> 'bytes | str | None'
| Serialize this DataFrame to a file or string in JSON format.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| If set to `None` (default), the output is returned as a string instead.
| format
| The format in which to serialize. Options:
|
| - `"binary"`: Serialize to binary format (bytes). This is the default.
| - `"json"`: Serialize to JSON format (string).
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| Serialize the DataFrame into a binary representation.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... }
| ... )
| >>> bytes = df.serialize()
| >>> type(bytes)
| <class 'bytes'>
|
| The bytes can later be deserialized back into a DataFrame.
|
| >>> import io
| >>> pl.DataFrame.deserialize(io.BytesIO(bytes))
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 6 │
| │ 2 ┆ 7 │
| │ 3 ┆ 8 │
| └─────┴─────┘
|
| set_sorted(self, column: 'str', *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'DataFrame'
| Flag a column as sorted.
|
| This can speed up future operations.
|
| Parameters
| ----------
| column
| Column that is sorted
| descending
| Whether the column is sorted in descending order.
| nulls_last
| Whether the nulls are at the end.
|
| Warnings
| --------
| This can lead to incorrect results if the data is NOT sorted!!
| Use with care!
|
| shift(self, n: 'int' = 1, *, fill_value: 'IntoExpr | None' = None) -> 'DataFrame'
| Shift values by the given number of indices.
|
| Parameters
| ----------
| n
| Number of indices to shift forward. If a negative value is passed, values
| are shifted in the opposite direction instead.
| fill_value
| Fill the resulting null values with this value. Accepts scalar expression
| input. Non-expression inputs are parsed as literals.
|
| Notes
| -----
| This method is similar to the `LAG` operation in SQL when the value for `n`
| is positive. With a negative value for `n`, it is similar to `LEAD`.
|
| Examples
| --------
| By default, values are shifted forward by one index.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [5, 6, 7, 8],
| ... }
| ... )
| >>> df.shift()
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ null ┆ null │
| │ 1 ┆ 5 │
| │ 2 ┆ 6 │
| │ 3 ┆ 7 │
| └──────┴──────┘
|
| Pass a negative value to shift in the opposite direction instead.
|
| >>> df.shift(-2)
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ 3 ┆ 7 │
| │ 4 ┆ 8 │
| │ null ┆ null │
| │ null ┆ null │
| └──────┴──────┘
|
| Specify `fill_value` to fill the resulting null values.
|
| >>> df.shift(-2, fill_value=100)
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 7 │
| │ 4 ┆ 8 │
| │ 100 ┆ 100 │
| │ 100 ┆ 100 │
| └─────┴─────┘
|
| show(self, limit: 'int | None' = 5, *, ascii_tables: 'bool | None' = None, decimal_separator: 'str | None' = None, thousands_separator: 'str | bool | None' = None, float_precision: 'int | None' = None, fmt_float: 'FloatFmt | None' = None, fmt_str_lengths: 'int | None' = None, fmt_table_cell_list_len: 'int | None' = None, tbl_cell_alignment: 'Alignment | None' = None, tbl_cell_numeric_alignment: 'Alignment | None' = None, tbl_cols: 'int | None' = None, tbl_column_data_type_inline: 'bool | None' = None, tbl_dataframe_shape_below: 'bool | None' = None, tbl_formatting: 'TableFormatNames | None' = None, tbl_hide_column_data_types: 'bool | None' = None, tbl_hide_column_names: 'bool | None' = None, tbl_hide_dtype_separator: 'bool | None' = None, tbl_hide_dataframe_shape: 'bool | None' = None, tbl_width_chars: 'int | None' = None, trim_decimal_zeros: 'bool | None' = True) -> 'None'
| Show the first `n` rows.
|
| Parameters
| ----------
| limit : int
| Numbers of rows to show. If a negative value is passed, return all rows
| except the last `abs(n)`. If None is passed, return all rows.
| ascii_tables : bool
| Use ASCII characters to display table outlines. Set False to revert to the
| default UTF8_FULL_CONDENSED formatting style. See
| :func:`Config.set_ascii_tables` for more information.
| decimal_separator : str
| Set the decimal separator character. See
| :func:`Config.set_decimal_separator` for more information.
| thousands_separator : str, bool
| Set the thousands grouping separator character. See
| :func:`Config.set_thousands_separator` for more information.
| float_precision : int
| Number of decimal places to display for floating point values. See
| :func:`Config.set_float_precision` for more information.
| fmt_float : {"mixed", "full"}
| Control how floating point values are displayed. See
| :func:`Config.set_fmt_float` for more information. Supported options are:
|
| * "mixed": Limit the number of decimal places and use scientific notation
| for large/small values.
| * "full": Print the full precision of the floating point number.
|
| fmt_str_lengths : int
| Number of characters to display for string values. See
| :func:`Config.set_fmt_str_lengths` for more information.
| fmt_table_cell_list_len : int
| Number of elements to display for List values. See
| :func:`Config.set_fmt_table_cell_list_len` for more information.
| tbl_cell_alignment : str
| Set table cell alignment. See :func:`Config.set_tbl_cell_alignment` for more
| information. Supported options are:
|
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| tbl_cell_numeric_alignment : str
| Set table cell alignment for numeric columns. See
| :func:`Config.set_tbl_cell_numeric_alignment` for more information.
| Supported options are:
|
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| tbl_cols : int
| Number of columns to display. See :func:`Config.set_tbl_cols` for more
| information.
| tbl_column_data_type_inline : bool
| Moves the data type inline with the column name (to the right, in
| parentheses). See :func:`Config.set_tbl_column_data_type_inline` for more
| information.
| tbl_dataframe_shape_below : bool
| Print the DataFrame shape information below the data when displaying tables.
| See :func:`Config.set_tbl_dataframe_shape_below` for more information.
| tbl_formatting : str
| Set table formatting style. See :func:`Config.set_tbl_formatting` for more
| information. Supported options are:
|
| * "ASCII_FULL": ASCII, with all borders and lines, including row dividers.
| * "ASCII_FULL_CONDENSED": Same as ASCII_FULL, but with dense row spacing.
| * "ASCII_NO_BORDERS": ASCII, no borders.
| * "ASCII_BORDERS_ONLY": ASCII, borders only.
| * "ASCII_BORDERS_ONLY_CONDENSED": ASCII, borders only, dense row spacing.
| * "ASCII_HORIZONTAL_ONLY": ASCII, horizontal lines only.
| * "ASCII_MARKDOWN": Markdown format (ascii ellipses for truncated values).
| * "MARKDOWN": Markdown format (utf8 ellipses for truncated values).
| * "UTF8_FULL": UTF8, with all borders and lines, including row dividers.
| * "UTF8_FULL_CONDENSED": Same as UTF8_FULL, but with dense row spacing.
| * "UTF8_NO_BORDERS": UTF8, no borders.
| * "UTF8_BORDERS_ONLY": UTF8, borders only.
| * "UTF8_HORIZONTAL_ONLY": UTF8, horizontal lines only.
| * "NOTHING": No borders or other lines.
|
| tbl_hide_column_data_types : bool
| Hide table column data types (i64, f64, str etc.). See
| :func:`Config.set_tbl_hide_column_data_types` for more information.
| tbl_hide_column_names : bool
| Hide table column names. See :func:`Config.set_tbl_hide_column_names` for
| more information.
| tbl_hide_dtype_separator : bool
| Hide the '---' separator between the column names and column types. See
| :func:`Config.set_tbl_hide_dtype_separator` for more information.
| tbl_hide_dataframe_shape : bool
| Hide the DataFrame shape information when displaying tables. See
| :func:`Config.set_tbl_hide_dataframe_shape` for more information.
| tbl_width_chars : int
| Set the maximum width of a table in characters. See
| :func:`Config.set_tbl_width_chars` for more information.
| trim_decimal_zeros : bool
| Strip trailing zeros from Decimal data type values. See
| :func:`Config.set_trim_decimal_zeros` for more information.
|
| See Also
| --------
| head
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> df.show(3)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Pass a negative value to get all rows `except` the last `abs(n)`.
|
| >>> df.show(-3)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| └─────┴─────┴─────┘
|
| shrink_to_fit(self, *, in_place: 'bool' = False) -> 'DataFrame'
| Shrink DataFrame memory usage.
|
| Shrinks to fit the exact capacity needed to hold the data.
|
| slice(self, offset: 'int', length: 'int | None' = None) -> 'DataFrame'
| Get a slice of this DataFrame.
|
| Parameters
| ----------
| offset
| Start index. Negative indexing is supported.
| length
| Length of the slice. If set to `None`, all rows starting at the offset
| will be selected.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.slice(1, 2)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ 7.0 ┆ b │
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| sort(self, by: 'IntoExpr | Iterable[IntoExpr]', *more_by: 'IntoExpr', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False, multithreaded: 'bool' = True, maintain_order: 'bool' = False) -> 'DataFrame'
| Sort the dataframe by the given columns.
|
| Parameters
| ----------
| by
| Column(s) to sort by. Accepts expression input, including selectors. Strings
| are parsed as column names.
| *more_by
| Additional columns to sort by, specified as positional arguments.
| descending
| Sort in descending order. When sorting by multiple columns, can be specified
| per column by passing a sequence of booleans.
| nulls_last
| Place null values last; can specify a single boolean applying to all columns
| or a sequence of booleans for per-column control.
| multithreaded
| Sort using multiple threads.
| maintain_order
| Whether the order should be maintained if elements are equal.
|
| Examples
| --------
| Pass a single column name to sort by that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": [6.0, 5.0, 4.0],
| ... "c": ["a", "c", "b"],
| ... }
| ... )
| >>> df.sort("a")
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ null ┆ 4.0 ┆ b │
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 5.0 ┆ c │
| └──────┴─────┴─────┘
|
| Sorting by expressions is also supported.
|
| >>> df.sort(pl.col("a") + pl.col("b") * 2, nulls_last=True)
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 2 ┆ 5.0 ┆ c │
| │ 1 ┆ 6.0 ┆ a │
| │ null ┆ 4.0 ┆ b │
| └──────┴─────┴─────┘
|
| Sort by multiple columns by passing a list of columns.
|
| >>> df.sort(["c", "a"], descending=True)
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 2 ┆ 5.0 ┆ c │
| │ null ┆ 4.0 ┆ b │
| │ 1 ┆ 6.0 ┆ a │
| └──────┴─────┴─────┘
|
| Or use positional arguments to sort by multiple columns in the same way.
|
| >>> df.sort("c", "a", descending=[False, True])
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ null ┆ 4.0 ┆ b │
| │ 2 ┆ 5.0 ┆ c │
| └──────┴─────┴─────┘
|
| sql(self, query: 'str', *, table_name: 'str' = 'self') -> 'DataFrame'
| Execute a SQL query against the DataFrame.
|
| .. versionadded:: 0.20.24
|
| .. warning::
| This functionality is considered **unstable**, although it is close to
| being considered stable. It may be changed at any point without it being
| considered a breaking change.
|
| Parameters
| ----------
| query
| SQL query to execute.
| table_name
| Optionally provide an explicit name for the table that represents the
| calling frame (defaults to "self").
|
| Notes
| -----
| * The calling DataFrame is automatically registered as a table in the SQLContext
| under the name "self". If you want access to the DataFrames and LazyFrames
| found in the current globals, use the top-level :meth:`pl.sql <polars.sql>`.
| * More control over registration and execution behaviour is available by
| using the :class:`SQLContext` object.
| * The SQL query executes in lazy mode before being collected and returned
| as a DataFrame.
|
| See Also
| --------
| SQLContext
|
| Examples
| --------
| >>> from datetime import date
| >>> df1 = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["zz", "yy", "xx"],
| ... "c": [date(1999, 12, 31), date(2010, 10, 10), date(2077, 8, 8)],
| ... }
| ... )
|
| Query the DataFrame using SQL:
|
| >>> df1.sql("SELECT c, b FROM self WHERE a > 1")
| shape: (2, 2)
| ┌────────────┬─────┐
| │ c ┆ b │
| │ --- ┆ --- │
| │ date ┆ str │
| ╞════════════╪═════╡
| │ 2010-10-10 ┆ yy │
| │ 2077-08-08 ┆ xx │
| └────────────┴─────┘
|
| Apply transformations to a DataFrame using SQL, aliasing "self" to "frame".
|
| >>> df1.sql(
| ... query='''
| ... SELECT
| ... a,
| ... (a % 2 == 0) AS a_is_even,
| ... CONCAT_WS(':', b, b) AS b_b,
| ... EXTRACT(year FROM c) AS year,
| ... 0::float4 AS "zero",
| ... FROM frame
| ... ''',
| ... table_name="frame",
| ... )
| shape: (3, 5)
| ┌─────┬───────────┬───────┬──────┬──────┐
| │ a ┆ a_is_even ┆ b_b ┆ year ┆ zero │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ bool ┆ str ┆ i32 ┆ f32 │
| ╞═════╪═══════════╪═══════╪══════╪══════╡
| │ 1 ┆ false ┆ zz:zz ┆ 1999 ┆ 0.0 │
| │ 2 ┆ true ┆ yy:yy ┆ 2010 ┆ 0.0 │
| │ 3 ┆ false ┆ xx:xx ┆ 2077 ┆ 0.0 │
| └─────┴───────────┴───────┴──────┴──────┘
|
| std(self, ddof: 'int' = 1) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their standard deviation value.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.std()
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 1.0 ┆ 1.0 ┆ null │
| └─────┴─────┴──────┘
| >>> df.std(ddof=0)
| shape: (1, 3)
| ┌──────────┬──────────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════════╪══════════╪══════╡
| │ 0.816497 ┆ 0.816497 ┆ null │
| └──────────┴──────────┴──────┘
|
| sum(self) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their sum value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.sum()
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 6 ┆ 21 ┆ null │
| └─────┴─────┴──────┘
|
| sum_horizontal(self, *, ignore_nulls: 'bool' = True) -> 'Series'
| Sum all values horizontally across columns.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default).
| If set to `False`, any null value in the input will lead to a null output.
|
| Returns
| -------
| Series
| A Series named `"sum"`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4.0, 5.0, 6.0],
| ... }
| ... )
| >>> df.sum_horizontal()
| shape: (3,)
| Series: 'sum' [f64]
| [
| 5.0
| 7.0
| 9.0
| ]
|
| tail(self, n: 'int' = 5) -> 'DataFrame'
| Get the last `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return. If a negative value is passed, return all rows
| except the first `abs(n)`.
|
| See Also
| --------
| head, slice
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> df.tail(3)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ 9 ┆ d │
| │ 5 ┆ 10 ┆ e │
| └─────┴─────┴─────┘
|
| Pass a negative value to get all rows `except` the first `abs(n)`.
|
| >>> df.tail(-3)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 4 ┆ 9 ┆ d │
| │ 5 ┆ 10 ┆ e │
| └─────┴─────┴─────┘
|
| to_arrow(self, *, compat_level: 'CompatLevel | None' = None) -> 'pa.Table'
| Collect the underlying arrow arrays in an Arrow Table.
|
| This operation is mostly zero copy.
|
| Data types that do copy:
| - CategoricalType
|
| .. versionchanged:: 1.1
| The `future` parameter was renamed `compat_level`.
|
| Parameters
| ----------
| compat_level
| Compatibility level to use when exporting Polars data structures.
| The default compatibility level is recommended for most users.
| Use ``pl.CompatLevel.oldest()`` for the most compatible level.
| ``pl.CompatLevel.newest()`` uses the highest supported compatibility
| level, but is considered unstable and may change without it being
| considered a breaking change.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"foo": [1, 2, 3, 4, 5, 6], "bar": ["a", "b", "c", "d", "e", "f"]}
| ... )
| >>> df.to_arrow()
| pyarrow.Table
| foo: int64
| bar: large_string
| ----
| foo: [[1,2,3,4,5,6]]
| bar: [["a","b","c","d","e","f"]]
|
| to_dict(self, *, as_series: 'bool' = True) -> 'dict[str, Series] | dict[str, list[Any]]'
| Convert DataFrame to a dictionary mapping column name to values.
|
| Parameters
| ----------
| as_series
| True -> Values are Series
| False -> Values are List[Any]
|
| See Also
| --------
| rows_by_key
| to_dicts
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1, 2, 3, 4, 5],
| ... "fruits": ["banana", "banana", "apple", "apple", "banana"],
| ... "B": [5, 4, 3, 2, 1],
| ... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"],
| ... "optional": [28, 300, None, 2, -30],
| ... }
| ... )
| >>> df
| shape: (5, 5)
| ┌─────┬────────┬─────┬────────┬──────────┐
| │ A ┆ fruits ┆ B ┆ cars ┆ optional │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ i64 ┆ str ┆ i64 │
| ╞═════╪════════╪═════╪════════╪══════════╡
| │ 1 ┆ banana ┆ 5 ┆ beetle ┆ 28 │
| │ 2 ┆ banana ┆ 4 ┆ audi ┆ 300 │
| │ 3 ┆ apple ┆ 3 ┆ beetle ┆ null │
| │ 4 ┆ apple ┆ 2 ┆ beetle ┆ 2 │
| │ 5 ┆ banana ┆ 1 ┆ beetle ┆ -30 │
| └─────┴────────┴─────┴────────┴──────────┘
| >>> df.to_dict(as_series=False)
| {'A': [1, 2, 3, 4, 5],
| 'fruits': ['banana', 'banana', 'apple', 'apple', 'banana'],
| 'B': [5, 4, 3, 2, 1],
| 'cars': ['beetle', 'audi', 'beetle', 'beetle', 'beetle'],
| 'optional': [28, 300, None, 2, -30]}
| >>> df.to_dict(as_series=True)
| {'A': shape: (5,)
| Series: 'A' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| ], 'fruits': shape: (5,)
| Series: 'fruits' [str]
| [
| "banana"
| "banana"
| "apple"
| "apple"
| "banana"
| ], 'B': shape: (5,)
| Series: 'B' [i64]
| [
| 5
| 4
| 3
| 2
| 1
| ], 'cars': shape: (5,)
| Series: 'cars' [str]
| [
| "beetle"
| "audi"
| "beetle"
| "beetle"
| "beetle"
| ], 'optional': shape: (5,)
| Series: 'optional' [i64]
| [
| 28
| 300
| null
| 2
| -30
| ]}
|
| to_dicts(self) -> 'list[dict[str, Any]]'
| Convert every row to a dictionary of Python-native values.
|
| Notes
| -----
| If you have `ns`-precision temporal values you should be aware that Python
| natively only supports up to `μs`-precision; `ns`-precision values will be
| truncated to microseconds on conversion to Python. If this matters to your
| use-case you should export to a different format (such as Arrow or NumPy).
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
| >>> df.to_dicts()
| [{'foo': 1, 'bar': 4}, {'foo': 2, 'bar': 5}, {'foo': 3, 'bar': 6}]
|
| to_dummies(self, columns: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, *, separator: 'str' = '_', drop_first: 'bool' = False, drop_nulls: 'bool' = False) -> 'DataFrame'
| Convert categorical variables into dummy/indicator variables.
|
| Parameters
| ----------
| columns
| Column name(s) or selector(s) that should be converted to dummy
| variables. If set to `None` (default), convert all columns.
| separator
| Separator/delimiter used when generating column names.
| drop_first
| Remove the first category from the variables being encoded.
| drop_nulls
| If there are `None` values in the series, a `null` column is not generated
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2],
| ... "bar": [3, 4],
| ... "ham": ["a", "b"],
| ... }
| ... )
| >>> df.to_dummies()
| shape: (2, 6)
| ┌───────┬───────┬───────┬───────┬───────┬───────┐
| │ foo_1 ┆ foo_2 ┆ bar_3 ┆ bar_4 ┆ ham_a ┆ ham_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ u8 ┆ u8 ┆ u8 ┆ u8 │
| ╞═══════╪═══════╪═══════╪═══════╪═══════╪═══════╡
| │ 1 ┆ 0 ┆ 1 ┆ 0 ┆ 1 ┆ 0 │
| │ 0 ┆ 1 ┆ 0 ┆ 1 ┆ 0 ┆ 1 │
| └───────┴───────┴───────┴───────┴───────┴───────┘
|
| >>> df.to_dummies(drop_first=True)
| shape: (2, 3)
| ┌───────┬───────┬───────┐
| │ foo_2 ┆ bar_4 ┆ ham_b │
| │ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ u8 │
| ╞═══════╪═══════╪═══════╡
| │ 0 ┆ 0 ┆ 0 │
| │ 1 ┆ 1 ┆ 1 │
| └───────┴───────┴───────┘
|
| >>> import polars.selectors as cs
| >>> df.to_dummies(cs.integer(), separator=":")
| shape: (2, 5)
| ┌───────┬───────┬───────┬───────┬─────┐
| │ foo:1 ┆ foo:2 ┆ bar:3 ┆ bar:4 ┆ ham │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ u8 ┆ u8 ┆ str │
| ╞═══════╪═══════╪═══════╪═══════╪═════╡
| │ 1 ┆ 0 ┆ 1 ┆ 0 ┆ a │
| │ 0 ┆ 1 ┆ 0 ┆ 1 ┆ b │
| └───────┴───────┴───────┴───────┴─────┘
|
| >>> df.to_dummies(cs.integer(), drop_first=True, separator=":")
| shape: (2, 3)
| ┌───────┬───────┬─────┐
| │ foo:2 ┆ bar:4 ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ str │
| ╞═══════╪═══════╪═════╡
| │ 0 ┆ 0 ┆ a │
| │ 1 ┆ 1 ┆ b │
| └───────┴───────┴─────┘
|
| to_init_repr(self, n: 'int' = 1000) -> 'str'
| Convert DataFrame to instantiable string representation.
|
| Parameters
| ----------
| n
| Only use first n rows.
|
| See Also
| --------
| polars.Series.to_init_repr
| polars.from_repr
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... [
| ... pl.Series("foo", [1, 2, 3], dtype=pl.UInt8),
| ... pl.Series("bar", [6.0, 7.0, 8.0], dtype=pl.Float32),
| ... pl.Series("ham", ["a", "b", "c"], dtype=pl.String),
| ... ]
| ... )
| >>> print(df.to_init_repr())
| pl.DataFrame(
| [
| pl.Series('foo', [1, 2, 3], dtype=pl.UInt8),
| pl.Series('bar', [6.0, 7.0, 8.0], dtype=pl.Float32),
| pl.Series('ham', ['a', 'b', 'c'], dtype=pl.String),
| ]
| )
|
| >>> df_from_str_repr = eval(df.to_init_repr())
| >>> df_from_str_repr
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u8 ┆ f32 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 7.0 ┆ b │
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| to_jax(self, return_type: 'JaxExportType' = 'array', *, device: 'jax.Device | str | None' = None, label: 'str | Expr | Sequence[str | Expr] | None' = None, features: 'str | Expr | Sequence[str | Expr] | None' = None, dtype: 'PolarsDataType | None' = None, order: 'IndexOrder' = 'fortran') -> 'jax.Array | dict[str, jax.Array]'
| Convert DataFrame to a Jax Array, or dict of Jax Arrays.
|
| .. versionadded:: 0.20.27
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| return_type : {"array", "dict"}
| Set return type; a Jax Array, or dict of Jax Arrays.
| device
| Specify the jax `Device` on which the array will be created; can provide
| a string (such as "cpu", "gpu", or "tpu") in which case the device is
| retrieved as `jax.devices(string)[0]`. For more specific control you
| can supply the instantiated `Device` directly. If None, arrays are
| created on the default device.
| label
| One or more column names, expressions, or selectors that label the feature
| data; results in a `{"label": ..., "features": ...}` dict being returned
| when `return_type` is "dict" instead of a `{"col": array, }` dict.
| features
| One or more column names, expressions, or selectors that contain the feature
| data; if omitted, all columns that are not designated as part of the label
| are used. Only applies when `return_type` is "dict".
| dtype
| Unify the dtype of all returned arrays; this casts any column that is
| not already of the required dtype before converting to Array. Note that
| export will be single-precision (32bit) unless the Jax config/environment
| directs otherwise (eg: "jax_enable_x64" was set True in the config object
| at startup, or "JAX_ENABLE_X64" is set to "1" in the environment).
| order : {"c", "fortran"}
| The index order of the returned Jax array, either C-like (row-major) or
| Fortran-like (column-major).
|
| See Also
| --------
| to_dummies
| to_numpy
| to_torch
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "lbl": [0, 1, 2, 3],
| ... "feat1": [1, 0, 0, 1],
| ... "feat2": [1.5, -0.5, 0.0, -2.25],
| ... }
| ... )
|
| Standard return type (2D Array), on the standard device:
|
| >>> df.to_jax()
| Array([[ 0. , 1. , 1.5 ],
| [ 1. , 0. , -0.5 ],
| [ 2. , 0. , 0. ],
| [ 3. , 1. , -2.25]], dtype=float32)
|
| Create the Array on the default GPU device:
|
| >>> a = df.to_jax(device="gpu") # doctest: +SKIP
| >>> a.device() # doctest: +SKIP
| GpuDevice(id=0, process_index=0)
|
| Create the Array on a specific GPU device:
|
| >>> gpu_device = jax.devices("gpu")[1] # doctest: +SKIP
| >>> a = df.to_jax(device=gpu_device) # doctest: +SKIP
| >>> a.device() # doctest: +SKIP
| GpuDevice(id=1, process_index=0)
|
| As a dictionary of individual Arrays:
|
| >>> df.to_jax("dict")
| {'lbl': Array([0, 1, 2, 3], dtype=int32),
| 'feat1': Array([1, 0, 0, 1], dtype=int32),
| 'feat2': Array([ 1.5 , -0.5 , 0. , -2.25], dtype=float32)}
|
| As a "label" and "features" dictionary; note that as "features" is not
| declared, it defaults to all the columns that are not in "label":
|
| >>> df.to_jax("dict", label="lbl")
| {'label': Array([[0],
| [1],
| [2],
| [3]], dtype=int32),
| 'features': Array([[ 1. , 1.5 ],
| [ 0. , -0.5 ],
| [ 0. , 0. ],
| [ 1. , -2.25]], dtype=float32)}
|
| As a "label" and "features" dictionary where each is designated using
| a col or selector expression (which can also be used to cast the data
| if the label and features are better-represented with different dtypes):
|
| >>> import polars.selectors as cs
| >>> df.to_jax(
| ... return_type="dict",
| ... features=cs.float(),
| ... label=pl.col("lbl").cast(pl.UInt8),
| ... )
| {'label': Array([[0],
| [1],
| [2],
| [3]], dtype=uint8),
| 'features': Array([[ 1.5 ],
| [-0.5 ],
| [ 0. ],
| [-2.25]], dtype=float32)}
|
| to_numpy(self, *, order: 'IndexOrder' = 'fortran', writable: 'bool' = False, allow_copy: 'bool' = True, structured: 'bool' = False, use_pyarrow: 'bool | None' = None) -> 'np.ndarray[Any, Any]'
| Convert this DataFrame to a NumPy ndarray.
|
| This operation copies data only when necessary. The conversion is zero copy when
| all of the following hold:
|
| - The DataFrame is fully contiguous in memory, with all Series back-to-back and
| all Series consisting of a single chunk.
| - The data type is an integer or float.
| - The DataFrame contains no null values.
| - The `order` parameter is set to `fortran` (default).
| - The `writable` parameter is set to `False` (default).
|
| Parameters
| ----------
| order
| The index order of the returned NumPy array, either C-like or
| Fortran-like. In general, using the Fortran-like index order is faster.
| However, the C-like order might be more appropriate to use for downstream
| applications to prevent cloning data, e.g. when reshaping into a
| one-dimensional array.
| writable
| Ensure the resulting array is writable. This will force a copy of the data
| if the array was created without copy, as the underlying Arrow data is
| immutable.
| allow_copy
| Allow memory to be copied to perform the conversion. If set to `False`,
| causes conversions that are not zero-copy to fail.
| structured
| Return a `structured array`_ with a data type that corresponds to the
| DataFrame schema. If set to `False` (default), a 2D ndarray is
| returned instead.
|
| .. _structured array: https://numpy.org/doc/stable/user/basics.rec.html
|
| use_pyarrow
| Use `pyarrow.Array.to_numpy
| <https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy>`_
|
| function for the conversion to NumPy if necessary.
|
| .. deprecated:: 0.20.28
| Polars now uses its native engine by default for conversion to NumPy.
|
| Examples
| --------
| Numeric data without nulls can be converted without copying data in some cases.
| The resulting array will not be writable.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> arr = df.to_numpy()
| >>> arr
| array([[1],
| [2],
| [3]])
| >>> arr.flags.writeable
| False
|
| Set `writable=True` to force data copy to make the array writable.
|
| >>> df.to_numpy(writable=True).flags.writeable
| True
|
| If the DataFrame contains different numeric data types, the resulting data type
| will be the supertype. This requires data to be copied. Integer types with
| nulls are cast to a float type with `nan` representing a null value.
|
| >>> df = pl.DataFrame({"a": [1, 2, None], "b": [4.0, 5.0, 6.0]})
| >>> df.to_numpy()
| array([[ 1., 4.],
| [ 2., 5.],
| [nan, 6.]])
|
| Set `allow_copy=False` to raise an error if data would be copied.
|
| >>> s.to_numpy(allow_copy=False) # doctest: +SKIP
| Traceback (most recent call last):
| ...
| RuntimeError: copy not allowed: cannot convert to a NumPy array without copying data
|
| Polars defaults to F-contiguous order. Use `order="c"` to force the resulting
| array to be C-contiguous.
|
| >>> df.to_numpy(order="c").flags.c_contiguous
| True
|
| DataFrames with mixed types will result in an array with an object dtype.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.5, 7.0, 8.5],
| ... "ham": ["a", "b", "c"],
| ... },
| ... schema_overrides={"foo": pl.UInt8, "bar": pl.Float32},
| ... )
| >>> df.to_numpy()
| array([[1, 6.5, 'a'],
| [2, 7.0, 'b'],
| [3, 8.5, 'c']], dtype=object)
|
| Set `structured=True` to convert to a structured array, which can better
| preserve individual column data such as name and data type.
|
| >>> df.to_numpy(structured=True)
| array([(1, 6.5, 'a'), (2, 7. , 'b'), (3, 8.5, 'c')],
| dtype=[('foo', 'u1'), ('bar', '<f4'), ('ham', '<U1')])
|
| to_pandas(self, *, use_pyarrow_extension_array: 'bool' = False, **kwargs: 'Any') -> 'pd.DataFrame'
| Convert this DataFrame to a pandas DataFrame.
|
| This operation copies data if `use_pyarrow_extension_array` is not enabled.
|
| Parameters
| ----------
| use_pyarrow_extension_array
| Use PyArrow-backed extension arrays instead of NumPy arrays for the columns
| of the pandas DataFrame. This allows zero copy operations and preservation
| of null values. Subsequent operations on the resulting pandas DataFrame may
| trigger conversion to NumPy if those operations are not supported by PyArrow
| compute functions.
| **kwargs
| Additional keyword arguments to be passed to
| :meth:`pyarrow.Table.to_pandas`.
|
| Returns
| -------
| :class:`pandas.DataFrame`
|
| Notes
| -----
| This operation requires that both :mod:`pandas` and :mod:`pyarrow` are
| installed.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.to_pandas()
| foo bar ham
| 0 1 6.0 a
| 1 2 7.0 b
| 2 3 8.0 c
|
| Null values in numeric columns are converted to `NaN`.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, None],
| ... "bar": [6.0, None, 8.0],
| ... "ham": [None, "b", "c"],
| ... }
| ... )
| >>> df.to_pandas()
| foo bar ham
| 0 1.0 6.0 NaN
| 1 2.0 NaN b
| 2 NaN 8.0 c
|
| Pass `use_pyarrow_extension_array=True` to get a pandas DataFrame with columns
| backed by PyArrow extension arrays. This will preserve null values.
|
| >>> df.to_pandas(use_pyarrow_extension_array=True)
| foo bar ham
| 0 1 6.0 <NA>
| 1 2 <NA> b
| 2 <NA> 8.0 c
| >>> _.dtypes
| foo int64[pyarrow]
| bar double[pyarrow]
| ham large_string[pyarrow]
| dtype: object
|
| to_series(self, index: 'int' = 0) -> 'Series'
| Select column as Series at index location.
|
| Parameters
| ----------
| index
| Location of selection.
|
| See Also
| --------
| get_column
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.to_series(1)
| shape: (3,)
| Series: 'bar' [i64]
| [
| 6
| 7
| 8
| ]
|
| to_struct(self, name: 'str' = '') -> 'Series'
| Convert a `DataFrame` to a `Series` of type `Struct`.
|
| Parameters
| ----------
| name
| Name for the struct Series
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5],
| ... "b": ["one", "two", "three", "four", "five"],
| ... }
| ... )
| >>> df.to_struct("nums")
| shape: (5,)
| Series: 'nums' [struct[2]]
| [
| {1,"one"}
| {2,"two"}
| {3,"three"}
| {4,"four"}
| {5,"five"}
| ]
|
| to_torch(self, return_type: 'TorchExportType' = 'tensor', *, label: 'str | Expr | Sequence[str | Expr] | None' = None, features: 'str | Expr | Sequence[str | Expr] | None' = None, dtype: 'PolarsDataType | None' = None) -> 'torch.Tensor | dict[str, torch.Tensor] | PolarsDataset'
| Convert DataFrame to a PyTorch Tensor, Dataset, or dict of Tensors.
|
| .. versionadded:: 0.20.23
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| return_type : {"tensor", "dataset", "dict"}
| Set return type; a PyTorch Tensor, PolarsDataset (a frame-specialized
| TensorDataset), or dict of Tensors.
| label
| One or more column names, expressions, or selectors that label the feature
| data; when `return_type` is "dataset", the PolarsDataset will return
| `(features, label)` tensor tuples for each row. Otherwise, it returns
| `(features,)` tensor tuples where the feature contains all the row data.
| features
| One or more column names, expressions, or selectors that contain the feature
| data; if omitted, all columns that are not designated as part of the label
| are used.
| dtype
| Unify the dtype of all returned tensors; this casts any column that is
| not of the required dtype before converting to Tensor. This includes
| the label column *unless* the label is an expression (such as
| `pl.col("label_column").cast(pl.Int16)`).
|
| See Also
| --------
| to_dummies
| to_jax
| to_numpy
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "lbl": [0, 1, 2, 3],
| ... "feat1": [1, 0, 0, 1],
| ... "feat2": [1.5, -0.5, 0.0, -2.25],
| ... }
| ... )
|
| Standard return type (Tensor), with f32 supertype:
|
| >>> df.to_torch(dtype=pl.Float32)
| tensor([[ 0.0000, 1.0000, 1.5000],
| [ 1.0000, 0.0000, -0.5000],
| [ 2.0000, 0.0000, 0.0000],
| [ 3.0000, 1.0000, -2.2500]])
|
| As a dictionary of individual Tensors:
|
| >>> df.to_torch("dict")
| {'lbl': tensor([0, 1, 2, 3]),
| 'feat1': tensor([1, 0, 0, 1]),
| 'feat2': tensor([ 1.5000, -0.5000, 0.0000, -2.2500], dtype=torch.float64)}
|
| As a "label" and "features" dictionary; note that as "features" is not
| declared, it defaults to all the columns that are not in "label":
|
| >>> df.to_torch("dict", label="lbl", dtype=pl.Float32)
| {'label': tensor([[0.],
| [1.],
| [2.],
| [3.]]),
| 'features': tensor([[ 1.0000, 1.5000],
| [ 0.0000, -0.5000],
| [ 0.0000, 0.0000],
| [ 1.0000, -2.2500]])}
|
| As a PolarsDataset, with f64 supertype:
|
| >>> ds = df.to_torch("dataset", dtype=pl.Float64)
| >>> ds[3]
| (tensor([ 3.0000, 1.0000, -2.2500], dtype=torch.float64),)
| >>> ds[:2]
| (tensor([[ 0.0000, 1.0000, 1.5000],
| [ 1.0000, 0.0000, -0.5000]], dtype=torch.float64),)
| >>> ds[[0, 3]]
| (tensor([[ 0.0000, 1.0000, 1.5000],
| [ 3.0000, 1.0000, -2.2500]], dtype=torch.float64),)
|
| As a convenience the PolarsDataset can opt in to half-precision data
| for experimentation (usually this would be set on the model/pipeline):
|
| >>> list(ds.half())
| [(tensor([0.0000, 1.0000, 1.5000], dtype=torch.float16),),
| (tensor([ 1.0000, 0.0000, -0.5000], dtype=torch.float16),),
| (tensor([2., 0., 0.], dtype=torch.float16),),
| (tensor([ 3.0000, 1.0000, -2.2500], dtype=torch.float16),)]
|
| Pass PolarsDataset to a DataLoader, designating the label:
|
| >>> from torch.utils.data import DataLoader
| >>> ds = df.to_torch("dataset", label="lbl")
| >>> dl = DataLoader(ds, batch_size=2)
| >>> batches = list(dl)
| >>> batches[0]
| [tensor([[ 1.0000, 1.5000],
| [ 0.0000, -0.5000]], dtype=torch.float64), tensor([0, 1])]
|
| Note that labels can be given as expressions, allowing them to have
| a dtype independent of the feature columns (multi-column labels are
| supported).
|
| >>> ds = df.to_torch(
| ... return_type="dataset",
| ... dtype=pl.Float32,
| ... label=pl.col("lbl").cast(pl.Int16),
| ... )
| >>> ds[:2]
| (tensor([[ 1.0000, 1.5000],
| [ 0.0000, -0.5000]]), tensor([0, 1], dtype=torch.int16))
|
| Easily integrate with (for example) scikit-learn and other datasets:
|
| >>> from sklearn.datasets import fetch_california_housing # doctest: +SKIP
| >>> housing = fetch_california_housing() # doctest: +SKIP
| >>> df = pl.DataFrame(
| ... data=housing.data,
| ... schema=housing.feature_names,
| ... ).with_columns(
| ... Target=housing.target,
| ... ) # doctest: +SKIP
| >>> train = df.to_torch("dataset", label="Target") # doctest: +SKIP
| >>> loader = DataLoader(
| ... train,
| ... shuffle=True,
| ... batch_size=64,
| ... ) # doctest: +SKIP
|
| top_k(self, k: 'int', *, by: 'IntoExpr | Iterable[IntoExpr]', reverse: 'bool | Sequence[bool]' = False) -> 'DataFrame'
| Return the `k` largest rows.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| k
| Number of rows to return.
| by
| Column(s) used to determine the top rows.
| Accepts expression input. Strings are parsed as column names.
| reverse
| Consider the `k` smallest elements of the `by` column(s) (instead of the `k`
| largest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| bottom_k
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [2, 1, 1, 3, 2, 1],
| ... }
| ... )
|
| Get the rows which contain the 4 largest values in column b.
|
| >>> df.top_k(4, by="b")
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 3 │
| │ a ┆ 2 │
| │ b ┆ 2 │
| │ b ┆ 1 │
| └─────┴─────┘
|
| Get the rows which contain the 4 largest values when sorting on column b and a.
|
| >>> df.top_k(4, by=["b", "a"])
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 3 │
| │ b ┆ 2 │
| │ a ┆ 2 │
| │ c ┆ 1 │
| └─────┴─────┘
|
| transpose(self, *, include_header: 'bool' = False, header_name: 'str' = 'column', column_names: 'str | Iterable[str] | None' = None) -> 'DataFrame'
| Transpose a DataFrame over the diagonal.
|
| Parameters
| ----------
| include_header
| If set, the column names will be added as first column.
| header_name
| If `include_header` is set, this determines the name of the column that will
| be inserted.
| column_names
| Optional iterable yielding strings or a string naming an existing column.
| These will name the value (non-header) columns in the transposed data.
|
| Notes
| -----
| This is a very expensive operation. Perhaps you can do it differently.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
| >>> df.transpose(include_header=True)
| shape: (2, 4)
| ┌────────┬──────────┬──────────┬──────────┐
| │ column ┆ column_0 ┆ column_1 ┆ column_2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞════════╪══════════╪══════════╪══════════╡
| │ a ┆ 1 ┆ 2 ┆ 3 │
| │ b ┆ 4 ┆ 5 ┆ 6 │
| └────────┴──────────┴──────────┴──────────┘
|
| Replace the auto-generated column names with a list
|
| >>> df.transpose(include_header=False, column_names=["x", "y", "z"])
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ x ┆ y ┆ z │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| Include the header as a separate column
|
| >>> df.transpose(
| ... include_header=True, header_name="foo", column_names=["x", "y", "z"]
| ... )
| shape: (2, 4)
| ┌─────┬─────┬─────┬─────┐
| │ foo ┆ x ┆ y ┆ z │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╡
| │ a ┆ 1 ┆ 2 ┆ 3 │
| │ b ┆ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┴─────┘
|
| Replace the auto-generated column with column names from a generator function
|
| >>> def name_generator():
| ... base_name = "my_column_"
| ... count = 0
| ... while True:
| ... yield f"{base_name}{count}"
| ... count += 1
| >>> df.transpose(include_header=False, column_names=name_generator())
| shape: (2, 3)
| ┌─────────────┬─────────────┬─────────────┐
| │ my_column_0 ┆ my_column_1 ┆ my_column_2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════════════╪═════════════╪═════════════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────────────┴─────────────┴─────────────┘
|
| Use an existing column as the new column names
|
| >>> df = pl.DataFrame(dict(id=["i", "j", "k"], a=[1, 2, 3], b=[4, 5, 6]))
| >>> df.transpose(column_names="id")
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ i ┆ j ┆ k │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
| >>> df.transpose(include_header=True, header_name="new_id", column_names="id")
| shape: (2, 4)
| ┌────────┬─────┬─────┬─────┐
| │ new_id ┆ i ┆ j ┆ k │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╪═════╡
| │ a ┆ 1 ┆ 2 ┆ 3 │
| │ b ┆ 4 ┆ 5 ┆ 6 │
| └────────┴─────┴─────┴─────┘
|
| unique(self, subset: 'IntoExpr | Collection[IntoExpr] | None' = None, *, keep: 'UniqueKeepStrategy' = 'any', maintain_order: 'bool' = False) -> 'DataFrame'
| Drop duplicate rows from this DataFrame.
|
| Parameters
| ----------
| subset
| Column name(s), selector(s), or expressions to consider when identifying
| duplicate rows. If set to `None` (default), all columns are considered.
| keep : {'first', 'last', 'any', 'none'}
| Which of the duplicate rows to keep.
|
| * 'any': Does not give any guarantee of which row is kept.
| This allows more optimizations.
| * 'none': Don't keep duplicate rows.
| * 'first': Keep the first unique row.
| * 'last': Keep the last unique row.
| maintain_order
| Keep the same order as the original DataFrame. This is more expensive
| to compute. Settings this to `True` blocks the possibility to run on
| the streaming engine.
|
| Returns
| -------
| DataFrame
| DataFrame with unique rows.
|
| Notes
| -----
| If you're coming from Pandas, this is similar to
| `pandas.DataFrame.drop_duplicates`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 1, 1],
| ... "bar": ["a", "a", "a", "x", "x"],
| ... "ham": ["b", "b", "b", "y", "y"],
| ... }
| ... )
|
| By default, all columns are considered when determining which rows are unique:
|
| >>> df.unique(maintain_order=True)
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
|
| We can also consider only a subset of columns when determining uniqueness,
| controlling which row we keep when duplicates are found:
|
| >>> df.unique(subset="foo", keep="first", maintain_order=True)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| └─────┴─────┴─────┘
| >>> df.unique(subset="foo", keep="last", maintain_order=True)
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
| >>> df.unique(subset="foo", keep="none", maintain_order=True)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| └─────┴─────┴─────┘
|
| Selectors can be used to define the "subset" parameter:
|
| >>> import polars.selectors as cs
| >>> df.unique(subset=cs.string(), maintain_order=True)
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
|
| We can also use an arbitrary expression in the "subset" parameter; in this
| example we use the part of the label in front of ":" to determine uniqueness:
|
| >>> df = pl.DataFrame(
| ... {
| ... "label": ["xx:1", "xx:2", "yy:3", "yy:4"],
| ... "value": [100, 200, 300, 400],
| ... }
| ... )
| >>> df.unique(
| ... subset=pl.col("label").str.extract(r"^(\w+):"),
| ... maintain_order=True,
| ... keep="first",
| ... )
| shape: (2, 2)
| ┌───────┬───────┐
| │ label ┆ value │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═══════╡
| │ xx:1 ┆ 100 │
| │ yy:3 ┆ 300 │
| └───────┴───────┘
|
| unnest(self, columns: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None, *more_columns: 'ColumnNameOrSelector', separator: 'str | None' = None) -> 'DataFrame'
| Decompose struct columns into separate columns for each of their fields.
|
| The new columns will be inserted into the dataframe at the location of the
| struct column.
|
| If no columns are provided, all struct columns are unnested.
|
| Parameters
| ----------
| columns
| Name of the struct column(s) that should be unnested.
| *more_columns
| Additional columns to unnest, specified as positional arguments.
| separator
| Rename output column names as combination of the struct column name,
| name separator and field name.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "before": ["foo", "bar"],
| ... "t_a": [1, 2],
| ... "t_b": ["a", "b"],
| ... "t_c": [True, None],
| ... "t_d": [[1, 2], [3]],
| ... "after": ["baz", "womp"],
| ... }
| ... ).select("before", pl.struct(pl.col("^t_.$")).alias("t_struct"), "after")
| >>> df
| shape: (2, 3)
| ┌────────┬─────────────────────┬───────┐
| │ before ┆ t_struct ┆ after │
| │ --- ┆ --- ┆ --- │
| │ str ┆ struct[4] ┆ str │
| ╞════════╪═════════════════════╪═══════╡
| │ foo ┆ {1,"a",true,[1, 2]} ┆ baz │
| │ bar ┆ {2,"b",null,[3]} ┆ womp │
| └────────┴─────────────────────┴───────┘
| >>> df.unnest("t_struct")
| shape: (2, 6)
| ┌────────┬─────┬─────┬──────┬───────────┬───────┐
| │ before ┆ t_a ┆ t_b ┆ t_c ┆ t_d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪═════╪═════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴─────┴─────┴──────┴───────────┴───────┘
|
| Unnest all struct columns by calling without arguments:
|
| >>> df.unnest()
| shape: (2, 6)
| ┌────────┬─────┬─────┬──────┬───────────┬───────┐
| │ before ┆ t_a ┆ t_b ┆ t_c ┆ t_d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪═════╪═════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴─────┴─────┴──────┴───────────┴───────┘
|
| >>> df = pl.DataFrame(
| ... {
| ... "before": ["foo", "bar"],
| ... "t_a": [1, 2],
| ... "t_b": ["a", "b"],
| ... "t_c": [True, None],
| ... "t_d": [[1, 2], [3]],
| ... "after": ["baz", "womp"],
| ... }
| ... ).select(
| ... "before",
| ... pl.struct(pl.col("^t_.$").name.map(lambda t: t[2:])).alias("t"),
| ... "after",
| ... )
| >>> df.unnest("t", separator="::")
| shape: (2, 6)
| ┌────────┬──────┬──────┬──────┬───────────┬───────┐
| │ before ┆ t::a ┆ t::b ┆ t::c ┆ t::d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪══════╪══════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴──────┴──────┴──────┴───────────┴───────┘
|
| unpivot(self, on: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, *, index: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, variable_name: 'str | None' = None, value_name: 'str | None' = None) -> 'DataFrame'
| Unpivot a DataFrame from wide to long format.
|
| Optionally leaves identifiers set.
|
| This function is useful to massage a DataFrame into a format where one or more
| columns are identifier variables (index) while all other columns, considered
| measured variables (on), are "unpivoted" to the row axis leaving just
| two non-identifier columns, 'variable' and 'value'.
|
| Parameters
| ----------
| on
| Column(s) or selector(s) to use as values variables; if `on`
| is empty no columns will be used. If set to `None` (default)
| all columns that are not in `index` will be used.
| index
| Column(s) or selector(s) to use as identifier variables.
| variable_name
| Name to give to the `variable` column. Defaults to "variable"
| value_name
| Name to give to the `value` column. Defaults to "value"
|
| Notes
| -----
| If you're coming from pandas, this is similar to `pandas.DataFrame.melt`,
| but with `index` replacing `id_vars` and `on` replacing `value_vars`.
| In other frameworks, you might know this operation as `pivot_longer`.
|
| The resulting row order is unspecified.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["x", "y", "z"],
| ... "b": [1, 3, 5],
| ... "c": [2, 4, 6],
| ... }
| ... )
| >>> import polars.selectors as cs
| >>> df.unpivot(cs.numeric(), index="a")
| shape: (6, 3)
| ┌─────┬──────────┬───────┐
| │ a ┆ variable ┆ value │
| │ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 │
| ╞═════╪══════════╪═══════╡
| │ x ┆ b ┆ 1 │
| │ y ┆ b ┆ 3 │
| │ z ┆ b ┆ 5 │
| │ x ┆ c ┆ 2 │
| │ y ┆ c ┆ 4 │
| │ z ┆ c ┆ 6 │
| └─────┴──────────┴───────┘
|
| unstack(self, *, step: 'int', how: 'UnstackDirection' = 'vertical', columns: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, fill_values: 'list[Any] | None' = None) -> 'DataFrame'
| Unstack a long table to a wide form without doing an aggregation.
|
| This can be much faster than a pivot, because it can skip the grouping phase.
|
| Parameters
| ----------
| step
| Number of rows in the unstacked frame.
| how : { 'vertical', 'horizontal' }
| Direction of the unstack.
| columns
| Column name(s) or selector(s) to include in the operation.
| If set to `None` (default), use all columns.
| fill_values
| Fill values that don't fit the new size with this value.
|
| Examples
| --------
| >>> from string import ascii_uppercase
| >>> df = pl.DataFrame(
| ... {
| ... "x": list(ascii_uppercase[0:8]),
| ... "y": pl.int_range(1, 9, eager=True),
| ... }
| ... ).with_columns(
| ... z=pl.int_ranges(pl.col("y"), pl.col("y") + 2, dtype=pl.UInt8),
| ... )
| >>> df
| shape: (8, 3)
| ┌─────┬─────┬──────────┐
| │ x ┆ y ┆ z │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ list[u8] │
| ╞═════╪═════╪══════════╡
| │ A ┆ 1 ┆ [1, 2] │
| │ B ┆ 2 ┆ [2, 3] │
| │ C ┆ 3 ┆ [3, 4] │
| │ D ┆ 4 ┆ [4, 5] │
| │ E ┆ 5 ┆ [5, 6] │
| │ F ┆ 6 ┆ [6, 7] │
| │ G ┆ 7 ┆ [7, 8] │
| │ H ┆ 8 ┆ [8, 9] │
| └─────┴─────┴──────────┘
| >>> df.unstack(step=4, how="vertical")
| shape: (4, 6)
| ┌─────┬─────┬─────┬─────┬──────────┬──────────┐
| │ x_0 ┆ x_1 ┆ y_0 ┆ y_1 ┆ z_0 ┆ z_1 │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 ┆ i64 ┆ list[u8] ┆ list[u8] │
| ╞═════╪═════╪═════╪═════╪══════════╪══════════╡
| │ A ┆ E ┆ 1 ┆ 5 ┆ [1, 2] ┆ [5, 6] │
| │ B ┆ F ┆ 2 ┆ 6 ┆ [2, 3] ┆ [6, 7] │
| │ C ┆ G ┆ 3 ┆ 7 ┆ [3, 4] ┆ [7, 8] │
| │ D ┆ H ┆ 4 ┆ 8 ┆ [4, 5] ┆ [8, 9] │
| └─────┴─────┴─────┴─────┴──────────┴──────────┘
| >>> df.unstack(step=2, how="horizontal")
| shape: (4, 6)
| ┌─────┬─────┬─────┬─────┬──────────┬──────────┐
| │ x_0 ┆ x_1 ┆ y_0 ┆ y_1 ┆ z_0 ┆ z_1 │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 ┆ i64 ┆ list[u8] ┆ list[u8] │
| ╞═════╪═════╪═════╪═════╪══════════╪══════════╡
| │ A ┆ B ┆ 1 ┆ 2 ┆ [1, 2] ┆ [2, 3] │
| │ C ┆ D ┆ 3 ┆ 4 ┆ [3, 4] ┆ [4, 5] │
| │ E ┆ F ┆ 5 ┆ 6 ┆ [5, 6] ┆ [6, 7] │
| │ G ┆ H ┆ 7 ┆ 8 ┆ [7, 8] ┆ [8, 9] │
| └─────┴─────┴─────┴─────┴──────────┴──────────┘
| >>> import polars.selectors as cs
| >>> df.unstack(step=5, columns=cs.numeric(), fill_values=0)
| shape: (5, 2)
| ┌─────┬─────┐
| │ y_0 ┆ y_1 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 6 │
| │ 2 ┆ 7 │
| │ 3 ┆ 8 │
| │ 4 ┆ 0 │
| │ 5 ┆ 0 │
| └─────┴─────┘
|
| update(self, other: 'DataFrame', on: 'str | Sequence[str] | None' = None, how: "Literal['left', 'inner', 'full']" = 'left', *, left_on: 'str | Sequence[str] | None' = None, right_on: 'str | Sequence[str] | None' = None, include_nulls: 'bool' = False, maintain_order: 'MaintainOrderJoin | None' = 'left') -> 'DataFrame'
| Update the values in this `DataFrame` with the values in `other`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| other
| DataFrame that will be used to update the values
| on
| Column names that will be joined on. If set to `None` (default),
| the implicit row index of each frame is used as a join key.
| how : {'left', 'inner', 'full'}
| * 'left' will keep all rows from the left table; rows may be duplicated
| if multiple rows in the right frame match the left row's key.
| * 'inner' keeps only those rows where the key exists in both frames.
| * 'full' will update existing rows where the key matches while also
| adding any new rows contained in the given frame.
| left_on
| Join column(s) of the left DataFrame.
| right_on
| Join column(s) of the right DataFrame.
| include_nulls
| Overwrite values in the left frame with null values from the right frame.
| If set to `False` (default), null values in the right frame are ignored.
| maintain_order : {'none', 'left', 'right', 'left_right', 'right_left'}
| Which order of rows from the inputs to preserve. See :func:`~DataFrame.join`
| for details. Unlike `join` this function preserves the left order by
| default.
|
| Notes
| -----
| This is syntactic sugar for a left/inner join that preserves the order
| of the left `DataFrame` by default, with an optional coalesce when
| `include_nulls = False`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1, 2, 3, 4],
| ... "B": [400, 500, 600, 700],
| ... }
| ... )
| >>> df
| shape: (4, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 400 │
| │ 2 ┆ 500 │
| │ 3 ┆ 600 │
| │ 4 ┆ 700 │
| └─────┴─────┘
| >>> new_df = pl.DataFrame(
| ... {
| ... "B": [-66, None, -99],
| ... "C": [5, 3, 1],
| ... }
| ... )
|
| Update `df` values with the non-null values in `new_df`, by row index:
|
| >>> df.update(new_df)
| shape: (4, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -66 │
| │ 2 ┆ 500 │
| │ 3 ┆ -99 │
| │ 4 ┆ 700 │
| └─────┴─────┘
|
| Update `df` values with the non-null values in `new_df`, by row index,
| but only keeping those rows that are common to both frames:
|
| >>> df.update(new_df, how="inner")
| shape: (3, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -66 │
| │ 2 ┆ 500 │
| │ 3 ┆ -99 │
| └─────┴─────┘
|
| Update `df` values with the non-null values in `new_df`, using a full
| outer join strategy that defines explicit join columns in each frame:
|
| >>> df.update(new_df, left_on=["A"], right_on=["C"], how="full")
| shape: (5, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -99 │
| │ 2 ┆ 500 │
| │ 3 ┆ 600 │
| │ 4 ┆ 700 │
| │ 5 ┆ -66 │
| └─────┴─────┘
|
| Update `df` values including null values in `new_df`, using a full outer
| join strategy that defines explicit join columns in each frame:
|
| >>> df.update(new_df, left_on="A", right_on="C", how="full", include_nulls=True)
| shape: (5, 2)
| ┌─────┬──────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════╡
| │ 1 ┆ -99 │
| │ 2 ┆ 500 │
| │ 3 ┆ null │
| │ 4 ┆ 700 │
| │ 5 ┆ -66 │
| └─────┴──────┘
|
| upsample(self, time_column: 'str', *, every: 'str | timedelta', group_by: 'str | Sequence[str] | None' = None, maintain_order: 'bool' = False) -> 'DataFrame'
| Upsample a DataFrame at a regular frequency.
|
| The `every` argument is created with the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them:
|
| - "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| time_column
| Time column will be used to determine a date_range.
| Note that this column has to be sorted for the output to make sense.
| every
| Interval will start 'every' duration.
| group_by
| First group by these columns and then upsample for every group.
| maintain_order
| Keep the ordering predictable. This is slower.
|
| Returns
| -------
| DataFrame
| Result will be sorted by `time_column` (but note that if `group_by` columns
| are passed, it will only be sorted within each group).
|
| Examples
| --------
| Upsample a DataFrame by a certain interval.
|
| >>> from datetime import datetime
| >>> df = pl.DataFrame(
| ... {
| ... "time": [
| ... datetime(2021, 2, 1),
| ... datetime(2021, 4, 1),
| ... datetime(2021, 5, 1),
| ... datetime(2021, 6, 1),
| ... ],
| ... "groups": ["A", "B", "A", "B"],
| ... "values": [0, 1, 2, 3],
| ... }
| ... ).set_sorted("time")
| >>> df.upsample(
| ... time_column="time", every="1mo", group_by="groups", maintain_order=True
| ... ).select(pl.all().fill_null(strategy="forward"))
| shape: (7, 3)
| ┌─────────────────────┬────────┬────────┐
| │ time ┆ groups ┆ values │
| │ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ str ┆ i64 │
| ╞═════════════════════╪════════╪════════╡
| │ 2021-02-01 00:00:00 ┆ A ┆ 0 │
| │ 2021-03-01 00:00:00 ┆ A ┆ 0 │
| │ 2021-04-01 00:00:00 ┆ A ┆ 0 │
| │ 2021-05-01 00:00:00 ┆ A ┆ 2 │
| │ 2021-04-01 00:00:00 ┆ B ┆ 1 │
| │ 2021-05-01 00:00:00 ┆ B ┆ 1 │
| │ 2021-06-01 00:00:00 ┆ B ┆ 3 │
| └─────────────────────┴────────┴────────┘
|
| var(self, ddof: 'int' = 1) -> 'DataFrame'
| Aggregate the columns of this DataFrame to their variance value.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.var()
| shape: (1, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 1.0 ┆ 1.0 ┆ null │
| └─────┴─────┴──────┘
| >>> df.var(ddof=0)
| shape: (1, 3)
| ┌──────────┬──────────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════════╪══════════╪══════╡
| │ 0.666667 ┆ 0.666667 ┆ null │
| └──────────┴──────────┴──────┘
|
| vstack(self, other: 'DataFrame', *, in_place: 'bool' = False) -> 'DataFrame'
| Grow this DataFrame vertically by stacking a DataFrame to it.
|
| Parameters
| ----------
| other
| DataFrame to stack.
| in_place
| Modify in place.
|
| See Also
| --------
| extend
|
| Examples
| --------
| >>> df1 = pl.DataFrame(
| ... {
| ... "foo": [1, 2],
| ... "bar": [6, 7],
| ... "ham": ["a", "b"],
| ... }
| ... )
| >>> df2 = pl.DataFrame(
| ... {
| ... "foo": [3, 4],
| ... "bar": [8, 9],
| ... "ham": ["c", "d"],
| ... }
| ... )
| >>> df1.vstack(df2)
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ 9 ┆ d │
| └─────┴─────┴─────┘
|
| with_columns(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'DataFrame'
| Add columns to this DataFrame.
|
| Added columns will replace existing columns with the same name.
|
| Parameters
| ----------
| *exprs
| Column(s) to add, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names, other
| non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to add, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| DataFrame
| A new DataFrame with the columns added.
|
| Notes
| -----
| Creating a new DataFrame using this method does not create a new copy of
| existing data.
|
| Examples
| --------
| Pass an expression to add it as a new column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> df.with_columns((pl.col("a") ** 2).alias("a^2"))
| shape: (4, 4)
| ┌─────┬──────┬───────┬─────┐
| │ a ┆ b ┆ c ┆ a^2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 │
| ╞═════╪══════╪═══════╪═════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 │
| │ 2 ┆ 4.0 ┆ true ┆ 4 │
| │ 3 ┆ 10.0 ┆ false ┆ 9 │
| │ 4 ┆ 13.0 ┆ true ┆ 16 │
| └─────┴──────┴───────┴─────┘
|
| Added columns will replace existing columns with the same name.
|
| >>> df.with_columns(pl.col("a").cast(pl.Float64))
| shape: (4, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╡
| │ 1.0 ┆ 0.5 ┆ true │
| │ 2.0 ┆ 4.0 ┆ true │
| │ 3.0 ┆ 10.0 ┆ false │
| │ 4.0 ┆ 13.0 ┆ true │
| └─────┴──────┴───────┘
|
| Multiple columns can be added using positional arguments.
|
| >>> df.with_columns(
| ... (pl.col("a") ** 2).alias("a^2"),
| ... (pl.col("b") / 2).alias("b/2"),
| ... (pl.col("c").not_()).alias("not c"),
| ... )
| shape: (4, 6)
| ┌─────┬──────┬───────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ a^2 ┆ b/2 ┆ not c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 ┆ 0.25 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 4 ┆ 2.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 9 ┆ 5.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 16 ┆ 6.5 ┆ false │
| └─────┴──────┴───────┴─────┴──────┴───────┘
|
| Multiple columns can also be added by passing a list of expressions.
|
| >>> df.with_columns(
| ... [
| ... (pl.col("a") ** 2).alias("a^2"),
| ... (pl.col("b") / 2).alias("b/2"),
| ... (pl.col("c").not_()).alias("not c"),
| ... ]
| ... )
| shape: (4, 6)
| ┌─────┬──────┬───────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ a^2 ┆ b/2 ┆ not c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 ┆ 0.25 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 4 ┆ 2.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 9 ┆ 5.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 16 ┆ 6.5 ┆ false │
| └─────┴──────┴───────┴─────┴──────┴───────┘
|
| Use keyword arguments to easily name your expression inputs.
|
| >>> df.with_columns(
| ... ab=pl.col("a") * pl.col("b"),
| ... not_c=pl.col("c").not_(),
| ... )
| shape: (4, 5)
| ┌─────┬──────┬───────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ ab ┆ not_c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 0.5 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 8.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 30.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 52.0 ┆ false │
| └─────┴──────┴───────┴──────┴───────┘
|
| with_columns_seq(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'DataFrame'
| Add columns to this DataFrame.
|
| Added columns will replace existing columns with the same name.
|
| This will run all expression sequentially instead of in parallel.
| Use this when the work per expression is cheap.
|
| Parameters
| ----------
| *exprs
| Column(s) to add, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names, other
| non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to add, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| DataFrame
| A new DataFrame with the columns added.
|
| See Also
| --------
| with_columns
|
| with_row_count(self, name: 'str' = 'row_nr', offset: 'int' = 0) -> 'DataFrame'
| Add a column at index 0 that counts the rows.
|
| .. deprecated:: 0.20.4
| Use the :meth:`with_row_index` method instead.
| Note that the default column name has changed from 'row_nr' to 'index'.
|
| Parameters
| ----------
| name
| Name of the column to add.
| offset
| Start the row count at this offset. Default = 0
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> df.with_row_count() # doctest: +SKIP
| shape: (3, 3)
| ┌────────┬─────┬─────┐
| │ row_nr ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └────────┴─────┴─────┘
|
| with_row_index(self, name: 'str' = 'index', offset: 'int' = 0) -> 'DataFrame'
| Add a row index as the first column in the DataFrame.
|
| Parameters
| ----------
| name
| Name of the index column.
| offset
| Start the index at this offset. Cannot be negative.
|
| Notes
| -----
| The resulting column does not have any special properties. It is a regular
| column of type `UInt32` (or `UInt64` in `polars[rt64]`).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> df.with_row_index()
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ index ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞═══════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └───────┴─────┴─────┘
| >>> df.with_row_index("id", offset=1000)
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ id ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞══════╪═════╪═════╡
| │ 1000 ┆ 1 ┆ 2 │
| │ 1001 ┆ 3 ┆ 4 │
| │ 1002 ┆ 5 ┆ 6 │
| └──────┴─────┴─────┘
|
| An index column can also be created using the expressions :func:`int_range`
| and :func:`len`.
|
| >>> df.select(
| ... pl.int_range(pl.len(), dtype=pl.UInt32).alias("index"),
| ... pl.all(),
| ... )
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ index ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞═══════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └───────┴─────┴─────┘
|
| write_avro(self, file: 'str | Path | IO[bytes]', compression: 'AvroCompression' = 'uncompressed', name: 'str' = '') -> 'None'
| Write to Apache Avro file.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the data will be written.
| compression : {'uncompressed', 'snappy', 'deflate'}
| Compression method. Defaults to "uncompressed".
| name
| Schema name. Defaults to empty string.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.avro"
| >>> df.write_avro(path)
|
| write_clipboard(self, *, separator: 'str' = '\t', **kwargs: 'Any') -> 'None'
| Copy `DataFrame` in csv format to the system clipboard with `write_csv`.
|
| Useful for pasting into Excel or other similar spreadsheet software.
|
| Parameters
| ----------
| separator
| Separate CSV fields with this symbol.
| kwargs
| Additional arguments to pass to `write_csv`.
|
| See Also
| --------
| polars.read_clipboard: Read a DataFrame from the clipboard.
| write_csv: Write to comma-separated values (CSV) file.
|
| write_csv(self, file: 'str | Path | IO[str] | IO[bytes] | None' = None, *, include_bom: 'bool' = False, compression: "Literal['uncompressed', 'gzip', 'zstd']" = 'uncompressed', compression_level: 'int | None' = None, check_extension: 'bool' = True, include_header: 'bool' = True, separator: 'str' = ',', line_terminator: 'str' = '\n', quote_char: 'str' = '"', batch_size: 'int' = 1024, datetime_format: 'str | None' = None, date_format: 'str | None' = None, time_format: 'str | None' = None, float_scientific: 'bool | None' = None, float_precision: 'int | None' = None, decimal_comma: 'bool' = False, null_value: 'str | None' = None, quote_style: 'CsvQuoteStyle | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None) -> 'str | None'
| Write to comma-separated values (CSV) file.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| If set to `None` (default), the output is returned as a string instead.
| include_bom
| Whether to include UTF-8 BOM in the CSV output.
| compression
| What compression format to use.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| compression_level
| The compression level to use, typically 0-9 or `None` to let the
| engine choose.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| check_extension
| Whether to check if the filename matches the compression settings.
| Will raise an error if compression is set to 'uncompressed' and the
| filename ends in one of (".gz", ".zst", ".zstd") or if
| compression != 'uncompressed' and the filename does not end in the
| appropriate extension. Only applies if file is a path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| include_header
| Whether to include header in the CSV output.
| separator
| Separate CSV fields with this symbol.
| line_terminator
| String used to end each row.
| quote_char
| Byte to use as quoting character.
| batch_size
| Number of rows that will be processed per thread.
| datetime_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate. If no format specified, the default fractional-second
| precision is inferred from the maximum timeunit found in the frame's
| Datetime cols (if any).
| date_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate.
| time_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate.
| float_scientific
| Whether to use scientific form always (true), never (false), or
| automatically (None) for floating-point datatypes.
| float_precision
| Number of decimal places to write, applied to both floating-point
| data types.
| decimal_comma
| Use a comma as the decimal separator instead of a point in standard
| notation. Floats will be encapsulated in quotes if necessary; set the
| field separator to override.
| null_value
| A string representing null values (defaulting to the empty string).
| quote_style : {'necessary', 'always', 'non_numeric', 'never'}
| Determines the quoting strategy used.
|
| - necessary (default): This puts quotes around fields only when necessary.
| They are necessary when fields contain a quote,
| separator or record terminator.
| Quotes are also necessary when writing an empty record
| (which is indistinguishable from a record with one empty field).
| This is the default.
| - always: This puts quotes around every field. Always.
| - never: This never puts quotes around fields, even if that results in
| invalid CSV data (e.g.: by not quoting strings containing the separator).
| - non_numeric: This puts quotes around all fields that are non-numeric.
| Namely, when writing a field that does not parse as a valid float
| or integer, then quotes will be used even if they aren`t strictly
| necessary.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.csv"
| >>> df.write_csv(path, separator=",")
|
| write_database(self, table_name: 'str', connection: 'ConnectionOrCursor | str', *, if_table_exists: 'DbWriteMode' = 'fail', engine: 'DbWriteEngine | None' = None, engine_options: 'dict[str, Any] | None' = None) -> 'int'
| Write the data in a Polars DataFrame to a database.
|
| .. versionadded:: 0.20.26
| Support for instantiated connection objects in addition to URI strings, and
| a new `engine_options` parameter.
|
| Parameters
| ----------
| table_name
| Schema-qualified name of the table to create or append to in the target
| SQL database. If your table name contains special characters, it should
| be quoted.
| connection
| An existing SQLAlchemy or ADBC connection against the target database, or
| a URI string that will be used to instantiate such a connection, such as:
|
| * "postgresql://user:pass@server:port/database"
| * "sqlite:////path/to/database.db"
| if_table_exists : {'append', 'replace', 'fail'}
| The insert mode:
|
| * 'replace' will create a new database table, overwriting an existing one.
| * 'append' will append to an existing table.
| * 'fail' will fail if table already exists.
| engine : {'sqlalchemy', 'adbc'}
| Select the engine to use for writing frame data; only necessary when
| supplying a URI string (defaults to 'sqlalchemy' if unset)
| engine_options
| Additional options to pass to the insert method associated with the engine
| specified by the option `engine`.
|
| * Setting `engine` to "sqlalchemy" currently inserts using Pandas' `to_sql`
| method (though this will eventually be phased out in favor of a native
| solution).
| * Setting `engine` to "adbc" inserts using the ADBC cursor's `adbc_ingest`
| method. Note that when passing an instantiated connection object, PyArrow
| is required for SQLite and Snowflake drivers.
|
| Examples
| --------
| Insert into a temporary table using a PostgreSQL URI and the ADBC engine:
|
| >>> df.write_database(
| ... table_name="target_table",
| ... connection="postgresql://user:pass@server:port/database",
| ... engine="adbc",
| ... engine_options={"temporary": True},
| ... ) # doctest: +SKIP
|
| Insert into a table using a `pyodbc` SQLAlchemy connection to SQL Server
| that was instantiated with "fast_executemany=True" to improve performance:
|
| >>> pyodbc_uri = (
| ... "mssql+pyodbc://user:pass@server:1433/test?"
| ... "driver=ODBC+Driver+18+for+SQL+Server"
| ... )
| >>> engine = create_engine(pyodbc_uri, fast_executemany=True) # doctest: +SKIP
| >>> df.write_database(
| ... table_name="target_table",
| ... connection=engine,
| ... ) # doctest: +SKIP
|
| Returns
| -------
| int
| The number of rows affected, if the driver provides this information.
| Otherwise, returns -1.
|
| write_delta(self, target: 'str | Path | deltalake.DeltaTable', *, mode: "Literal['error', 'append', 'overwrite', 'ignore', 'merge']" = 'error', overwrite_schema: 'bool | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', delta_write_options: 'dict[str, Any] | None' = None, delta_merge_options: 'dict[str, Any] | None' = None) -> 'deltalake.table.TableMerger | None'
| Write DataFrame as delta table.
|
| Parameters
| ----------
| target
| URI of a table or a DeltaTable object.
| mode : {'error', 'append', 'overwrite', 'ignore', 'merge'}
| How to handle existing data.
|
| - If 'error', throw an error if the table already exists (default).
| - If 'append', will add new data.
| - If 'overwrite', will replace table with new data.
| - If 'ignore', will not write anything if table already exists.
| - If 'merge', return a `TableMerger` object to merge data from the DataFrame
| with the existing data.
| overwrite_schema
| If True, allows updating the schema of the table.
|
| .. deprecated:: 0.20.14
| Use the parameter `delta_write_options` instead and pass
| `{"schema_mode": "overwrite"}`.
| storage_options
| Extra options for the storage backends supported by `deltalake`.
| For cloud storages, this may include configurations for authentication etc.
|
| - See a list of supported storage options for S3 `here <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html#variants>`__.
| - See a list of supported storage options for GCS `here <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html#variants>`__.
| - See a list of supported storage options for Azure `here <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants>`__.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| delta_write_options
| Additional keyword arguments while writing a Delta lake Table.
| See a list of supported write options `here <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake>`__.
| delta_merge_options
| Keyword arguments which are required to `MERGE` a Delta lake Table.
| See a list of supported merge options `here <https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.merge>`__.
|
| Raises
| ------
| TypeError
| If the DataFrame contains unsupported data types.
| ArrowInvalidError
| If the DataFrame contains data types that could not be cast to their
| primitive type.
| TableNotFoundError
| If the delta table doesn't exist and MERGE action is triggered
|
| Notes
| -----
| The Polars data types :class:`Null` and :class:`Time` are not supported
| by the delta protocol specification and will raise a TypeError. Columns
| using The :class:`Categorical` data type will be converted to
| normal (non-categorical) strings when written.
|
| Polars columns are always nullable. To write data to a delta table with
| non-nullable columns, a custom pyarrow schema has to be passed to the
| `delta_write_options`. See the last example below.
|
| Examples
| --------
| Write a dataframe to the local filesystem as a Delta Lake table.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> table_path = "/path/to/delta-table/"
| >>> df.write_delta(table_path) # doctest: +SKIP
|
| Append data to an existing Delta Lake table on the local filesystem.
| Note that this will fail if the schema of the new data does not match the
| schema of the existing table.
|
| >>> df.write_delta(table_path, mode="append") # doctest: +SKIP
|
| Overwrite a Delta Lake table as a new version.
| If the schemas of the new and old data are the same, specifying the
| `schema_mode` is not required.
|
| >>> existing_table_path = "/path/to/delta-table/"
| >>> df.write_delta(
| ... existing_table_path,
| ... mode="overwrite",
| ... delta_write_options={"schema_mode": "overwrite"},
| ... ) # doctest: +SKIP
|
| Write a DataFrame as a Delta Lake table to a cloud object store like S3.
|
| >>> table_path = "s3://bucket/prefix/to/delta-table/"
| >>> df.write_delta(
| ... table_path,
| ... storage_options={
| ... "AWS_REGION": "THE_AWS_REGION",
| ... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
| ... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
| ... },
| ... ) # doctest: +SKIP
|
| Write DataFrame as a Delta Lake table with non-nullable columns.
|
| >>> import pyarrow as pa
| >>> existing_table_path = "/path/to/delta-table/"
| >>> df.write_delta(
| ... existing_table_path,
| ... delta_write_options={
| ... "schema": pa.schema([pa.field("foo", pa.int64(), nullable=False)])
| ... },
| ... ) # doctest: +SKIP
|
| Write DataFrame as a Delta Lake table with zstd compression.
| For all `delta_write_options` keyword arguments, check the deltalake docs
| `here
| <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake>`__,
| and for Writer Properties in particular `here
| <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.WriterProperties>`__.
|
| >>> import deltalake
| >>> df.write_delta(
| ... table_path,
| ... delta_write_options={
| ... "writer_properties": deltalake.WriterProperties(compression="zstd"),
| ... },
| ... ) # doctest: +SKIP
|
| Merge the DataFrame with an existing Delta Lake table.
| For all `TableMerger` methods, check the deltalake docs
| `here <https://delta-io.github.io/delta-rs/api/delta_table/delta_table_merger/>`__.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> table_path = "/path/to/delta-table/"
| >>> (
| ... df.write_delta(
| ... "table_path",
| ... mode="merge",
| ... delta_merge_options={
| ... "predicate": "s.foo = t.foo",
| ... "source_alias": "s",
| ... "target_alias": "t",
| ... },
| ... )
| ... .when_matched_update_all()
| ... .when_not_matched_insert_all()
| ... .execute()
| ... ) # doctest: +SKIP
|
| write_excel(self, workbook: 'str | Workbook | IO[bytes] | Path | None' = None, worksheet: 'str | Worksheet | None' = None, *, position: 'tuple[int, int] | str' = 'A1', table_style: 'str | dict[str, Any] | None' = None, table_name: 'str | None' = None, column_formats: 'ColumnFormatDict | None' = None, dtype_formats: 'dict[OneOrMoreDataTypes, str] | None' = None, conditional_formats: 'ConditionalFormatDict | None' = None, header_format: 'dict[str, Any] | None' = None, column_totals: 'ColumnTotalsDefinition | None' = None, column_widths: 'ColumnWidthsDefinition | None' = None, row_totals: 'RowTotalsDefinition | None' = None, row_heights: 'dict[int | tuple[int, ...], int] | int | None' = None, sparklines: 'dict[str, Sequence[str] | dict[str, Any]] | None' = None, formulas: 'dict[str, str | dict[str, str]] | None' = None, float_precision: 'int' = 3, include_header: 'bool' = True, autofilter: 'bool' = True, autofit: 'bool' = False, hidden_columns: 'Sequence[str] | SelectorType | None' = None, hide_gridlines: 'bool' = False, sheet_zoom: 'int | None' = None, freeze_panes: 'str | tuple[int, int] | tuple[str, int, int] | tuple[int, int, int, int] | None' = None, use_zip64: 'bool' = False) -> 'Workbook'
| Write frame data to a table in an Excel workbook/worksheet.
|
| Parameters
| ----------
| workbook : {str, Workbook}
| String name or path of the workbook to create, BytesIO object, file opened
| in binary-mode, or an `xlsxwriter.Workbook` object that has not been closed.
| If None, writes to `dataframe.xlsx` in the working directory.
| worksheet : {str, Worksheet}
| Name of target worksheet or an `xlsxwriter.Worksheet` object (in which
| case `workbook` must be the parent `xlsxwriter.Workbook` object); if None,
| writes to "Sheet1" when creating a new workbook (note that writing to an
| existing workbook requires a valid existing -or new- worksheet name).
| position : {str, tuple}
| Table position in Excel notation (eg: "A1"), or a (row,col) integer tuple.
| table_style : {str, dict}
| A named Excel table style, such as "Table Style Medium 4", or a dictionary
| of `{"key":value,}` options containing one or more of the following keys:
| "style", "first_column", "last_column", "banded_columns, "banded_rows".
| table_name : str
| Name of the output table object in the worksheet; can then be referred to
| in the sheet by formulae/charts, or by subsequent `xlsxwriter` operations.
| column_formats : dict
| A `{colname(s):str,}` or `{selector:str,}` dictionary for applying an
| Excel format string to the given columns. Formats defined here (such as
| "dd/mm/yyyy", "0.00%", etc) will override any defined in `dtype_formats`.
| dtype_formats : dict
| A `{dtype:str,}` dictionary that sets the default Excel format for the
| given dtype. (This can be overridden on a per-column basis by the
| `column_formats` param).
| conditional_formats : dict
| A dictionary of colname (or selector) keys to a format str, dict, or list
| that defines conditional formatting options for the specified columns.
|
| * If supplying a string typename, should be one of the valid `xlsxwriter`
| types such as "3_color_scale", "data_bar", etc.
| * If supplying a dictionary you can make use of any/all `xlsxwriter`
| supported options, including icon sets, formulae, etc.
| * Supplying multiple columns as a tuple/key will apply a single format
| across all columns - this is effective in creating a heatmap, as the
| min/max values will be determined across the entire range, not per-column.
| * Finally, you can also supply a list made up from the above options
| in order to apply *more* than one conditional format to the same range.
| header_format : dict
| A `{key:value,}` dictionary of `xlsxwriter` format options to apply
| to the table header row, such as `{"bold":True, "font_color":"#702963"}`.
| column_totals : {bool, list, dict}
| Add a column-total row to the exported table.
|
| * If True, all numeric columns will have an associated total using "sum".
| * If passing a string, it must be one of the valid total function names
| and all numeric columns will have an associated total using that function.
| * If passing a list of colnames, only those given will have a total.
| * For more control, pass a `{colname:funcname,}` dict.
|
| Valid column-total function names are "average", "count_nums", "count",
| "max", "min", "std_dev", "sum", and "var".
| column_widths : {dict, int}
| A `{colname:int,}` or `{selector:int,}` dict or a single integer that
| sets (or overrides if autofitting) table column widths, in integer pixel
| units. If given as an integer the same value is used for all table columns.
| row_totals : {dict, list, bool}
| Add a row-total column to the right-hand side of the exported table.
|
| * If True, a column called "total" will be added at the end of the table
| that applies a "sum" function row-wise across all numeric columns.
| * If passing a list/sequence of column names, only the matching columns
| will participate in the sum.
| * Can also pass a `{colname:columns,}` dictionary to create one or
| more total columns with distinct names, referencing different columns.
| row_heights : {dict, int}
| An int or `{row_index:int,}` dictionary that sets the height of the given
| rows (if providing a dictionary) or all rows (if providing an integer) that
| intersect with the table body (including any header and total row) in
| integer pixel units. Note that `row_index` starts at zero and will be
| the header row (unless `include_header` is False).
| sparklines : dict
| A `{colname:list,}` or `{colname:dict,}` dictionary defining one or more
| sparklines to be written into a new column in the table.
|
| * If passing a list of colnames (used as the source of the sparkline data)
| the default sparkline settings are used (eg: line chart with no markers).
| * For more control an `xlsxwriter`-compliant options dict can be supplied,
| in which case three additional polars-specific keys are available:
| "columns", "insert_before", and "insert_after". These allow you to define
| the source columns and position the sparkline(s) with respect to other
| table columns. If no position directive is given, sparklines are added to
| the end of the table (eg: to the far right) in the order they are given.
| formulas : dict
| A `{colname:formula,}` or `{colname:dict,}` dictionary defining one or
| more formulas to be written into a new column in the table. Note that you
| are strongly advised to use structured references in your formulae wherever
| possible to make it simple to reference columns by name.
|
| * If providing a string formula (such as "=[@colx]*[@coly]") the column will
| be added to the end of the table (eg: to the far right), after any default
| sparklines and before any row_totals.
| * For the most control supply an options dictionary with the following keys:
| "formula" (mandatory), one of "insert_before" or "insert_after", and
| optionally "return_dtype". The latter is used to appropriately format the
| output of the formula and allow it to participate in row/column totals.
| float_precision : int
| Default number of decimals displayed for floating point columns (note that
| this is purely a formatting directive; the actual values are not rounded).
| include_header : bool
| Indicate if the table should be created with a header row.
| autofilter : bool
| If the table has headers, provide autofilter capability.
| autofit : bool
| Calculate individual column widths from the data.
| hidden_columns : str | list
| A column name, list of column names, or a selector representing table
| columns to mark as hidden in the output worksheet.
| hide_gridlines : bool
| Do not display any gridlines on the output worksheet.
| sheet_zoom : int
| Set the default zoom level of the output worksheet.
| freeze_panes : str | (str, int, int) | (int, int) | (int, int, int, int)
| Freeze workbook panes.
|
| * If (row, col) is supplied, panes are split at the top-left corner of the
| specified cell, which are 0-indexed. Thus, to freeze only the top row,
| supply (1, 0).
| * Alternatively, cell notation can be used to supply the cell. For example,
| "A2" indicates the split occurs at the top-left of cell A2, which is the
| equivalent of (1, 0).
| * If (row, col, top_row, top_col) are supplied, the panes are split based on
| the `row` and `col`, and the scrolling region is initialized to begin at
| the `top_row` and `top_col`. Thus, to freeze only the top row and have the
| scrolling region begin at row 10, column D (5th col), supply (1, 0, 9, 4).
| Using cell notation for (row, col), supplying ("A2", 9, 4) is equivalent.
| use_zip64 : bool
| Whether to use ZIP64 extensions when writing the Workbook. This allows for
| writing exceptionally large workbook files (>=4GB when uncompressed), but
| is less broadly compatible.
|
| Notes
| -----
| * A list of compatible `xlsxwriter` format property names can be found here:
| https://xlsxwriter.readthedocs.io/format.html#format-methods-and-format-properties
|
| * Conditional formatting dictionaries should provide xlsxwriter-compatible
| definitions; polars will take care of how they are applied on the worksheet
| with respect to the relative sheet/column position. For supported options,
| see: https://xlsxwriter.readthedocs.io/working_with_conditional_formats.html
|
| * Similarly, sparkline option dictionaries should contain xlsxwriter-compatible
| key/values, as well as a mandatory polars "columns" key that defines the
| sparkline source data; these source columns should all be adjacent. Two other
| polars-specific keys are available to help define where the sparkline appears
| in the table: "insert_after", and "insert_before". The value associated with
| these keys should be the name of a column in the exported table.
| https://xlsxwriter.readthedocs.io/working_with_sparklines.html
|
| * Formula dictionaries *must* contain a key called "formula", and then optional
| "insert_after", "insert_before", and/or "return_dtype" keys. These additional
| keys allow the column to be injected into the table at a specific location,
| and/or to define the return type of the formula (eg: "Int64", "Float64", etc).
| Formulas that refer to table columns should use Excel's structured references
| syntax to ensure the formula is applied correctly and is table-relative.
| https://support.microsoft.com/en-us/office/using-structured-references-with-excel-tables-f5ed2452-2337-4f71-bed3-c8ae6d2b276e
|
| * If you want unformatted output, you can use a selector to apply the "General"
| format to all columns (or all *non-temporal* columns to preserve formatting
| of date/datetime columns), eg: `column_formats={~cs.temporal(): "General"}`.
|
| Examples
| --------
| Instantiate a basic DataFrame:
|
| >>> from random import uniform
| >>> from datetime import date
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "dtm": [date(2023, 1, 1), date(2023, 1, 2), date(2023, 1, 3)],
| ... "num": [uniform(-500, 500), uniform(-500, 500), uniform(-500, 500)],
| ... "val": [10_000, 20_000, 30_000],
| ... }
| ... )
|
| Export to "dataframe.xlsx" (the default workbook name, if not specified) in the
| working directory, add column totals on all numeric columns ("sum" by default),
| then autofit:
|
| >>> df.write_excel(column_totals=True, autofit=True) # doctest: +SKIP
|
| Write frame to a specific location on the sheet, set a named table style,
| apply US-style date formatting, increase floating point formatting precision,
| apply a non-default column total function to a specific column, autofit:
|
| >>> df.write_excel( # doctest: +SKIP
| ... position="B4",
| ... table_style="Table Style Light 16",
| ... dtype_formats={pl.Date: "mm/dd/yyyy"},
| ... column_totals={"num": "average"},
| ... float_precision=6,
| ... autofit=True,
| ... )
|
| Write the same frame to a named worksheet twice, applying different styles
| and conditional formatting to each table, adding custom-formatted table
| titles using explicit `xlsxwriter` integration:
|
| >>> from xlsxwriter import Workbook
| >>> with Workbook("multi_frame.xlsx") as wb: # doctest: +SKIP
| ... # basic/default conditional formatting
| ... df.write_excel(
| ... workbook=wb,
| ... worksheet="data",
| ... position=(3, 1), # specify position as (row,col) coordinates
| ... conditional_formats={"num": "3_color_scale", "val": "data_bar"},
| ... table_style="Table Style Medium 4",
| ... )
| ...
| ... # advanced conditional formatting, custom styles
| ... df.write_excel(
| ... workbook=wb,
| ... worksheet="data",
| ... position=(df.height + 7, 1),
| ... table_style={
| ... "style": "Table Style Light 4",
| ... "first_column": True,
| ... },
| ... conditional_formats={
| ... "num": {
| ... "type": "3_color_scale",
| ... "min_color": "#76933c",
| ... "mid_color": "#c4d79b",
| ... "max_color": "#ebf1de",
| ... },
| ... "val": {
| ... "type": "data_bar",
| ... "data_bar_2010": True,
| ... "bar_color": "#9bbb59",
| ... "bar_negative_color_same": True,
| ... "bar_negative_border_color_same": True,
| ... },
| ... },
| ... column_formats={"num": "#,##0.000;[White]-#,##0.000"},
| ... column_widths={"val": 125},
| ... autofit=True,
| ... )
| ...
| ... # add some table titles (with a custom format)
| ... ws = wb.get_worksheet_by_name("data")
| ... fmt_title = wb.add_format(
| ... {
| ... "font_color": "#4f6228",
| ... "font_size": 12,
| ... "italic": True,
| ... "bold": True,
| ... }
| ... )
| ... ws.write(2, 1, "Basic/default conditional formatting", fmt_title)
| ... ws.write(df.height + 6, 1, "Custom conditional formatting", fmt_title)
|
| Export a table containing two different types of sparklines. Use default
| options for the "trend" sparkline and customized options (and positioning)
| for the "+/-" `win_loss` sparkline, with non-default integer formatting,
| column totals, a subtle two-tone heatmap and hidden worksheet gridlines:
|
| >>> df = pl.DataFrame(
| ... {
| ... "id": ["aaa", "bbb", "ccc", "ddd", "eee"],
| ... "q1": [100, 55, -20, 0, 35],
| ... "q2": [30, -10, 15, 60, 20],
| ... "q3": [-50, 0, 40, 80, 80],
| ... "q4": [75, 55, 25, -10, -55],
| ... }
| ... )
| >>> df.write_excel( # doctest: +SKIP
| ... table_style="Table Style Light 2",
| ... # apply accounting format to all flavours of integer
| ... dtype_formats={dt: "#,##0_);(#,##0)" for dt in [pl.Int32, pl.Int64]},
| ... sparklines={
| ... # default options; just provide source cols
| ... "trend": ["q1", "q2", "q3", "q4"],
| ... # customized sparkline type, with positioning directive
| ... "+/-": {
| ... "columns": ["q1", "q2", "q3", "q4"],
| ... "insert_after": "id",
| ... "type": "win_loss",
| ... },
| ... },
| ... conditional_formats={
| ... # create a unified multi-column heatmap
| ... ("q1", "q2", "q3", "q4"): {
| ... "type": "2_color_scale",
| ... "min_color": "#95b3d7",
| ... "max_color": "#ffffff",
| ... },
| ... },
| ... column_totals=["q1", "q2", "q3", "q4"],
| ... row_totals=True,
| ... hide_gridlines=True,
| ... )
|
| Export a table containing an Excel formula-based column that calculates a
| standardised Z-score, showing use of structured references in conjunction
| with positioning directives, column totals, and custom formatting.
|
| >>> df = pl.DataFrame(
| ... {
| ... "id": ["a123", "b345", "c567", "d789", "e101"],
| ... "points": [99, 45, 50, 85, 35],
| ... }
| ... )
| >>> df.write_excel( # doctest: +SKIP
| ... table_style={
| ... "style": "Table Style Medium 15",
| ... "first_column": True,
| ... },
| ... column_formats={
| ... "id": {"font": "Consolas"},
| ... "points": {"align": "center"},
| ... "z-score": {"align": "center"},
| ... },
| ... column_totals="average",
| ... formulas={
| ... "z-score": {
| ... # use structured references to refer to the table columns and 'totals' row
| ... "formula": "=STANDARDIZE([@points], [[#Totals],[points]], STDEV([points]))",
| ... "insert_after": "points",
| ... "return_dtype": pl.Float64,
| ... }
| ... },
| ... hide_gridlines=True,
| ... sheet_zoom=125,
| ... )
|
| Create and reference a Worksheet object directly, adding a basic chart.
| Taking advantage of structured references to set chart series values and
| categories is *strongly* recommended so you do not have to calculate
| cell positions with respect to the frame data and worksheet:
|
| >>> with Workbook("basic_chart.xlsx") as wb: # doctest: +SKIP
| ... # create worksheet object and write frame data to it
| ... ws = wb.add_worksheet("demo")
| ... df.write_excel(
| ... workbook=wb,
| ... worksheet=ws,
| ... table_name="DataTable",
| ... table_style="Table Style Medium 26",
| ... hide_gridlines=True,
| ... )
| ... # create chart object, point to the written table
| ... # data using structured references, and style it
| ... chart = wb.add_chart({"type": "column"})
| ... chart.set_title({"name": "Example Chart"})
| ... chart.set_legend({"none": True})
| ... chart.set_style(38)
| ... chart.add_series(
| ... { # note the use of structured references
| ... "values": "=DataTable[points]",
| ... "categories": "=DataTable[id]",
| ... "data_labels": {"value": True},
| ... }
| ... )
| ... # add chart to the worksheet
| ... ws.insert_chart("D1", chart)
|
| Export almost entirely unformatted data (no numeric styling or standardised
| floating point precision), omit autofilter, but keep date/datetime formatting:
|
| >>> import polars.selectors as cs
| >>> df = pl.DataFrame(
| ... {
| ... "n1": [-100, None, 200, 555],
| ... "n2": [987.4321, -200, 44.444, 555.5],
| ... }
| ... )
| >>> df.write_excel( # doctest: +SKIP
| ... column_formats={~cs.temporal(): "General"},
| ... autofilter=False,
| ... )
|
| write_iceberg(self, target: 'str | pyiceberg.table.Table', mode: "Literal['append', 'overwrite']") -> 'None'
| Write DataFrame to an Iceberg table.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| target
| Name of the table or the Table object representing an Iceberg table.
| mode : {'append', 'overwrite'}
| How to handle existing data.
|
| - If 'append', will add new data.
| - If 'overwrite', will replace table with new data.
|
| write_ipc(self, file: 'str | Path | IO[bytes] | None', *, compression: 'IpcCompression' = 'uncompressed', compat_level: 'CompatLevel | None' = None, record_batch_size: 'int | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None) -> 'BytesIO | None'
| Write to Arrow IPC binary stream or Feather file.
|
| See "File or Random Access format" in https://arrow.apache.org/docs/python/ipc.html.
|
| .. versionchanged:: 1.1
| The `future` parameter was renamed `compat_level`.
|
| Parameters
| ----------
| file
| Path or writable file-like object to which the IPC data will be
| written. If set to `None`, the output is returned as a BytesIO object.
| compression : {'uncompressed', 'lz4', 'zstd'}
| Compression method. Defaults to "uncompressed".
| compat_level
| Use a specific compatibility level
| when exporting Polars' internal data structures.
| record_batch_size
| Size of the record batches in number of rows.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.arrow"
| >>> df.write_ipc(path)
|
| Write to a ``BytesIO`` object by passing ``file=None``. The returned
| buffer's position is at the end of the written data, so call ``seek(0)``
| before reading it back.
|
| >>> buf = df.write_ipc(file=None)
| >>> buf.seek(0)
| 0
| >>> pl.read_ipc(buf).equals(df)
| True
|
| write_ipc_stream(self, file: 'str | Path | IO[bytes] | None', *, compression: 'IpcCompression' = 'uncompressed', compat_level: 'CompatLevel | None' = None) -> 'BytesIO | None'
| Write to Arrow IPC record batch stream.
|
| See "Streaming format" in https://arrow.apache.org/docs/python/ipc.html.
|
| .. versionchanged:: 1.1
| The `future` parameter was renamed `compat_level`.
|
| Parameters
| ----------
| file
| Path or writable file-like object to which the IPC record batch data will
| be written. If set to `None`, the output is returned as a BytesIO object.
| compression : {'uncompressed', 'lz4', 'zstd'}
| Compression method. Defaults to "uncompressed".
| compat_level
| Use a specific compatibility level
| when exporting Polars' internal data structures.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.arrow"
| >>> df.write_ipc_stream(path)
|
| write_json(self, file: 'IOBase | str | Path | None' = None) -> 'str | None'
| Serialize to JSON representation.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| If set to `None` (default), the output is returned as a string instead.
|
| See Also
| --------
| DataFrame.write_ndjson
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... }
| ... )
| >>> df.write_json()
| '[{"foo":1,"bar":6},{"foo":2,"bar":7},{"foo":3,"bar":8}]'
|
| write_ndjson(self, file: 'str | Path | IO[bytes] | IO[str] | None' = None, *, compression: "Literal['uncompressed', 'gzip', 'zstd']" = 'uncompressed', compression_level: 'int | None' = None, check_extension: 'bool' = True) -> 'str | None'
| Serialize to newline delimited JSON representation.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| If set to `None` (default), the output is returned as a string instead.
| compression
| What compression format to use.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| compression_level
| The compression level to use, typically 0-9 or `None` to let the
| engine choose.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| check_extension
| Whether to check if the filename matches the compression settings.
| Will raise an error if compression is set to 'uncompressed' and the
| filename ends in one of (".gz", ".zst", ".zstd") or if
| compression != 'uncompressed' and the file uses an mismatched
| extension. Only applies if file is a path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... }
| ... )
| >>> df.write_ndjson()
| '{"foo":1,"bar":6}\n{"foo":2,"bar":7}\n{"foo":3,"bar":8}\n'
|
| write_parquet(self, file: 'str | Path | IO[bytes]', *, compression: 'ParquetCompression' = 'zstd', compression_level: 'int | None' = None, statistics: 'bool | str | dict[str, bool]' = True, row_group_size: 'int | None' = None, data_page_size: 'int | None' = None, use_pyarrow: 'bool' = False, pyarrow_options: 'dict[str, Any] | None' = None, partition_by: 'str | Sequence[str] | None' = None, partition_chunk_size_bytes: 'int' = 4294967296, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, metadata: 'ParquetMetadata | None' = None, arrow_schema: 'ArrowSchemaExportable | None' = None, mkdir: 'bool' = False) -> 'None'
| Write to Apache Parquet file.
|
| Parameters
| ----------
| file
| File path or writable file-like object to which the result will be written.
| This should be a path to a directory if writing a partitioned dataset.
| compression : {'lz4', 'uncompressed', 'snappy', 'gzip', 'brotli', 'zstd'}
| Choose "zstd" for good compression performance.
| Choose "lz4" for fast compression/decompression.
| Choose "snappy" for more backwards compatibility guarantees
| when you deal with older parquet readers.
| compression_level
| The level of compression to use. Higher compression means smaller files on
| disk.
|
| - "gzip" : min-level: 0, max-level: 9, default: 6.
| - "brotli" : min-level: 0, max-level: 11, default: 1.
| - "zstd" : min-level: 1, max-level: 22, default: 3.
|
| statistics
| Write statistics to the parquet headers. This is the default behavior.
|
| Possible values:
|
| - `True`: enable default set of statistics (default). Some
| statistics may be disabled.
| - `False`: disable all statistics
| - "full": calculate and write all available statistics. Cannot be
| combined with `use_pyarrow`.
| - `{ "statistic-key": True / False, ... }`. Cannot be combined with
| `use_pyarrow`. Available keys:
|
| - "min": column minimum value (default: `True`)
| - "max": column maximum value (default: `True`)
| - "distinct_count": number of unique column values (default: `False`)
| - "null_count": number of null values in column (default: `True`)
| row_group_size
| Size of the row groups in number of rows. Defaults to 512^2 rows.
| data_page_size
| Size of the data page in bytes. Defaults to 1024^2 bytes.
| use_pyarrow
| Use PyArrow's C++ parquet implementation instead of Polars' native
| Rust implementation. This may be useful when specific PyArrow features
| are needed via ``pyarrow_options``. Some options are not supported when
| enabled (e.g. ``statistics="full"``, ``metadata``, ``mkdir``).
| pyarrow_options
| Arguments passed to `pyarrow.parquet.write_table`.
|
| If you pass `partition_cols` here, the dataset will be written
| using `pyarrow.parquet.write_to_dataset`.
| The `partition_cols` parameter leads to write the dataset to a directory.
| Similar to Spark's partitioned datasets. For native partitioned
| writes, consider using ``partition_by`` instead.
| partition_by
| Column(s) to partition by. A partitioned dataset will be written if this is
| specified. This parameter is considered unstable and is subject to change.
| partition_chunk_size_bytes
| Approximate size to split DataFrames within a single partition when
| writing. Note this is calculated using the size of the DataFrame in
| memory - the size of the output file may differ depending on the
| file format / compression.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| metadata
| A dictionary or callback to add key-values to the file-level Parquet
| metadata.
|
| .. warning::
| This functionality is considered **experimental**. It may be removed or
| changed at any point without it being considered a breaking change.
|
| arrow_schema
| Provide a custom arrow schema to write to the file. This allows
| setting custom schema and field-level metadata. Names and dtypes
| must match.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| Examples
| --------
| >>> import pathlib
| >>>
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> path: pathlib.Path = dirpath / "new_file.parquet"
| >>> df.write_parquet(path)
|
| We can write partitioned datasets. The following example will write
| the first row to ../watermark=1/*.parquet and the other rows to
| ../watermark=2/*.parquet.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3], "watermark": [1, 2, 2]})
| >>> path: pathlib.Path = dirpath / "partitioned_object"
| >>> df.write_parquet(
| ... path,
| ... partition_by=["watermark"],
| ... )
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| deserialize(source: 'str | bytes | Path | IOBase', *, format: 'SerializationFormat' = 'binary') -> 'DataFrame'
| Read a serialized DataFrame from a file.
|
| Parameters
| ----------
| source
| Path to a file or a file-like object (by file-like object, we refer to
| objects that have a `read()` method, such as a file handler (e.g.
| via builtin `open` function) or `BytesIO`).
| format
| The format with which the DataFrame was serialized. Options:
|
| - `"binary"`: Deserialize from binary format (bytes). This is the default.
| - `"json"`: Deserialize from JSON format (string).
|
| See Also
| --------
| DataFrame.serialize
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| >>> import io
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4.0, 5.0, 6.0]})
| >>> bytes = df.serialize()
| >>> pl.DataFrame.deserialize(io.BytesIO(bytes))
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 4.0 │
| │ 2 ┆ 5.0 │
| │ 3 ┆ 6.0 │
| └─────┴─────┘
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| dtypes
| Get the column data types.
|
| The data types can also be found in column headers when printing the DataFrame.
|
| Returns
| -------
| list of DataType
| A list containing the data type of each column in order.
|
| See Also
| --------
| schema
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.dtypes
| [Int64, Float64, String]
| >>> df
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 7.0 ┆ b │
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| flags
| Get flags that are set on the columns of this DataFrame.
|
| Returns
| -------
| dict
| Mapping from column names to column flags.
|
| height
| Get the number of rows.
|
| Returns
| -------
| int
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5]})
| >>> df.height
| 5
|
| plot
| Create a plot namespace.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.6.0
| In prior versions of Polars, HvPlot was the plotting backend. If you would
| like to restore the previous plotting functionality, all you need to do
| is add `import hvplot.polars` at the top of your script and replace
| `df.plot` with `df.hvplot`.
|
| Polars does not implement plotting logic itself, but instead defers to
| `Altair <https://altair-viz.github.io/>`_:
|
| - `df.plot.line(**kwargs)`
| is shorthand for
| `alt.Chart(df).mark_line(tooltip=True).encode(**kwargs).interactive()`
| - `df.plot.point(**kwargs)`
| is shorthand for
| `alt.Chart(df).mark_point(tooltip=True).encode(**kwargs).interactive()` (and
| `plot.scatter` is provided as an alias)
| - `df.plot.bar(**kwargs)`
| is shorthand for
| `alt.Chart(df).mark_bar(tooltip=True).encode(**kwargs).interactive()`
| - for any other attribute `attr`, `df.plot.attr(**kwargs)`
| is shorthand for
| `alt.Chart(df).mark_attr(tooltip=True).encode(**kwargs).interactive()`
|
| For configuration, we suggest reading
| `Chart Configuration <https://altair-viz.github.io/altair-tutorial/notebooks/08-Configuration.html>`_.
| For example, you can:
|
| - Change the width/height/title with
| ``.properties(width=500, height=350, title="My amazing plot")``.
| - Change the x-axis label rotation with ``.configure_axisX(labelAngle=30)``.
| - Change the opacity of the points in your scatter plot with
| ``.configure_point(opacity=.5)``.
|
| Examples
| --------
| Scatter plot:
|
| >>> df = pl.DataFrame(
| ... {
| ... "length": [1, 4, 6],
| ... "width": [4, 5, 6],
| ... "species": ["setosa", "setosa", "versicolor"],
| ... }
| ... )
| >>> df.plot.point(x="length", y="width", color="species") # doctest: +SKIP
|
| Set the x-axis title by using ``altair.X``:
|
| >>> import altair as alt
| >>> df.plot.point(
| ... x=alt.X("length", title="Length"), y="width", color="species"
| ... ) # doctest: +SKIP
|
| Line plot:
|
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "date": [date(2020, 1, 2), date(2020, 1, 3), date(2020, 1, 4)] * 2,
| ... "price": [1, 4, 6, 1, 5, 2],
| ... "stock": ["a", "a", "a", "b", "b", "b"],
| ... }
| ... )
| >>> df.plot.line(x="date", y="price", color="stock") # doctest: +SKIP
|
| Bar plot:
|
| >>> df = pl.DataFrame(
| ... {
| ... "day": ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"] * 2,
| ... "group": ["a"] * 7 + ["b"] * 7,
| ... "value": [1, 3, 2, 4, 5, 6, 1, 1, 3, 2, 4, 5, 1, 2],
| ... }
| ... )
| >>> df.plot.bar(
| ... x="day", y="value", color="day", column="group"
| ... ) # doctest: +SKIP
|
| Or, to make a stacked version of the plot above:
|
| >>> df.plot.bar(x="day", y="value", color="group") # doctest: +SKIP
|
| schema
| Get an ordered mapping of column names to their data type.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.schema
| Schema({'foo': Int64, 'bar': Float64, 'ham': String})
|
| shape
| Get the shape of the DataFrame.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5]})
| >>> df.shape
| (5, 1)
|
| style
| Create a Great Table for styling.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Polars does not implement styling logic itself, but instead defers to
| the Great Tables package. Please see the `Great Tables reference <https://posit-dev.github.io/great-tables/reference/>`_
| for more information and documentation.
|
| Examples
| --------
| Import some styling helpers, and create example data:
|
| >>> import polars.selectors as cs
| >>> from great_tables import loc, style
| >>> df = pl.DataFrame(
| ... {
| ... "site_id": [0, 1, 2],
| ... "measure_a": [5, 4, 6],
| ... "measure_b": [7, 3, 3],
| ... }
| ... )
|
| Emphasize the site_id as row names:
|
| >>> df.style.tab_stub(rowname_col="site_id") # doctest: +SKIP
|
| Fill the background for the highest measure_a value row:
|
| >>> df.style.tab_style(
| ... style.fill("yellow"),
| ... loc.body(rows=pl.col("measure_a") == pl.col("measure_a").max()),
| ... ) # doctest: +SKIP
|
| Put a spanner (high-level label) over measure columns:
|
| >>> df.style.tab_spanner(
| ... "Measures", cs.starts_with("measure")
| ... ) # doctest: +SKIP
|
| Format measure_b values to two decimal places:
|
| >>> df.style.fmt_number("measure_b", decimals=2) # doctest: +SKIP
|
| width
| Get the number of columns.
|
| Returns
| -------
| int
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4, 5, 6],
| ... }
| ... )
| >>> df.width
| 2
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| columns
| Get or set column names.
|
| Returns
| -------
| list of str
| A list containing the name of each column in order.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> df.columns
| ['foo', 'bar', 'ham']
|
| Set column names:
|
| >>> df.columns = ["apple", "banana", "orange"]
| >>> df
| shape: (3, 3)
| ┌───────┬────────┬────────┐
| │ apple ┆ banana ┆ orange │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═══════╪════════╪════════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └───────┴────────┴────────┘
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set[str]]', '_df': 'PyDataF...
|
| __hash__ = None
DATA
__all__ = ['DataFrame']
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\dataframe\__init__.py
polars.datatype_expr
Python Library Documentation: package polars.datatype_expr in polars
NAME
polars.datatype_expr
PACKAGE CONTENTS
array
datatype_expr
list
struct
CLASSES
builtins.object
polars.datatype_expr.datatype_expr.DataTypeExpr
class DataTypeExpr(builtins.object)
| A lazily instantiated :class:`DataType` that can be used in an :class:`Expr`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| This expression is made to represent a :class:`DataType` that can be used to
| reference a datatype in a lazy context.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"a": [1, 2, 3]})
| >>> lf.with_columns(
| ... pl.col.a.map_batches(lambda x: x * 2, return_dtype=pl.dtype_of("a"))
| ... ).collect()
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 4 │
| │ 6 │
| └─────┘
|
| Methods defined here:
|
| __eq__(self, value: 'PolarsDataType | DataTypeExpr') -> 'pl.Expr'
| Return self==value.
|
| __ne__(self, value: 'PolarsDataType | DataTypeExpr') -> 'pl.Expr'
| Return self!=value.
|
| collect_dtype(self, context: 'SchemaDict | pl.Schema | pl.DataFrame | pl.LazyFrame') -> 'DataType'
| Materialize the :class:`DataTypeExpr` in a specific context.
|
| This is a useful function when debugging datatype expressions.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3],
| ... }
| ... )
| >>> pl.dtype_of("a").collect_dtype(lf)
| Int64
| >>> pl.dtype_of("a").collect_dtype({"a": pl.String})
| String
|
| default_value(self, n: 'int' = 1, *, numeric_to_one: 'bool' = False, num_list_values: 'int' = 0) -> 'pl.Expr'
| Get a default value of a specific type.
|
| - Integers and floats are their zero value as default, unless otherwise
| specified
| - Temporals are a physical zero as default
| - `pl.Decimal` is zero as default
| - `pl.String` and `pl.Binary` are an empty string
| - `pl.List` is an empty list, unless otherwise specified
| - `pl.Array` is the inner default value repeated over the shape
| - `pl.Struct` is the inner default value for all fields
| - `pl.Enum` is the first category if it exists
| - `pl.Null`, `pl.Object` and `pl.Categorical` are `null`.
|
| Parameters
| ----------
| n
| Number of types you want the value
| numeric_to_one
| Use `1` instead of `0` as the default value for numeric types
| num_list_values
| The amount of values a list contains
|
| Examples
| --------
| >>> uint32 = pl.UInt32.to_dtype_expr()
| >>> pl.select(default=uint32.default_value())
| shape: (1, 1)
| ┌─────────┐
| │ default │
| │ --- │
| │ u32 │
| ╞═════════╡
| │ 0 │
| └─────────┘
|
| display(self) -> 'pl.Expr'
| Get a formatted version of the output DataType.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["X", "Y", "Z"],
| ... "c": [1.3, 3.7, 4.2],
| ... }
| ... )
| >>> df.select(
| ... a=pl.dtype_of("a").display(),
| ... b=pl.dtype_of("b").display(),
| ... c=pl.dtype_of("c").display(),
| ... ).transpose(include_header=True, column_names=["dtype"])
| shape: (3, 2)
| ┌────────┬───────┐
| │ column ┆ dtype │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞════════╪═══════╡
| │ a ┆ i64 │
| │ b ┆ str │
| │ c ┆ f64 │
| └────────┴───────┘
|
| inner_dtype(self) -> 'DataTypeExpr'
| Get the inner DataType of a List or Array.
|
| matches(self, selector: 'pl.Selector') -> 'pl.Expr'
| Get whether the output DataType is matches a certain selector.
|
| Examples
| --------
| >>> import polars.selectors as cs
| >>> pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... }
| ... ).select(
| ... a_is_string=pl.dtype_of("a").matches(cs.string()),
| ... a_is_integer=pl.dtype_of("a").matches(cs.integer()),
| ... )
| shape: (1, 2)
| ┌─────────────┬──────────────┐
| │ a_is_string ┆ a_is_integer │
| │ --- ┆ --- │
| │ bool ┆ bool │
| ╞═════════════╪══════════════╡
| │ false ┆ true │
| └─────────────┴──────────────┘
|
| to_signed_integer(self) -> 'pl.DataTypeExpr'
| Get the signed integer version of the same bitsize.
|
| Examples
| --------
| >>> uint32 = pl.UInt32.to_dtype_expr()
| >>> uint32.to_signed_integer().collect_dtype({})
| Int32
|
| to_unsigned_integer(self) -> 'pl.DataTypeExpr'
| Get the unsigned integer version of the same bitsize.
|
| Examples
| --------
| >>> int32 = pl.Int32.to_dtype_expr()
| >>> int32.to_unsigned_integer().collect_dtype({})
| UInt32
|
| wrap_in_array(self, *, width: 'int') -> 'DataTypeExpr'
| Get the DataType wrapped in an array.
|
| Examples
| --------
| >>> pl.Int32.to_dtype_expr().wrap_in_array(width=5).collect_dtype({})
| Array(Int32, shape=(5,))
|
| wrap_in_list(self) -> 'DataTypeExpr'
| Get the DataType wrapped in a list.
|
| Examples
| --------
| >>> pl.Int32.to_dtype_expr().wrap_in_list().collect_dtype({})
| List(Int32)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| arr
| Create an object namespace of all array related methods.
|
| list
| Create an object namespace of all list related methods.
|
| struct
| Create an object namespace of all struct related methods.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set[str]]', '_pydatatype_ex...
|
| __hash__ = None
DATA
__all__ = ['DataTypeExpr']
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\datatype_expr\__init__.py
polars.datatypes
Python Library Documentation: package polars.datatypes in polars
NAME
polars.datatypes
PACKAGE CONTENTS
_parse
_utils
classes
constants
constructor
convert
extension
group
CLASSES
builtins.object
polars.datatypes.classes.Categories
polars.datatypes.classes.DataType
polars.datatypes.classes.BaseExtension
polars.datatypes.classes.Extension
polars.datatypes.classes.Binary
polars.datatypes.classes.Boolean
polars.datatypes.classes.Categorical
polars.datatypes.classes.Enum
polars.datatypes.classes.Null
polars.datatypes.classes.String
polars.datatypes.classes.TemporalType
polars.datatypes.classes.Date
polars.datatypes.classes.Datetime
polars.datatypes.classes.Duration
polars.datatypes.classes.Time
polars.datatypes.classes.Unknown
polars.datatypes.classes.Field
builtins.type(builtins.object)
polars.datatypes.classes.DataTypeClass
polars.datatypes.classes.NestedType(polars.datatypes.classes.DataType)
polars.datatypes.classes.Array
polars.datatypes.classes.List
polars.datatypes.classes.Struct
polars.datatypes.classes.NumericType(polars.datatypes.classes.DataType)
polars.datatypes.classes.Decimal
polars.datatypes.classes.FloatType
polars.datatypes.classes.Float16
polars.datatypes.classes.Float32
polars.datatypes.classes.Float64
polars.datatypes.classes.IntegerType
polars.datatypes.classes.ObjectType(polars.datatypes.classes.DataType)
polars.datatypes.classes.Object
polars.datatypes.classes.SignedIntegerType(polars.datatypes.classes.IntegerType)
polars.datatypes.classes.Int128
polars.datatypes.classes.Int16
polars.datatypes.classes.Int32
polars.datatypes.classes.Int64
polars.datatypes.classes.Int8
polars.datatypes.classes.UnsignedIntegerType(polars.datatypes.classes.IntegerType)
polars.datatypes.classes.UInt128
polars.datatypes.classes.UInt16
polars.datatypes.classes.UInt32
polars.datatypes.classes.UInt64
polars.datatypes.classes.UInt8
class Array(NestedType)
| Array(inner: 'PolarsDataType | PythonDataType', shape: 'int | tuple[int, ...] | None' = None, *, width: 'int | None' = None) -> 'None'
|
| Fixed length list type.
|
| Parameters
| ----------
| inner
| The `DataType` of the values within each array.
| shape
| The shape of the arrays.
| width
| The length of the arrays.
|
| .. deprecated:: 0.20.31
| The `width` parameter for `Array` is deprecated. Use `shape` instead.
|
| Examples
| --------
| >>> s = pl.Series("a", [[1, 2], [4, 3]], dtype=pl.Array(pl.Int64, 2))
| >>> s
| shape: (2,)
| Series: 'a' [array[i64, 2]]
| [
| [1, 2]
| [4, 3]
| ]
|
| Method resolution order:
| Array
| NestedType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, inner: 'PolarsDataType | PythonDataType', shape: 'int | tuple[int, ...] | None' = None, *, width: 'int | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| width
| The size of the Array.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'inner': 'PolarsDataType', 'shape': 'tuple[int, ......
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class BaseExtension(DataType)
| BaseExtension(name: 'str', storage: 'PolarsDataType', metadata: 'str | None' = None) -> 'None'
|
| Base class for extension data types.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| See Also
| --------
| Extension
| polars.register_extension_type
|
| Method resolution order:
| BaseExtension
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __getstate__(self) -> 'tuple[str, PolarsDataType, str | None]'
| Helper for pickle.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, name: 'str', storage: 'PolarsDataType', metadata: 'str | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, state: 'tuple[str, PolarsDataType, str | None]') -> 'None'
|
| ext_metadata(self) -> 'str | None'
| Returns the metadata for this extension type.
|
| ext_name(self) -> 'str'
| Returns the name of this extension type.
|
| ext_storage(self) -> 'PolarsDataType'
| Returns the storage type for this extension type.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| ext_from_params(name: 'str', storage: 'PolarsDataType', metadata: 'str | None') -> 'Any'
| Creates an Extension type instance from its parameters.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Binary(DataType)
| Binary type.
|
| Method resolution order:
| Binary
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Boolean(DataType)
| Boolean type.
|
| Method resolution order:
| Boolean
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Categorical(DataType)
| Categorical(categories: 'Categories | str | None' = None, *, ordering: 'CategoricalOrdering | None' = None) -> 'None'
|
| A categorical encoding of a set of strings.
|
| Parameters
| ----------
| categories
| The categories used for this type; must be a :py:class:`Categories`
| instance, or a string which is interpreted as the name of a
| :py:class:`Categories`. If not provided, the global categories
| (`pl.Categories()`) are used.
|
| For legacy reasons if the string is either `"physical"` or `"lexical"`,
| it is ignored and a warning is issued. If you wish to use a `Categories`
| named `"physical"` or `"lexical"`, please pass it using
| :py:class:`Categories` explicitly.
|
| ordering : {'lexical', 'physical'}
| This used to specify how this type was ordered, but now does nothing.
|
| .. deprecated:: 1.32.0
| Parameter is now ignored. Always behaves as if `'lexical'` was passed.
|
| See Also
| --------
| Categories
|
| Method resolution order:
| Categorical
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, categories: 'Categories | str | None' = None, *, ordering: 'CategoricalOrdering | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'categories': 'Categories', 'ordering': 'Categorica...
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Categories(builtins.object)
| Categories(name: 'str | None' = None, namespace: 'str' = '', physical: 'PolarsDataType' = UInt32) -> 'None'
|
| A named collection of categories for :py:class:`Categorical`.
|
| Two categories are considered equal (and will use the same physical mapping of
| categories to strings) if they have the same name, namespace and physical backing
| type, even if they are created in separate calls to `Categories`.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| name
| The name of this `Categories`. If set to `None` or an empty string, this
| refers to the global categories.
|
| namespace
| An optional namespace for this `Categories`. Defaults to the empty string.
| If the name is empty or `None` indicating the global categories, the
| namespace must also be empty.
|
| physical : {UInt8, UInt16, UInt32}
| The physical type used to represent the categories. Defaults to
| :py:class:`UInt32`.
|
| See Also
| --------
| Categorical
|
| Examples
| --------
| A `Categories` instance can be indexed using either string or integer keys:
|
| >>> fruit = pl.Categories("fruit")
| >>> s = pl.Series(["apple", "banana", "orange"], dtype=pl.Categorical(fruit))
| >>> fruit[0]
| 'apple'
| >>> fruit["apple"]
| 0
|
| All `Categories` objects with the same name, namespace and physical type
| share the same mapping, even if they're created separately:
|
| >>> fruit2 = pl.Categories("fruit")
| >>> fruit2["banana"]
| 1
|
| To get a list of all categories, you can iterate over the `Categories` instance:
|
| >>> list(fruit)
| ['apple', 'banana', 'orange']
|
| .. note::
| Because the categories are backed by a concurrent data structure, physical
| category values may be reserved before they are assigned a string lexical
| value if concurrent queries are running. As a result, the resulting `Series`
| may contain `None` values.
|
| The `Categories` instance is only a weak reference to the actual
| mapping stored in Polars. If no actual data exists using this mapping (like
| a `Series` or `DataFrame`), the mapping is cleaned up by Polars:
|
| >>> del s
| >>> print(fruit["apple"])
| None
|
| If you wish to keep a persistent mapping, simply keep alive some object which
| uses the mapping, e.g. `keepalive = pl.Series([], dtype=pl.Categorical(fruit))`.
|
| Methods defined here:
|
| __contains__(self, item: 'str | int') -> 'bool'
|
| __eq__(self, other: 'object') -> 'bool'
| Return self==value.
|
| __getitem__(self, key: 'str | int | None') -> 'str | int | None'
|
| __getstate__(self) -> 'tuple[str, str, PolarsDataType]'
| Helper for pickle.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, name: 'str | None' = None, namespace: 'str' = '', physical: 'PolarsDataType' = UInt32) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> 'Iterator[str | None]'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, state: 'tuple[str, str, PolarsDataType]') -> 'None'
|
| is_global(self) -> 'bool'
| Returns whether this refers to the global categories.
|
| name(self) -> 'str'
| The name of this `Categories`.
|
| namespace(self) -> 'str'
| The namespace of this `Categories`.
|
| physical(self) -> 'PolarsDataType'
| The physical type used to represent the categories.
|
| to_dict(self) -> 'dict[str, int]'
| Return a dictionary mapping category strings to their physical category values.
|
| Examples
| --------
| >>> fruit = pl.Categories("fruit")
| >>> s = pl.Series(["apple", "banana", "orange"], dtype=pl.Categorical(fruit))
| >>> fruit.to_dict()
| {'apple': 0, 'banana': 1, 'orange': 2}
|
| to_series(self) -> 'Series'
| Return a :class:`Series` containing all categories in this `Categories`.
|
| The categories are ordered by their physical category value.
|
| .. note::
| Because the categories are backed by a concurrent data structure, physical
| category values may be reserved before they are assigned a string lexical
| value if concurrent queries are running. As a result, the resulting `Series`
| may contain `None` values.
|
| Examples
| --------
| >>> fruit = pl.Categories("fruit")
| >>> s = pl.Series(["apple", "banana", "orange"], dtype=pl.Categorical(fruit))
| >>> fruit.to_series()
| shape: (3,)
| Series: 'fruit' [str]
| [
| "apple"
| "banana"
| "orange"
| ]
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| random(namespace: 'str' = '', physical: 'PolarsDataType' = UInt32) -> 'Categories'
| Creates a new `Categories` with a random name.
|
| Parameters
| ----------
| namespace
| An optional namespace for this `Categories`. Defaults to the empty string.
|
| physical : {UInt8, UInt16, UInt32}
| The physical type used to represent the categories. Defaults
| to :py:class:`UInt32`.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_categories': 'PyCategories'}
class DataType(builtins.object)
| Base class for all Polars data types.
|
| Methods defined here:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
class DataTypeClass(builtins.type)
| Metaclass for nicely printing DataType classes.
|
| Method resolution order:
| DataTypeClass
| builtins.type
| builtins.object
|
| Methods defined here:
|
| __repr__(cls) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| base_type() -> 'DataTypeClass'
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
|
| is_(other: 'PolarsDataType') -> 'bool'
|
| is_decimal() -> 'bool'
|
| is_extension() -> 'bool'
|
| is_float() -> 'bool'
|
| is_integer() -> 'bool'
|
| is_nested() -> 'bool'
|
| is_numeric() -> 'bool'
|
| is_object() -> 'bool'
|
| is_signed_integer() -> 'bool'
|
| is_temporal() -> 'bool'
|
| is_unsigned_integer() -> 'bool'
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
|
| to_python() -> 'PythonDataType'
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.type:
|
| __call__(self, /, *args, **kwargs)
| Call self as a function.
|
| __delattr__(self, name, /)
| Implement delattr(self, name).
|
| __dir__(self, /)
| Specialized __dir__ implementation for types.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __instancecheck__(self, instance, /)
| Check if an object is an instance.
|
| __or__(self, value, /)
| Return self|value.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setattr__(self, name, value, /)
| Implement setattr(self, name, value).
|
| __sizeof__(self, /)
| Return memory consumption of the type object.
|
| __subclasscheck__(self, subclass, /)
| Check if a class is a subclass.
|
| __subclasses__(self, /)
| Return a list of immediate subclasses.
|
| mro(self, /)
| Return a type's method resolution order.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.type:
|
| __prepare__(...)
| __prepare__() -> dict
| used to create the namespace for the class statement
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.type:
|
| __new__(*args, **kwargs) class method of builtins.type
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.type:
|
| __abstractmethods__
|
| __dict__
|
| __text_signature__
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.type:
|
| __base__ = <class 'type'>
| type(object) -> the object's type
| type(name, bases, dict, **kwds) -> a new type
|
|
| __bases__ = (<class 'type'>,)
|
| __basicsize__ = 920
|
| __dictoffset__ = 264
|
| __flags__ = 2156420608
|
| __itemsize__ = 40
|
| __mro__ = (<class 'polars.datatypes.classes.DataTypeClass'>, <class 't...
|
| __type_params__ = ()
|
| __weakrefoffset__ = 368
class Date(TemporalType)
| Data type representing a calendar date.
|
| Notes
| -----
| The underlying representation of this type is a 32-bit signed integer.
| The integer indicates the number of days since the Unix epoch (1970-01-01).
| The number can be negative to indicate dates before the epoch.
|
| Method resolution order:
| Date
| TemporalType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Datetime(TemporalType)
| Datetime(time_unit: 'TimeUnit' = 'us', time_zone: 'str | tzinfo | None' = None) -> 'None'
|
| Data type representing a calendar date and time of day.
|
| Parameters
| ----------
| time_unit : {'us', 'ns', 'ms'}
| Unit of time. Defaults to `'us'` (microseconds).
| time_zone
| Time zone string, as defined in zoneinfo (to see valid strings run
| `import zoneinfo; zoneinfo.available_timezones()` for a full list).
| When used to match dtypes, can set this to "*" to check for Datetime
| columns that have any (non-null) timezone.
|
| Notes
| -----
| The underlying representation of this type is a 64-bit signed integer.
| The integer indicates the number of time units since the Unix epoch
| (1970-01-01 00:00:00). The number can be negative to indicate datetimes before the
| epoch.
|
| Method resolution order:
| Datetime
| TemporalType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, time_unit: 'TimeUnit' = 'us', time_zone: 'str | tzinfo | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'time_unit': 'TimeUnit', 'time_zone': 'str | None'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Decimal(NumericType)
| Decimal(precision: 'int | None' = None, scale: 'int' = 0) -> 'None'
|
| Decimal 128-bit type with an optional precision and non-negative scale.
|
| Parameters
| ----------
| precision
| Maximum number of digits in each number.
| If set to `None` (default), the precision is set to 38 (the maximum
| supported by Polars).
| scale
| Number of digits to the right of the decimal point in each number.
|
| Method resolution order:
| Decimal
| NumericType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, precision: 'int | None' = None, scale: 'int' = 0) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'precision': 'int', 'scale': 'int'}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Duration(TemporalType)
| Duration(time_unit: 'TimeUnit' = 'us') -> 'None'
|
| Data type representing a time duration.
|
| Parameters
| ----------
| time_unit : {'us', 'ns', 'ms'}
| Unit of time. Defaults to `'us'` (microseconds).
|
| Notes
| -----
| The underlying representation of this type is a 64-bit signed integer.
| The integer indicates an amount of time units and can be negative to indicate
| negative time offsets.
|
| Method resolution order:
| Duration
| TemporalType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, time_unit: 'TimeUnit' = 'us') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'time_unit': 'TimeUnit'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Enum(DataType)
| Enum(categories: 'Series | Iterable[str] | type[enum.Enum]') -> 'None'
|
| A fixed categorical encoding of a unique set of strings.
|
| Parameters
| ----------
| categories
| The categories in the dataset; must be a unique set of strings, or an
| existing Python string-valued enum.
|
| Examples
| --------
| Explicitly define enumeration categories:
|
| >>> pl.Enum(["north", "south", "east", "west"])
| Enum(categories=['north', 'south', 'east', 'west'])
|
| Initialise from an existing Python enumeration:
|
| >>> from http import HTTPMethod
| >>> pl.Enum(HTTPMethod)
| Enum(categories=['CONNECT', 'DELETE', 'GET', 'HEAD', 'OPTIONS', 'PATCH', 'POST', 'PUT', 'TRACE'])
|
| Method resolution order:
| Enum
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, categories: 'Series | Iterable[str] | type[enum.Enum]') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __or__ = union(self, other: 'Enum') -> 'Enum'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| union(self, other: 'Enum') -> 'Enum'
| Union of two Enums.
|
| .. deprecated:: 1.38
| `Enum.union()` is deprecated and will be removed in version 2.0.
| Enums are ordered sets and union cannot preserve both orderings.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'categories': 'Series'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Extension(BaseExtension)
| Extension(name: 'str', storage: 'PolarsDataType', metadata: 'str | None' = None) -> 'None'
|
| Generic extension data type.
|
| When `UNKNOWN_EXTENSION_TYPE_BEHAVIOR` is set to `"load_as_extension"`, any
| non-registered extension type will be loaded as this type.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| See Also
| --------
| BaseExtension
| polars.register_extension_type
|
| Method resolution order:
| Extension
| BaseExtension
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from BaseExtension:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __getstate__(self) -> 'tuple[str, PolarsDataType, str | None]'
| Helper for pickle.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, name: 'str', storage: 'PolarsDataType', metadata: 'str | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, state: 'tuple[str, PolarsDataType, str | None]') -> 'None'
|
| ext_metadata(self) -> 'str | None'
| Returns the metadata for this extension type.
|
| ext_name(self) -> 'str'
| Returns the name of this extension type.
|
| ext_storage(self) -> 'PolarsDataType'
| Returns the storage type for this extension type.
|
| ----------------------------------------------------------------------
| Class methods inherited from BaseExtension:
|
| ext_from_params(name: 'str', storage: 'PolarsDataType', metadata: 'str | None') -> 'Any'
| Creates an Extension type instance from its parameters.
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Field(builtins.object)
| Field(name: 'str', dtype: 'PolarsDataType') -> 'None'
|
| Definition of a single field within a `Struct` DataType.
|
| Parameters
| ----------
| name
| The name of the field within its parent `Struct`.
| dtype
| The `DataType` of the field's values.
|
| Methods defined here:
|
| __eq__(self, other: 'Field') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, name: 'str', dtype: 'PolarsDataType') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'dtype': 'PolarsDataType', 'name': 'str'}
class Float16(FloatType)
| 16-bit floating point type.
|
| .. warning::
| Regular computing platforms do not natively support `Float16` operations,
| and compute operations on `Float16` will be significantly slower as a result
| than operation on :class:`Float32` or :class:`Float64`.
| As such, it is recommended to cast to `Float32` before doing any compute
| operations, and cast back to `Float16` afterward if needed.
|
| Method resolution order:
| Float16
| FloatType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Float32(FloatType)
| 32-bit floating point type.
|
| Method resolution order:
| Float32
| FloatType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Float64(FloatType)
| 64-bit floating point type.
|
| Method resolution order:
| Float64
| FloatType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class FloatType(NumericType)
| Base class for float data types.
|
| Method resolution order:
| FloatType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Int128(SignedIntegerType)
| 128-bit signed integer type.
|
| .. warning::
| This functionality is considered **unstable**.
| It is a work-in-progress feature and may not always work as expected.
| It may be changed at any point without it being considered a breaking change.
|
| Method resolution order:
| Int128
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Int16(SignedIntegerType)
| 16-bit signed integer type.
|
| Method resolution order:
| Int16
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Int32(SignedIntegerType)
| 32-bit signed integer type.
|
| Method resolution order:
| Int32
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Int64(SignedIntegerType)
| 64-bit signed integer type.
|
| Method resolution order:
| Int64
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Int8(SignedIntegerType)
| 8-bit signed integer type.
|
| Method resolution order:
| Int8
| SignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class IntegerType(NumericType)
| Base class for integer data types.
|
| Method resolution order:
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class List(NestedType)
| List(inner: 'PolarsDataType | PythonDataType') -> 'None'
|
| Variable length list type.
|
| Parameters
| ----------
| inner
| The `DataType` of the values within each list.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "integer_lists": [[1, 2], [3, 4]],
| ... "float_lists": [[1.0, 2.0], [3.0, 4.0]],
| ... }
| ... )
| >>> df
| shape: (2, 2)
| ┌───────────────┬─────────────┐
| │ integer_lists ┆ float_lists │
| │ --- ┆ --- │
| │ list[i64] ┆ list[f64] │
| ╞═══════════════╪═════════════╡
| │ [1, 2] ┆ [1.0, 2.0] │
| │ [3, 4] ┆ [3.0, 4.0] │
| └───────────────┴─────────────┘
|
| Method resolution order:
| List
| NestedType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, inner: 'PolarsDataType | PythonDataType') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'inner': 'PolarsDataType'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Null(DataType)
| Data type representing null values.
|
| Method resolution order:
| Null
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Object(ObjectType)
| Data type for wrapping arbitrary Python objects.
|
| Method resolution order:
| Object
| ObjectType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class String(DataType)
| UTF-8 encoded string type.
|
| Method resolution order:
| String
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Struct(NestedType)
| Struct(fields: 'Sequence[Field] | SchemaDict') -> 'None'
|
| Struct composite type.
|
| Parameters
| ----------
| fields
| The fields that make up the struct. Can be either a sequence of Field
| objects or a mapping of column names to data types.
|
| Examples
| --------
| Initialize using a dictionary:
|
| >>> dtype = pl.Struct({"a": pl.Int8, "b": pl.List(pl.String)})
| >>> dtype
| Struct({'a': Int8, 'b': List(String)})
|
| Initialize using a list of Field objects:
|
| >>> dtype = pl.Struct([pl.Field("a", pl.Int8), pl.Field("b", pl.List(pl.String))])
| >>> dtype
| Struct({'a': Int8, 'b': List(String)})
|
| When initializing a Series, Polars can infer a struct data type from the data.
|
| >>> s = pl.Series([{"a": 1, "b": ["x", "y"]}, {"a": 2, "b": ["z"]}])
| >>> s
| shape: (2,)
| Series: '' [struct[2]]
| [
| {1,["x", "y"]}
| {2,["z"]}
| ]
| >>> s.dtype
| Struct({'a': Int64, 'b': List(String)})
|
| Method resolution order:
| Struct
| NestedType
| DataType
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other: 'PolarsDataType') -> 'bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __init__(self, fields: 'Sequence[Field] | SchemaDict') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> 'Iterator[tuple[str, PolarsDataType]]'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __reversed__(self) -> 'Iterator[tuple[str, PolarsDataType]]'
|
| to_schema(self) -> 'OrderedDict[str, PolarsDataType]'
| Return Struct dtype as a schema dict.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'fields': 'list[Field]'}
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class TemporalType(DataType)
| Base class for temporal data types.
|
| Method resolution order:
| TemporalType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Time(TemporalType)
| Data type representing the time of day.
|
| Notes
| -----
| The underlying representation of this type is a 64-bit signed integer.
| The integer indicates the number of nanoseconds since midnight.
|
| Method resolution order:
| Time
| TemporalType
| DataType
| builtins.object
|
| Class methods defined here:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Time.max() == 86_399_999_999_999)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Time.min() == 0)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class UInt128(UnsignedIntegerType)
| 128-bit unsigned integer type.
|
| .. warning::
| This functionality is considered **unstable**.
| It is a work-in-progress feature and may not always work as expected.
| It may be changed at any point without it being considered a breaking change.
|
| Method resolution order:
| UInt128
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class UInt16(UnsignedIntegerType)
| 16-bit unsigned integer type.
|
| Method resolution order:
| UInt16
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class UInt32(UnsignedIntegerType)
| 32-bit unsigned integer type.
|
| Method resolution order:
| UInt32
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class UInt64(UnsignedIntegerType)
| 64-bit unsigned integer type.
|
| Method resolution order:
| UInt64
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class UInt8(UnsignedIntegerType)
| 8-bit unsigned integer type.
|
| Method resolution order:
| UInt8
| UnsignedIntegerType
| IntegerType
| NumericType
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Class methods inherited from NumericType:
|
| max() -> 'pl.Expr'
| Return a literal expression representing the maximum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.max() == 127)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| min() -> 'pl.Expr'
| Return a literal expression representing the minimum value of this data type.
|
| Examples
| --------
| >>> pl.select(pl.Int8.min() == -128)
| shape: (1, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ bool │
| ╞═════════╡
| │ true │
| └─────────┘
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Unknown(DataType)
| Type representing DataType values that could not be determined statically.
|
| Method resolution order:
| Unknown
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
Utf8 = class String(DataType)
| UTF-8 encoded string type.
|
| Method resolution order:
| String
| DataType
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from DataType:
|
| __eq__(self, other: 'pl.DataTypeExpr | PolarsDataType') -> 'pl.Expr | bool'
| Return self==value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods inherited from DataType:
|
| base_type() -> 'type[Self]'
| Return this DataType's fundamental/root type class.
|
| Examples
| --------
| >>> pl.Datetime("ns").base_type()
| Datetime
| >>> pl.List(pl.Int32).base_type()
| List
| >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type()
| Struct
|
| from_python(py_type: 'PythonDataType') -> 'PolarsDataType'
| Return the Polars data type corresponding to a given Python type.
|
| Notes
| -----
| Not every Python type has a corresponding Polars data type; in general
| you should declare Polars data types explicitly to exactly specify
| the desired type and its properties (such as scale/unit).
|
| Examples
| --------
| >>> pl.DataType.from_python(int)
| Int64
| >>> pl.DataType.from_python(float)
| Float64
| >>> from datetime import tzinfo
| >>> pl.DataType.from_python(tzinfo) # doctest: +SKIP
| TypeError: cannot parse input <class 'datetime.tzinfo'> into Polars data type
|
| is_decimal() -> 'bool'
| Check whether the data type is a decimal type.
|
| is_extension() -> 'bool'
| Check whether the data type is an extension type.
|
| is_float() -> 'bool'
| Check whether the data type is a floating point type.
|
| is_integer() -> 'bool'
| Check whether the data type is an integer type.
|
| is_nested() -> 'bool'
| Check whether the data type is a nested type.
|
| is_numeric() -> 'bool'
| Check whether the data type is a numeric type.
|
| is_object() -> 'bool'
| Check whether the data type is an object type.
|
| is_signed_integer() -> 'bool'
| Check whether the data type is a signed integer type.
|
| is_temporal() -> 'bool'
| Check whether the data type is a temporal type.
|
| is_unsigned_integer() -> 'bool'
| Check whether the data type is an unsigned integer type.
|
| ----------------------------------------------------------------------
| Static methods inherited from DataType:
|
| is_(other: 'PolarsDataType') -> 'bool'
| Check if this DataType is the same as another DataType.
|
| This is a stricter check than `self == other`, as it enforces an exact
| match of all dtype attributes for nested and/or uninitialised dtypes.
|
| Parameters
| ----------
| other
| the other Polars dtype to compare with.
|
| Examples
| --------
| >>> pl.List == pl.List(pl.Int32)
| True
| >>> pl.List.is_(pl.List(pl.Int32))
| False
|
| to_dtype_expr() -> 'pl.DataTypeExpr'
| Return a :class:`DataTypeExpr` with a static :class:`DataType`.
|
| Examples
| --------
| >>> pl.Int16().to_dtype_expr().collect_dtype({})
| Int16
|
| to_python() -> 'PythonDataType'
| Return the Python type corresponding to this Polars data type.
|
| Examples
| --------
| >>> pl.Int16().to_python()
| <class 'int'>
| >>> pl.Float32().to_python()
| <class 'float'>
| >>> pl.Array(pl.Date(), 10).to_python()
| <class 'list'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DataType:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
dtype_to_ffiname(dtype: 'PolarsDataType') -> 'str'
Return FFI function name associated with the given Polars dtype.
dtype_to_py_type(dtype: 'PolarsDataType') -> 'PythonDataType'
Convert a Polars dtype to a Python dtype.
is_polars_dtype(dtype: 'Any', *, include_unknown: 'bool' = False, require_instantiated: 'bool' = False) -> 'TypeGuard[PolarsDataType]'
Indicate whether the given input is a Polars dtype, or dtype specialization.
maybe_cast(el: 'Any', dtype: 'PolarsDataType') -> 'Any'
Try casting a value to a value that is valid for the given Polars dtype.
numpy_char_code_to_dtype(dtype_char: 'str') -> 'PolarsDataType'
Convert a numpy character dtype to a Polars dtype.
numpy_type_to_constructor(values: 'np.ndarray[Any, Any]', dtype: 'type[np.dtype[Any]]') -> 'Callable[..., PySeries]'
Get the right PySeries constructor for the given Polars dtype.
numpy_values_and_dtype(values: 'np.ndarray[Any, Any]') -> 'tuple[np.ndarray[Any, Any], type]'
Return numpy values and their associated dtype, adjusting if required.
parse_into_datatype_expr(input: 'Any') -> 'pl.DataTypeExpr'
Parse an input into a DataTypeExpr.
parse_into_dtype(input: 'Any') -> 'PolarsDataType'
Parse an input into a Polars data type.
Raises
------
TypeError
If the input cannot be parsed into a Polars data type.
polars_type_to_constructor(dtype: 'PolarsDataType') -> 'Callable[[str, Sequence[Any], bool], PySeries]'
Get the right PySeries constructor for the given Polars dtype.
py_type_to_arrow_type(dtype: 'PythonDataType') -> 'pa.DataType'
Convert a Python dtype to an Arrow dtype.
py_type_to_constructor(py_type: 'type[Any]') -> 'Callable[..., PySeries]'
Get the right PySeries constructor for the given Python dtype.
supported_numpy_char_code(dtype_char: 'str') -> 'bool'
Check if the input can be mapped to a Polars dtype.
try_parse_into_dtype(input: 'Any') -> 'PolarsDataType | None'
Try parsing an input into a Polars data type, returning None on failure.
unpack_dtypes(*dtypes: 'PolarsDataType | None', include_compound: 'bool' = False) -> 'set[PolarsDataType]'
Return a set of unique dtypes found in one or more (potentially compound) dtypes.
Parameters
----------
*dtypes
One or more Polars dtypes.
include_compound
* if True, any parent/compound dtypes (List, Struct) are included in the result.
* if False, only the child/scalar dtypes are returned from these types.
Examples
--------
>>> from polars.datatypes import unpack_dtypes
>>> list_dtype = [pl.List(pl.Float64)]
>>> struct_dtype = pl.Struct(
... [
... pl.Field("a", pl.Int64),
... pl.Field("b", pl.String),
... pl.Field("c", pl.List(pl.Float64)),
... ]
... )
>>> unpack_dtypes([struct_dtype, list_dtype]) # doctest: +IGNORE_RESULT
{Float64, Int64, String}
>>> unpack_dtypes(
... [struct_dtype, list_dtype], include_compound=True
... ) # doctest: +IGNORE_RESULT
{Float64, Int64, String, List(Float64), Struct([Field('a', Int64), Field('b', String), Field('c', List(Float64))])}
DATA
DTYPE_TEMPORAL_UNITS = frozenset({'ms', 'ns', 'us'})
N_INFER_DEFAULT = 100
__all__ = ['Array', 'BaseExtension', 'Binary', 'Boolean', 'Categorical...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\datatypes\__init__.py
polars.date
Python Library Documentation: function date_ in module polars.functions.as_datatype
date_(year: 'Expr | str | int', month: 'Expr | str | int', day: 'Expr | str | int') -> 'Expr'
Create a Polars literal expression of type Date.
Parameters
----------
year
column or literal.
month
column or literal, ranging from 1-12.
day
column or literal, ranging from 1-31.
Returns
-------
Expr
Expression of data type :class:`Date`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "month": [1, 2, 3],
... "day": [4, 5, 6],
... }
... )
>>> df.with_columns(pl.date(2024, pl.col("month"), pl.col("day")))
shape: (3, 3)
┌───────┬─────┬────────────┐
│ month ┆ day ┆ date │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ date │
╞═══════╪═════╪════════════╡
│ 1 ┆ 4 ┆ 2024-01-04 │
│ 2 ┆ 5 ┆ 2024-02-05 │
│ 3 ┆ 6 ┆ 2024-03-06 │
└───────┴─────┴────────────┘
We can also use `pl.date` for filtering:
>>> from datetime import date
>>> df = pl.DataFrame(
... {
... "start": [date(2024, 1, 1), date(2024, 1, 1), date(2024, 1, 1)],
... "end": [date(2024, 5, 1), date(2024, 7, 1), date(2024, 9, 1)],
... }
... )
>>> df.filter(pl.col("end") > pl.date(2024, 6, 1))
shape: (2, 2)
┌────────────┬────────────┐
│ start ┆ end │
│ --- ┆ --- │
│ date ┆ date │
╞════════════╪════════════╡
│ 2024-01-01 ┆ 2024-07-01 │
│ 2024-01-01 ┆ 2024-09-01 │
└────────────┴────────────┘
polars.date_range
Python Library Documentation: function date_range in module polars.functions.range.date_range
date_range(start: 'date | datetime | IntoExprColumn', end: 'date | datetime | IntoExprColumn', interval: 'str | timedelta' = '1d', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Series | Expr'
Generate a date range.
Parameters
----------
start
Lower bound of the date range.
end
Upper bound of the date range.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
Must consist of full days.
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type :class:`Date`.
See Also
--------
date_ranges
datetime_range
Notes
-----
`interval` is created according to the following string language:
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"1w2d" # 1 week, 2 days
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
Examples
--------
Using Polars duration string to specify the interval:
>>> from datetime import date
>>> pl.date_range(date(2022, 1, 1), date(2022, 3, 1), "1mo", eager=True).alias(
... "date"
... )
shape: (3,)
Series: 'date' [date]
[
2022-01-01
2022-02-01
2022-03-01
]
Using `timedelta` object to specify the interval:
>>> from datetime import timedelta
>>> pl.date_range(
... date(1985, 1, 1),
... date(1985, 1, 10),
... timedelta(days=2),
... eager=True,
... ).alias("date")
shape: (5,)
Series: 'date' [date]
[
1985-01-01
1985-01-03
1985-01-05
1985-01-07
1985-01-09
]
Omit `eager=True` if you want to use `date_range` as an expression:
>>> df = pl.DataFrame(
... {
... "date": [
... date(2024, 1, 1),
... date(2024, 1, 2),
... date(2024, 1, 1),
... date(2024, 1, 3),
... ],
... "key": ["one", "one", "two", "two"],
... }
... )
>>> result = (
... df.group_by("key")
... .agg(pl.date_range(pl.col("date").min(), pl.col("date").max()))
... .sort("key")
... )
>>> with pl.Config(fmt_str_lengths=50):
... print(result)
shape: (2, 2)
┌─────┬──────────────────────────────────────┐
│ key ┆ date │
│ --- ┆ --- │
│ str ┆ list[date] │
╞═════╪══════════════════════════════════════╡
│ one ┆ [2024-01-01, 2024-01-02] │
│ two ┆ [2024-01-01, 2024-01-02, 2024-01-03] │
└─────┴──────────────────────────────────────┘
polars.date_ranges
Python Library Documentation: function date_ranges in module polars.functions.range.date_range
date_ranges(start: 'date | datetime | IntoExprColumn', end: 'date | datetime | IntoExprColumn', interval: 'str | timedelta' = '1d', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Series | Expr'
Create a column of date ranges.
Parameters
----------
start
Lower bound of the date range.
end
Upper bound of the date range.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
Must consist of full days.
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type `List(Date)`.
See Also
--------
date_range
datetime_ranges
Notes
-----
`interval` is created according to the following string language:
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"1w2d" # 1 week, 2 days
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
Examples
--------
>>> from datetime import date
>>> df = pl.DataFrame(
... {
... "start": [date(2022, 1, 1), date(2022, 1, 2)],
... "end": date(2022, 1, 3),
... }
... )
>>> with pl.Config(fmt_str_lengths=50):
... df.with_columns(date_range=pl.date_ranges("start", "end"))
shape: (2, 3)
┌────────────┬────────────┬──────────────────────────────────────┐
│ start ┆ end ┆ date_range │
│ --- ┆ --- ┆ --- │
│ date ┆ date ┆ list[date] │
╞════════════╪════════════╪══════════════════════════════════════╡
│ 2022-01-01 ┆ 2022-01-03 ┆ [2022-01-01, 2022-01-02, 2022-01-03] │
│ 2022-01-02 ┆ 2022-01-03 ┆ [2022-01-02, 2022-01-03] │
└────────────┴────────────┴──────────────────────────────────────┘
polars.datetime
Python Library Documentation: function datetime_ in module polars.functions.as_datatype
datetime_(year: 'int | IntoExpr', month: 'int | IntoExpr', day: 'int | IntoExpr', hour: 'int | IntoExpr | None' = None, minute: 'int | IntoExpr | None' = None, second: 'int | IntoExpr | None' = None, microsecond: 'int | IntoExpr | None' = None, *, time_unit: 'TimeUnit' = 'us', time_zone: 'str | None' = None, ambiguous: 'Ambiguous | Expr' = 'raise') -> 'Expr'
Create a Polars literal expression of type Datetime.
Parameters
----------
year
Column or literal.
month
Column or literal, ranging from 1-12.
day
Column or literal, ranging from 1-31.
hour
Column or literal, ranging from 0-23.
minute
Column or literal, ranging from 0-59.
second
Column or literal, ranging from 0-59.
microsecond
Column or literal, ranging from 0-999999.
time_unit : {'us', 'ms', 'ns'}
Time unit of the resulting expression.
time_zone
Time zone of the resulting expression.
ambiguous
Determine how to deal with ambiguous datetimes:
- `'raise'` (default): raise
- `'earliest'`: use the earliest datetime
- `'latest'`: use the latest datetime
- `'null'`: set to null
Returns
-------
Expr
Expression of data type :class:`Datetime`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "month": [1, 2, 3],
... "day": [4, 5, 6],
... "hour": [12, 13, 14],
... "minute": [15, 30, 45],
... }
... )
>>> df.with_columns(
... pl.datetime(
... 2024,
... pl.col("month"),
... pl.col("day"),
... pl.col("hour"),
... pl.col("minute"),
... time_zone="Australia/Sydney",
... )
... )
shape: (3, 5)
┌───────┬─────┬──────┬────────┬────────────────────────────────┐
│ month ┆ day ┆ hour ┆ minute ┆ datetime │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 ┆ datetime[μs, Australia/Sydney] │
╞═══════╪═════╪══════╪════════╪════════════════════════════════╡
│ 1 ┆ 4 ┆ 12 ┆ 15 ┆ 2024-01-04 12:15:00 AEDT │
│ 2 ┆ 5 ┆ 13 ┆ 30 ┆ 2024-02-05 13:30:00 AEDT │
│ 3 ┆ 6 ┆ 14 ┆ 45 ┆ 2024-03-06 14:45:00 AEDT │
└───────┴─────┴──────┴────────┴────────────────────────────────┘
We can also use `pl.datetime` for filtering:
>>> from datetime import datetime
>>> df = pl.DataFrame(
... {
... "start": [
... datetime(2024, 1, 1, 0, 0, 0),
... datetime(2024, 1, 1, 0, 0, 0),
... datetime(2024, 1, 1, 0, 0, 0),
... ],
... "end": [
... datetime(2024, 5, 1, 20, 15, 10),
... datetime(2024, 7, 1, 21, 25, 20),
... datetime(2024, 9, 1, 22, 35, 30),
... ],
... }
... )
>>> df.filter(pl.col("end") > pl.datetime(2024, 6, 1))
shape: (2, 2)
┌─────────────────────┬─────────────────────┐
│ start ┆ end │
│ --- ┆ --- │
│ datetime[μs] ┆ datetime[μs] │
╞═════════════════════╪═════════════════════╡
│ 2024-01-01 00:00:00 ┆ 2024-07-01 21:25:20 │
│ 2024-01-01 00:00:00 ┆ 2024-09-01 22:35:30 │
└─────────────────────┴─────────────────────┘
polars.datetime_range
Python Library Documentation: function datetime_range in module polars.functions.range.datetime_range
datetime_range(start: 'datetime | date | IntoExprColumn', end: 'datetime | date | IntoExprColumn', interval: 'str | timedelta' = '1d', *, closed: 'ClosedInterval' = 'both', time_unit: 'TimeUnit | None' = None, time_zone: 'str | None' = None, eager: 'bool' = False) -> 'Series | Expr'
Generate a datetime range.
Parameters
----------
start
Lower bound of the datetime range.
end
Upper bound of the datetime range.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
time_unit : {None, 'ns', 'us', 'ms'}
Time unit of the resulting `Datetime` data type.
time_zone
Time zone of the resulting `Datetime` data type.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type :class:`Datetime`.
See Also
--------
datetime_ranges
date_range
Notes
-----
`interval` is created according to the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
Examples
--------
Using Polars duration string to specify the interval:
>>> from datetime import datetime
>>> pl.datetime_range(
... datetime(2022, 1, 1), datetime(2022, 3, 1), "1mo", eager=True
... ).alias("datetime")
shape: (3,)
Series: 'datetime' [datetime[μs]]
[
2022-01-01 00:00:00
2022-02-01 00:00:00
2022-03-01 00:00:00
]
Using `timedelta` object to specify the interval:
>>> from datetime import date, timedelta
>>> pl.datetime_range(
... date(1985, 1, 1),
... date(1985, 1, 10),
... timedelta(days=1, hours=12),
... time_unit="ms",
... eager=True,
... ).alias("datetime")
shape: (7,)
Series: 'datetime' [datetime[ms]]
[
1985-01-01 00:00:00
1985-01-02 12:00:00
1985-01-04 00:00:00
1985-01-05 12:00:00
1985-01-07 00:00:00
1985-01-08 12:00:00
1985-01-10 00:00:00
]
Specifying a time zone:
>>> pl.datetime_range(
... datetime(2022, 1, 1),
... datetime(2022, 3, 1),
... "1mo",
... time_zone="America/New_York",
... eager=True,
... ).alias("datetime")
shape: (3,)
Series: 'datetime' [datetime[μs, America/New_York]]
[
2022-01-01 00:00:00 EST
2022-02-01 00:00:00 EST
2022-03-01 00:00:00 EST
]
Omit `eager=True` if you want to use `datetime_range` as an expression:
>>> df = pl.DataFrame(
... {
... "date": [
... date(2024, 1, 1),
... date(2024, 1, 2),
... date(2024, 1, 1),
... date(2024, 1, 3),
... ],
... "key": ["one", "one", "two", "two"],
... }
... )
>>> result = (
... df.group_by("key")
... .agg(pl.datetime_range(pl.col("date").min(), pl.col("date").max()))
... .sort("key")
... )
>>> with pl.Config(fmt_str_lengths=70):
... print(result)
shape: (2, 2)
┌─────┬─────────────────────────────────────────────────────────────────┐
│ key ┆ date │
│ --- ┆ --- │
│ str ┆ list[datetime[μs]] │
╞═════╪═════════════════════════════════════════════════════════════════╡
│ one ┆ [2024-01-01 00:00:00, 2024-01-02 00:00:00] │
│ two ┆ [2024-01-01 00:00:00, 2024-01-02 00:00:00, 2024-01-03 00:00:00] │
└─────┴─────────────────────────────────────────────────────────────────┘
polars.datetime_ranges
Python Library Documentation: function datetime_ranges in module polars.functions.range.datetime_range
datetime_ranges(start: 'datetime | date | IntoExprColumn', end: 'datetime | date | IntoExprColumn', interval: 'str | timedelta' = '1d', *, closed: 'ClosedInterval' = 'both', time_unit: 'TimeUnit | None' = None, time_zone: 'str | None' = None, eager: 'bool' = False) -> 'Series | Expr'
Create a column of datetime ranges.
Parameters
----------
start
Lower bound of the datetime range.
end
Upper bound of the datetime range.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
time_unit : {None, 'ns', 'us', 'ms'}
Time unit of the resulting `Datetime` data type.
time_zone
Time zone of the resulting `Datetime` data type.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Notes
-----
`interval` is created according to the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
Returns
-------
Expr or Series
Column of data type `List(Datetime)`.
See Also
--------
datetime_range
date_ranges
Examples
--------
>>> from datetime import datetime
>>> df = pl.DataFrame(
... {
... "start": [datetime(2022, 1, 1), datetime(2022, 1, 2)],
... "end": datetime(2022, 1, 3),
... }
... )
>>> with pl.Config(fmt_str_lengths=100):
... df.select(datetime_range=pl.datetime_ranges("start", "end"))
shape: (2, 1)
┌─────────────────────────────────────────────────────────────────┐
│ datetime_range │
│ --- │
│ list[datetime[μs]] │
╞═════════════════════════════════════════════════════════════════╡
│ [2022-01-01 00:00:00, 2022-01-02 00:00:00, 2022-01-03 00:00:00] │
│ [2022-01-02 00:00:00, 2022-01-03 00:00:00] │
└─────────────────────────────────────────────────────────────────┘
polars.defer
Python Library Documentation: function _defer in module polars.io.plugins
_defer(function: 'Callable[[], DataFrame]', *, schema: 'SchemaDict | Callable[[], SchemaDict]', validate_schema: 'bool' = True) -> 'LazyFrame'
Deferred execution.
Takes a function that produces a `DataFrame` but defers execution until the
`LazyFrame` is collected.
Parameters
----------
function
Function that takes no arguments and produces a `DataFrame`.
schema
Schema of the `DataFrame` the deferred function will return.
The caller must ensure this schema is correct.
validate_schema
Whether the engine should validate if the batches generated match
the given schema. It's an implementation error if this isn't
the case and can lead to bugs that are hard to solve.
Examples
--------
Delay DataFrame execution until query is executed.
>>> import numpy as np
>>> np.random.seed(0)
>>> lf = pl.defer(
... lambda: pl.DataFrame({"a": np.random.randn(3)}), schema={"a": pl.Float64}
... )
>>> lf.collect()
shape: (3, 1)
┌──────────┐
│ a │
│ --- │
│ f64 │
╞══════════╡
│ 1.764052 │
│ 0.400157 │
│ 0.978738 │
└──────────┘
Run an eager source in Polars Cloud
>>> (
... pl.defer(
... lambda: pl.read_database("select * from tbl"),
... schema={"a": pl.Float64, "b": pl.Boolean},
... )
... .filter("b")
... .sum("a")
... .remote()
... .collect()
... ) # doctest: +SKIP
polars.disable_string_cache
Python Library Documentation: function disable_string_cache in module polars.string_cache
disable_string_cache() -> 'None'
Does nothing.
.. deprecated:: 1.41.0
The string cache was used to maintain the mapping for the Categorical
dtype, this is now done through ``pl.Categories``.
polars.dtype_of
Python Library Documentation: function dtype_of in module polars.functions.datatype
dtype_of(col_or_expr: 'str | Expr') -> 'pl.DataTypeExpr'
Get a lazily evaluated :class:`DataType` of a column or expression.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Examples
--------
>>> def inspect(expr: pl.Expr) -> pl.Expr:
... def print_and_return(s: pl.Series) -> pl.Series:
... print(s)
... return s
...
... return expr.map_batches(
... print_and_return,
... # Clarify that the expression returns the same datatype as the input
... # datatype.
... return_dtype=pl.dtype_of(expr),
... )
>>> df = pl.DataFrame(
... {
... "UserID": [1, 2, 3, 4, 5],
... "Name": ["Alice", "Bob", "Charlie", "Diana", "Ethan"],
... }
... )
>>> df.select(inspect(pl.col("Name")))
shape: (5,)
Series: 'Name' [str]
[
"Alice"
"Bob"
"Charlie"
"Diana"
"Ethan"
]
shape: (5, 1)
┌─────────┐
│ Name │
│ --- │
│ str │
╞═════════╡
│ Alice │
│ Bob │
│ Charlie │
│ Diana │
│ Ethan │
└─────────┘
polars.duration
Python Library Documentation: function duration in module polars.functions.as_datatype
duration(*, weeks: 'Expr | str | int | float | None' = None, days: 'Expr | str | int | float | None' = None, hours: 'Expr | str | int | float | None' = None, minutes: 'Expr | str | int | float | None' = None, seconds: 'Expr | str | int | float | None' = None, milliseconds: 'Expr | str | int | float | None' = None, microseconds: 'Expr | str | int | float | None' = None, nanoseconds: 'Expr | str | int | float | None' = None, time_unit: 'TimeUnit | None' = None) -> 'Expr'
Create polars `Duration` from distinct time components.
Parameters
----------
weeks
Number of weeks.
days
Number of days.
hours
Number of hours.
minutes
Number of minutes.
seconds
Number of seconds.
milliseconds
Number of milliseconds.
microseconds
Number of microseconds.
nanoseconds
Number of nanoseconds.
time_unit : {None, 'us', 'ms', 'ns'}
Time unit of the resulting expression. If set to `None` (default), the time
unit will be inferred from the other inputs: `'ns'` if `nanoseconds` was
specified, `'us'` otherwise.
Returns
-------
Expr
Expression of data type :class:`Duration`.
Notes
-----
A `duration` represents a fixed amount of time. For example,
`pl.duration(days=1)` means "exactly 24 hours". By contrast,
`Expr.dt.offset_by('1d')` means "1 calendar day", which could sometimes be
23 hours or 25 hours depending on Daylight Savings Time.
For non-fixed durations such as "calendar month" or "calendar day",
please use :meth:`polars.Expr.dt.offset_by` instead.
Examples
--------
>>> from datetime import datetime
>>> df = pl.DataFrame(
... {
... "dt": [datetime(2022, 1, 1), datetime(2022, 1, 2)],
... "add": [1, 2],
... }
... )
>>> df
shape: (2, 2)
┌─────────────────────┬─────┐
│ dt ┆ add │
│ --- ┆ --- │
│ datetime[μs] ┆ i64 │
╞═════════════════════╪═════╡
│ 2022-01-01 00:00:00 ┆ 1 │
│ 2022-01-02 00:00:00 ┆ 2 │
└─────────────────────┴─────┘
>>> with pl.Config(tbl_width_chars=120):
... df.select(
... (pl.col("dt") + pl.duration(weeks="add")).alias("add_weeks"),
... (pl.col("dt") + pl.duration(days="add")).alias("add_days"),
... (pl.col("dt") + pl.duration(seconds="add")).alias("add_seconds"),
... (pl.col("dt") + pl.duration(milliseconds="add")).alias("add_millis"),
... (pl.col("dt") + pl.duration(hours="add")).alias("add_hours"),
... )
shape: (2, 5)
┌─────────────────────┬─────────────────────┬─────────────────────┬─────────────────────────┬─────────────────────┐
│ add_weeks ┆ add_days ┆ add_seconds ┆ add_millis ┆ add_hours │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] │
╞═════════════════════╪═════════════════════╪═════════════════════╪═════════════════════════╪═════════════════════╡
│ 2022-01-08 00:00:00 ┆ 2022-01-02 00:00:00 ┆ 2022-01-01 00:00:01 ┆ 2022-01-01 00:00:00.001 ┆ 2022-01-01 01:00:00 │
│ 2022-01-16 00:00:00 ┆ 2022-01-04 00:00:00 ┆ 2022-01-02 00:00:02 ┆ 2022-01-02 00:00:00.002 ┆ 2022-01-02 02:00:00 │
└─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────────┴─────────────────────┘
If you need to add non-fixed durations, you should use :meth:`polars.Expr.dt.offset_by` instead:
>>> with pl.Config(tbl_width_chars=120):
... df.select(
... add_calendar_days=pl.col("dt").dt.offset_by(
... pl.format("{}d", pl.col("add"))
... ),
... add_calendar_months=pl.col("dt").dt.offset_by(
... pl.format("{}mo", pl.col("add"))
... ),
... add_calendar_years=pl.col("dt").dt.offset_by(
... pl.format("{}y", pl.col("add"))
... ),
... )
shape: (2, 3)
┌─────────────────────┬─────────────────────┬─────────────────────┐
│ add_calendar_days ┆ add_calendar_months ┆ add_calendar_years │
│ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] │
╞═════════════════════╪═════════════════════╪═════════════════════╡
│ 2022-01-02 00:00:00 ┆ 2022-02-01 00:00:00 ┆ 2023-01-01 00:00:00 │
│ 2022-01-04 00:00:00 ┆ 2022-03-02 00:00:00 ┆ 2024-01-02 00:00:00 │
└─────────────────────┴─────────────────────┴─────────────────────┘
polars.element
Python Library Documentation: function element in module polars.functions.lazy
element() -> 'Expr'
Alias for an element being evaluated in an `eval` or `filter` expression.
Examples
--------
A horizontal rank computation by taking the elements of a list
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... }
... )
>>> df.with_columns(
... pl.concat_list(["a", "b"]).list.eval(pl.element().rank()).alias("rank")
... )
shape: (3, 3)
┌─────┬─────┬────────────┐
│ a ┆ b ┆ rank │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[f64] │
╞═════╪═════╪════════════╡
│ 1 ┆ 4 ┆ [1.0, 2.0] │
│ 8 ┆ 5 ┆ [2.0, 1.0] │
│ 3 ┆ 2 ┆ [2.0, 1.0] │
└─────┴─────┴────────────┘
A mathematical operation on array elements
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... }
... )
>>> df.with_columns(
... pl.concat_list(["a", "b"]).list.eval(pl.element() * 2).alias("a_b_doubled")
... )
shape: (3, 3)
┌─────┬─────┬─────────────┐
│ a ┆ b ┆ a_b_doubled │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[i64] │
╞═════╪═════╪═════════════╡
│ 1 ┆ 4 ┆ [2, 8] │
│ 8 ┆ 5 ┆ [16, 10] │
│ 3 ┆ 2 ┆ [6, 4] │
└─────┴─────┴─────────────┘
A filter operation on list elements
>>> import polars as pl
>>> df = pl.DataFrame({"a": [1, 8, 3], "b": [4, 5, 2]})
>>> df.with_columns(
... evens=pl.concat_list("a", "b").list.filter(pl.element() % 2 == 0)
... )
shape: (3, 3)
┌─────┬─────┬───────────┐
│ a ┆ b ┆ evens │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[i64] │
╞═════╪═════╪═══════════╡
│ 1 ┆ 4 ┆ [4] │
│ 8 ┆ 5 ┆ [8] │
│ 3 ┆ 2 ┆ [2] │
└─────┴─────┴───────────┘
polars.enable_string_cache
Python Library Documentation: function enable_string_cache in module polars.string_cache
enable_string_cache() -> 'None'
Does nothing.
.. deprecated:: 1.41.0
The string cache was used to maintain the mapping for the Categorical
dtype, this is now done through ``pl.Categories``.
polars.escape_regex
Python Library Documentation: function escape_regex in module polars.functions.escape_regex
escape_regex(s: 'str') -> 'str'
Escapes string regex meta characters.
Parameters
----------
s
The string whose meta characters will be escaped.
polars.exceptions
Python Library Documentation: module polars.exceptions in polars
NAME
polars.exceptions
CLASSES
builtins.BaseException(builtins.object)
pyo3_runtime.PanicException
builtins.Exception(builtins.BaseException)
PolarsError
ColumnNotFoundError
ComputeError
DuplicateError
InvalidOperationError
NoDataError
OutOfBoundsError
ParameterCollisionError
RowsError
NoRowsReturnedError
TooManyRowsReturnedError
SQLInterfaceError
SQLSyntaxError
SchemaError
SchemaFieldNotFoundError
ShapeError
StringCacheMismatchError
StructFieldNotFoundError
UnsuitableSQLError
builtins.ModuleNotFoundError(builtins.ImportError)
ModuleUpgradeRequiredError
builtins.Warning(builtins.Exception)
PolarsWarning
ChronoFormatWarning
CustomUFuncWarning
DataOrientationWarning
MapWithoutReturnDtypeWarning
PerformanceWarning
CategoricalRemappingWarning
PolarsInefficientMapWarning
UnstableWarning
class CategoricalRemappingWarning(PerformanceWarning)
| Method resolution order:
| CategoricalRemappingWarning
| PerformanceWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ChronoFormatWarning(PolarsWarning)
| Warning issued when a chrono format string contains dubious patterns.
|
| Polars uses Rust's chrono crate to convert between string data and temporal data.
| The patterns used by chrono differ slightly from Python's built-in datetime module.
| Refer to the `chrono strftime documentation
| <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_ for the full
| specification.
|
| Method resolution order:
| ChronoFormatWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ColumnNotFoundError(PolarsError)
| Method resolution order:
| ColumnNotFoundError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ComputeError(PolarsError)
| Method resolution order:
| ComputeError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class CustomUFuncWarning(PolarsWarning)
| Warning issued when a custom ufunc is handled differently than numpy ufunc would.
|
| Method resolution order:
| CustomUFuncWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DataOrientationWarning(PolarsWarning)
| Warning issued to indicate row orientation was inferred from the inputs.
|
| Occurs when constructing a DataFrame from a list of rows without explicitly
| specifying row orientation. Polars is usually able to infer the data orientation
| from the data and schema, but there are cases where this is not possible. This is a
| common source of confusion. Use the `orient` parameter to be explicit about the
| data orientation.
|
| Examples
| --------
| >>> pl.DataFrame([(1, 2, 3), (4, 5, 6)], schema=["a", "b", "c"]) # doctest: +SKIP
| DataOrientationWarning: Row orientation inferred during DataFrame construction.
| Explicitly specify the orientation by passing `orient="row"` to silence this warning.
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| Pass `orient="row"` to silence the warning.
|
| >>> pl.DataFrame([[1, 2, 3], [4, 5, 6]], schema=["a", "b", "c"], orient="row")
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| Method resolution order:
| DataOrientationWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DuplicateError(PolarsError)
| Method resolution order:
| DuplicateError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class InvalidOperationError(PolarsError)
| Method resolution order:
| InvalidOperationError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class MapWithoutReturnDtypeWarning(PolarsWarning)
| Method resolution order:
| MapWithoutReturnDtypeWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ModuleUpgradeRequiredError(builtins.ModuleNotFoundError)
| Exception raised when a module is installed but needs to be upgraded.
|
| Method resolution order:
| ModuleUpgradeRequiredError
| builtins.ModuleNotFoundError
| builtins.ImportError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.ModuleNotFoundError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.ImportError:
|
| __reduce__(...)
| Helper for pickle.
|
| __str__(self, /)
| Return str(self).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.ImportError:
|
| msg
| exception message
|
| name
| module name
|
| name_from
| name imported from module
|
| path
| module path
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NoDataError(PolarsError)
| Method resolution order:
| NoDataError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NoRowsReturnedError(RowsError)
| Exception raised when no rows are returned, but at least one row is expected.
|
| Method resolution order:
| NoRowsReturnedError
| RowsError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class OutOfBoundsError(PolarsError)
| Method resolution order:
| OutOfBoundsError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PanicException(builtins.BaseException)
| The exception raised when Rust code called from Python panics.
|
| Like SystemExit, this exception is derived from BaseException so that
| it will typically propagate all the way through the stack and cause the
| Python interpreter to exit.
|
| Method resolution order:
| PanicException
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.BaseException:
|
| __new__(*args, **kwargs) class method of builtins.BaseException
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ParameterCollisionError(PolarsError)
| Exception raised when the same parameter occurs multiple times.
|
| Method resolution order:
| ParameterCollisionError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PerformanceWarning(PolarsWarning)
| Method resolution order:
| PerformanceWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PolarsError(builtins.Exception)
| Method resolution order:
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PolarsInefficientMapWarning(PerformanceWarning)
| Warning issued when a potentially slow `map_*` operation is performed.
|
| Method resolution order:
| PolarsInefficientMapWarning
| PerformanceWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PolarsWarning(builtins.Warning)
| Method resolution order:
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class RowsError(PolarsError)
| Exception raised when the number of returned rows does not match expectation.
|
| Method resolution order:
| RowsError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class SQLInterfaceError(PolarsError)
| Method resolution order:
| SQLInterfaceError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class SQLSyntaxError(PolarsError)
| Method resolution order:
| SQLSyntaxError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class SchemaError(PolarsError)
| Method resolution order:
| SchemaError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class SchemaFieldNotFoundError(PolarsError)
| Method resolution order:
| SchemaFieldNotFoundError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ShapeError(PolarsError)
| Method resolution order:
| ShapeError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class StringCacheMismatchError(PolarsError)
| Method resolution order:
| StringCacheMismatchError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class StructFieldNotFoundError(PolarsError)
| Method resolution order:
| StructFieldNotFoundError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class TooManyRowsReturnedError(RowsError)
| Exception raised when more rows than expected are returned.
|
| Method resolution order:
| TooManyRowsReturnedError
| RowsError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class UnstableWarning(PolarsWarning)
| Warning issued when unstable functionality is used.
|
| Method resolution order:
| UnstableWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class UnsuitableSQLError(PolarsError)
| Exception raised when unsuitable SQL is given to a database method.
|
| Method resolution order:
| UnsuitableSQLError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
DATA
__all__ = ['PolarsError', 'ColumnNotFoundError', 'ComputeError', 'Dupl...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\exceptions.py
polars.exclude
Python Library Documentation: function exclude in module polars.functions.lazy
exclude(columns: 'str | PolarsDataType | Collection[str] | Collection[PolarsDataType]', *more_columns: 'str | PolarsDataType') -> 'Expr'
Represent all columns except for the given columns.
Syntactic sugar for `pl.all().exclude(columns)`.
Parameters
----------
columns
The name or datatype of the column(s) to exclude. Accepts regular expression
input. Regular expressions should start with `^` and end with `$`.
*more_columns
Additional names or datatypes of columns to exclude, specified as positional
arguments.
Examples
--------
Exclude by column name(s):
>>> df = pl.DataFrame(
... {
... "aa": [1, 2, 3],
... "ba": ["a", "b", None],
... "cc": [None, 2.5, 1.5],
... }
... )
>>> df.select(pl.exclude("ba"))
shape: (3, 2)
┌─────┬──────┐
│ aa ┆ cc │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪══════╡
│ 1 ┆ null │
│ 2 ┆ 2.5 │
│ 3 ┆ 1.5 │
└─────┴──────┘
Exclude by regex, e.g. removing all columns whose names end with the letter "a":
>>> df.select(pl.exclude("^.*a$"))
shape: (3, 1)
┌──────┐
│ cc │
│ --- │
│ f64 │
╞══════╡
│ null │
│ 2.5 │
│ 1.5 │
└──────┘
Exclude by dtype(s), e.g. removing all columns of type Int64 or Float64:
>>> df.select(pl.exclude([pl.Int64, pl.Float64]))
shape: (3, 1)
┌──────┐
│ ba │
│ --- │
│ str │
╞══════╡
│ a │
│ b │
│ null │
└──────┘
polars.explain_all
Python Library Documentation: function explain_all in module polars.functions.lazy
explain_all(lazy_frames: 'Iterable[LazyFrame]', *, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'str'
Explain multiple LazyFrames as if passed to `collect_all`.
Common Subplan Elimination is applied on the combined plan, meaning
that diverging queries will run only once.
Parameters
----------
lazy_frames
A list of LazyFrames to collect.
optimizations
The optimization passes done during query optimization.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Returns
-------
Explained plan.
polars.expr
Python Library Documentation: package polars.expr in polars
NAME
polars.expr
PACKAGE CONTENTS
array
binary
categorical
datetime
expr
ext
list
meta
name
string
struct
whenthen
CLASSES
builtins.object
polars.expr.expr.Expr
polars.expr.whenthen.When
class Expr(builtins.object)
| Expressions that can be used in various contexts.
|
| Methods defined here:
|
| __abs__(self) -> 'Expr'
|
| __add__(self, other: 'IntoExpr') -> 'Expr'
| # operators
|
| __and__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
|
| __array_ufunc__(self, ufunc: 'Callable[..., Any]', method: 'str_', *inputs: 'Any', **kwargs: 'Any') -> 'Expr'
| Numpy universal functions.
|
| __bool__(self) -> 'NoReturn'
|
| __eq__(self, other: 'IntoExpr') -> 'Expr'
| Return self==value.
|
| __floordiv__(self, other: 'IntoExpr') -> 'Expr'
|
| __ge__(self, other: 'IntoExpr') -> 'Expr'
| Return self>=value.
|
| __getstate__(self) -> 'bytes'
| Helper for pickle.
|
| __gt__(self, other: 'IntoExpr') -> 'Expr'
| Return self>value.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __invert__(self) -> 'Expr'
|
| __le__(self, other: 'IntoExpr') -> 'Expr'
| Return self<=value.
|
| __lt__(self, other: 'IntoExpr') -> 'Expr'
| Return self<value.
|
| __mod__(self, other: 'IntoExpr') -> 'Expr'
|
| __mul__(self, other: 'IntoExpr') -> 'Expr'
|
| __ne__(self, other: 'IntoExpr') -> 'Expr'
| Return self!=value.
|
| __neg__(self) -> 'Expr'
|
| __or__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
| Return self|value.
|
| __pos__(self) -> 'Expr'
|
| __pow__(self, exponent: 'IntoExprColumn | int | float') -> 'Expr'
|
| __radd__(self, other: 'IntoExpr') -> 'Expr'
|
| __rand__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
|
| __repr__(self) -> 'str_'
| Return repr(self).
|
| __rfloordiv__(self, other: 'IntoExpr') -> 'Expr'
|
| __rmod__(self, other: 'IntoExpr') -> 'Expr'
|
| __rmul__(self, other: 'IntoExpr') -> 'Expr'
|
| __ror__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
| Return value|self.
|
| __rpow__(self, base: 'IntoExprColumn | int | float') -> 'Expr'
|
| __rsub__(self, other: 'IntoExpr') -> 'Expr'
|
| __rtruediv__(self, other: 'IntoExpr') -> 'Expr'
|
| __rxor__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
|
| __setstate__(self, state: 'bytes') -> 'None'
|
| __str__(self) -> 'str_'
| Return str(self).
|
| __sub__(self, other: 'IntoExpr') -> 'Expr'
|
| __truediv__(self, other: 'IntoExpr') -> 'Expr'
|
| __xor__(self, other: 'IntoExprColumn | int | bool') -> 'Expr'
|
| abs(self) -> 'Expr'
| Compute absolute values.
|
| Same as `abs(expr)`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [-1.0, 0.0, 1.0, 2.0],
| ... }
| ... )
| >>> df.select(pl.col("A").abs())
| shape: (4, 1)
| ┌─────┐
| │ A │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| │ 0.0 │
| │ 1.0 │
| │ 2.0 │
| └─────┘
|
| add(self, other: 'Any') -> 'Expr'
| Method equivalent of addition operator `expr + other`.
|
| Parameters
| ----------
| other
| numeric or string value; accepts expression input.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 3, 4, 5]})
| >>> df.with_columns(
| ... pl.col("x").add(2).alias("x+int"),
| ... pl.col("x").add(pl.col("x").cum_prod()).alias("x+expr"),
| ... )
| shape: (5, 3)
| ┌─────┬───────┬────────┐
| │ x ┆ x+int ┆ x+expr │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═══════╪════════╡
| │ 1 ┆ 3 ┆ 2 │
| │ 2 ┆ 4 ┆ 4 │
| │ 3 ┆ 5 ┆ 9 │
| │ 4 ┆ 6 ┆ 28 │
| │ 5 ┆ 7 ┆ 125 │
| └─────┴───────┴────────┘
|
| >>> df = pl.DataFrame(
| ... {"x": ["a", "d", "g"], "y": ["b", "e", "h"], "z": ["c", "f", "i"]}
| ... )
| >>> df.with_columns(pl.col("x").add(pl.col("y")).add(pl.col("z")).alias("xyz"))
| shape: (3, 4)
| ┌─────┬─────┬─────┬─────┐
| │ x ┆ y ┆ z ┆ xyz │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ str ┆ str │
| ╞═════╪═════╪═════╪═════╡
| │ a ┆ b ┆ c ┆ abc │
| │ d ┆ e ┆ f ┆ def │
| │ g ┆ h ┆ i ┆ ghi │
| └─────┴─────┴─────┴─────┘
|
| agg_groups(self) -> 'Expr'
| Get the group indexes of the group by operation.
|
| .. deprecated:: 1.35
| use `df.with_row_index().group_by(...).agg(pl.col('index'))` instead.
| This method will be removed in Polars 2.0.
|
| Should be used in aggregation context only.
|
| Examples
| --------
| >>> import warnings
| >>> warnings.filterwarnings("ignore", category=DeprecationWarning)
| >>> df = pl.DataFrame(
| ... {
| ... "group": [
| ... "one",
| ... "one",
| ... "one",
| ... "two",
| ... "two",
| ... "two",
| ... ],
| ... "value": [94, 95, 96, 97, 97, 99],
| ... }
| ... )
| >>> df.group_by("group", maintain_order=True).agg(pl.col("value").agg_groups())
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ list[u32] │
| ╞═══════╪═══════════╡
| │ one ┆ [0, 1, 2] │
| │ two ┆ [3, 4, 5] │
| └───────┴───────────┘
|
| New recommended approach:
| >>> (
| ... df.with_row_index()
| ... .group_by("group", maintain_order=True)
| ... .agg(pl.col("index"))
| ... )
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ index │
| │ --- ┆ --- │
| │ str ┆ list[u32] │
| ╞═══════╪═══════════╡
| │ one ┆ [0, 1, 2] │
| │ two ┆ [3, 4, 5] │
| └───────┴───────────┘
|
| alias(self, name: 'str_') -> 'Expr'
| Rename the expression.
|
| Parameters
| ----------
| name
| The new name.
|
| See Also
| --------
| name.map
| name.prefix
| name.suffix
|
| Examples
| --------
| Rename an expression to avoid overwriting an existing column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["x", "y", "z"],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("a") + 10,
| ... pl.col("b").str.to_uppercase().alias("c"),
| ... )
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 11 ┆ x ┆ X │
| │ 12 ┆ y ┆ Y │
| │ 13 ┆ z ┆ Z │
| └─────┴─────┴─────┘
|
| Overwrite the default name of literal columns to prevent errors due to duplicate
| column names.
|
| >>> df.with_columns(
| ... pl.lit(True).alias("c"),
| ... pl.lit(4.0).alias("d"),
| ... )
| shape: (3, 4)
| ┌─────┬─────┬──────┬─────┐
| │ a ┆ b ┆ c ┆ d │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ bool ┆ f64 │
| ╞═════╪═════╪══════╪═════╡
| │ 1 ┆ x ┆ true ┆ 4.0 │
| │ 2 ┆ y ┆ true ┆ 4.0 │
| │ 3 ┆ z ┆ true ┆ 4.0 │
| └─────┴─────┴──────┴─────┘
|
| all(self, *, ignore_nulls: 'bool' = True) -> 'Expr'
| Return whether all values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| .. note::
| This method is not to be confused with the function :func:`polars.all`,
| which can be used to select all columns.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `True`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `False` values,
| the output is null.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [True, True],
| ... "b": [False, True],
| ... "c": [None, True],
| ... }
| ... )
| >>> df.select(pl.col("*").all())
| shape: (1, 3)
| ┌──────┬───────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪══════╡
| │ true ┆ false ┆ true │
| └──────┴───────┴──────┘
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> df.select(pl.col("*").all(ignore_nulls=False))
| shape: (1, 3)
| ┌──────┬───────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪══════╡
| │ true ┆ false ┆ null │
| └──────┴───────┴──────┘
|
| and_(self, *others: 'Any') -> 'Expr'
| Method equivalent of bitwise "and" operator `expr & other & ...`.
|
| This has the effect of combining logical boolean expressions,
| but operates bitwise on integers.
|
| Parameters
| ----------
| *others
| One or more integer or boolean expressions to evaluate/combine.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5, 6, 7, 4, 8],
| ... "y": [1.5, 2.5, 1.0, 4.0, -5.75],
| ... "z": [-9, 2, -1, 4, 8],
| ... }
| ... )
|
| Combine logical "and" conditions:
|
| >>> df.select(
| ... (pl.col("x") >= pl.col("z"))
| ... .and_(
| ... pl.col("y") >= pl.col("z"),
| ... pl.col("y") == pl.col("y"),
| ... pl.col("z") <= pl.col("x"),
| ... pl.col("y") != pl.col("x"),
| ... )
| ... .alias("all")
| ... )
| shape: (5, 1)
| ┌───────┐
| │ all │
| │ --- │
| │ bool │
| ╞═══════╡
| │ true │
| │ true │
| │ true │
| │ false │
| │ false │
| └───────┘
|
| Bitwise "and" operation on integer columns:
|
| >>> df.select("x", "z", x_and_z=pl.col("x").and_(pl.col("z")))
| shape: (5, 3)
| ┌─────┬─────┬─────────┐
| │ x ┆ z ┆ x_and_z │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════════╡
| │ 5 ┆ -9 ┆ 5 │
| │ 6 ┆ 2 ┆ 2 │
| │ 7 ┆ -1 ┆ 7 │
| │ 4 ┆ 4 ┆ 4 │
| │ 8 ┆ 8 ┆ 8 │
| └─────┴─────┴─────────┘
|
| any(self, *, ignore_nulls: 'bool' = True) -> 'Expr'
| Return whether any of the values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `False`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `True` values,
| the output is null.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [True, False],
| ... "b": [False, False],
| ... "c": [None, False],
| ... }
| ... )
| >>> df.select(pl.col("*").any())
| shape: (1, 3)
| ┌──────┬───────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪═══════╡
| │ true ┆ false ┆ false │
| └──────┴───────┴───────┘
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> df.select(pl.col("*").any(ignore_nulls=False))
| shape: (1, 3)
| ┌──────┬───────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪══════╡
| │ true ┆ false ┆ null │
| └──────┴───────┴──────┘
|
| append(self, other: 'IntoExpr', *, upcast: 'bool' = True) -> 'Expr'
| Append expressions.
|
| This is done by adding the chunks of `other` to this `Series`.
|
| Parameters
| ----------
| other
| Expression to append.
| upcast
| Cast both `Series` to the same supertype.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [8, 9, 10],
| ... "b": [None, 4, 4],
| ... }
| ... )
| >>> df.select(pl.all().head(1).append(pl.all().tail(1)))
| shape: (2, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════╡
| │ 8 ┆ null │
| │ 10 ┆ 4 │
| └─────┴──────┘
|
| approx_n_unique(self) -> 'Expr'
| Approximate count of unique values.
|
| This is done using the HyperLogLog++ algorithm for cardinality estimation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [1, 1, 2]})
| >>> df.select(pl.col("n").approx_n_unique())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 2 │
| └─────┘
| >>> df = pl.DataFrame({"n": range(1000)})
| >>> df.select(
| ... exact=pl.col("n").n_unique(),
| ... approx=pl.col("n").approx_n_unique(),
| ... ) # doctest: +SKIP
| shape: (1, 2)
| ┌───────┬────────┐
| │ exact ┆ approx │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═══════╪════════╡
| │ 1000 ┆ 1005 │
| └───────┴────────┘
|
| arccos(self) -> 'Expr'
| Compute the element-wise value for the inverse cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`, measuring an angle in radians.
|
| Notes
| -----
| To convert the result from radians to degrees,
| call :meth:`.degrees() <polars.Expr.degrees>`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, 0.5, 0]})
| >>> df.select(pl.col("a").arccos())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.0 │
| │ 1.047198 │
| │ 1.570796 │
| └──────────┘
| >>> df.select(pl.col("a").arccos().degrees())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 0.0 │
| │ 60.0 │
| │ 90.0 │
| └──────┘
|
| arccosh(self) -> 'Expr'
| Compute the element-wise value for the inverse hyperbolic cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").arccosh())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| └─────┘
|
| arcsin(self) -> 'Expr'
| Compute the element-wise value for the inverse sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`, measuring an angle in radians.
|
| Notes
| -----
| To convert the result from radians to degrees,
| call :meth:`.degrees() <polars.Expr.degrees>`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, 0.5, 0]})
| >>> df.select(pl.col("a").arcsin())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.570796 │
| │ 0.523599 │
| │ 0.0 │
| └──────────┘
| >>> df.select(pl.col("a").arcsin().degrees())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 90.0 │
| │ 30.0 │
| │ 0.0 │
| └──────┘
|
| arcsinh(self) -> 'Expr'
| Compute the element-wise value for the inverse hyperbolic sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").arcsinh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.881374 │
| └──────────┘
|
| arctan(self) -> 'Expr'
| Compute the element-wise value for the inverse tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`, measuring an angle in radians.
|
| Notes
| -----
| To convert the result from radians to degrees,
| call :meth:`.degrees() <polars.Expr.degrees>`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [float("Inf"), 1, 0]})
| >>> df.select(pl.col("a").arctan())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.570796 │
| │ 0.785398 │
| │ 0.0 │
| └──────────┘
| >>> df.select(pl.col("a").arctan().degrees())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 90.0 │
| │ 45.0 │
| │ 0.0 │
| └──────┘
|
| arctanh(self) -> 'Expr'
| Compute the element-wise value for the inverse hyperbolic tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").arctanh())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ inf │
| └─────┘
|
| arg_max(self) -> 'Expr'
| Get the index of the maximal value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [20, 10, 30],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_max())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 2 │
| └─────┘
|
| arg_min(self) -> 'Expr'
| Get the index of the minimal value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [20, 10, 30],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_min())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| └─────┘
|
| arg_sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Expr'
| Get the index values that would sort this column.
|
| Parameters
| ----------
| descending
| Sort in descending (descending) order.
| nulls_last
| Place null values last instead of first.
|
| Returns
| -------
| Expr
| Expression of data type :class:`UInt32`.
|
| See Also
| --------
| Expr.gather: Take values by index.
| Expr.rank : Get the rank of each row.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [20, 10, 30],
| ... "b": [1, 2, 3],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_sort())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| │ 0 │
| │ 2 │
| └─────┘
|
| Use gather to apply the arg sort to other columns.
|
| >>> df.select(pl.col("b").gather(pl.col("a").arg_sort()))
| shape: (3, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 1 │
| │ 3 │
| └─────┘
|
| arg_true(self) -> 'Expr'
| Return indices where expression evaluates `True`.
|
| .. warning::
| Modifies number of rows returned, so will fail in combination with other
| expressions. Use as only expression in `select` / `with_columns`.
|
| See Also
| --------
| Series.arg_true : Return indices where Series is True
| polars.arg_where
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 1]})
| >>> df.select((pl.col("a") == 1).arg_true())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 0 │
| │ 1 │
| │ 3 │
| └─────┘
|
| arg_unique(self) -> 'Expr'
| Get index of first unique value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [8, 9, 10],
| ... "b": [None, 4, 4],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_unique())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 0 │
| │ 1 │
| │ 2 │
| └─────┘
| >>> df.select(pl.col("b").arg_unique())
| shape: (2, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 0 │
| │ 1 │
| └─────┘
|
| backward_fill(self, limit: 'int | None' = None) -> 'Expr'
| Fill missing values with the next non-null value.
|
| This is an alias of `.fill_null(strategy="backward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to backward fill.
|
| See Also
| --------
| fill_null
| forward_fill
| shift
|
| bitwise_and(self) -> 'Expr'
| Perform an aggregation of bitwise ANDs.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-1, 0, 1]})
| >>> df.select(pl.col("n").bitwise_and())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 0 │
| └─────┘
| >>> df = pl.DataFrame(
| ... {"grouper": ["a", "a", "a", "b", "b"], "n": [-1, 0, 1, -1, 1]}
| ... )
| >>> df.group_by("grouper", maintain_order=True).agg(pl.col("n").bitwise_and())
| shape: (2, 2)
| ┌─────────┬─────┐
| │ grouper ┆ n │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════╡
| │ a ┆ 0 │
| │ b ┆ 1 │
| └─────────┴─────┘
|
| bitwise_count_ones(self) -> 'Expr'
| Evaluate the number of set bits.
|
| bitwise_count_zeros(self) -> 'Expr'
| Evaluate the number of unset bits.
|
| bitwise_leading_ones(self) -> 'Expr'
| Evaluate the number most-significant set bits before seeing an unset bit.
|
| bitwise_leading_zeros(self) -> 'Expr'
| Evaluate the number most-significant unset bits before seeing a set bit.
|
| bitwise_or(self) -> 'Expr'
| Perform an aggregation of bitwise ORs.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-1, 0, 1]})
| >>> df.select(pl.col("n").bitwise_or())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ i64 │
| ╞═════╡
| │ -1 │
| └─────┘
| >>> df = pl.DataFrame(
| ... {"grouper": ["a", "a", "a", "b", "b"], "n": [-1, 0, 1, -1, 1]}
| ... )
| >>> df.group_by("grouper", maintain_order=True).agg(pl.col("n").bitwise_or())
| shape: (2, 2)
| ┌─────────┬─────┐
| │ grouper ┆ n │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════╡
| │ a ┆ -1 │
| │ b ┆ -1 │
| └─────────┴─────┘
|
| bitwise_trailing_ones(self) -> 'Expr'
| Evaluate the number least-significant set bits before seeing an unset bit.
|
| bitwise_trailing_zeros(self) -> 'Expr'
| Evaluate the number least-significant unset bits before seeing a set bit.
|
| bitwise_xor(self) -> 'Expr'
| Perform an aggregation of bitwise XORs.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-1, 0, 1]})
| >>> df.select(pl.col("n").bitwise_xor())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ i64 │
| ╞═════╡
| │ -2 │
| └─────┘
| >>> df = pl.DataFrame(
| ... {"grouper": ["a", "a", "a", "b", "b"], "n": [-1, 0, 1, -1, 1]}
| ... )
| >>> df.group_by("grouper", maintain_order=True).agg(pl.col("n").bitwise_xor())
| shape: (2, 2)
| ┌─────────┬─────┐
| │ grouper ┆ n │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════╡
| │ a ┆ -2 │
| │ b ┆ -2 │
| └─────────┴─────┘
|
| bottom_k(self, k: 'int | IntoExprColumn' = 5) -> 'Expr'
| Return the `k` smallest elements.
|
| Non-null elements are always preferred over null elements. The output is
| not guaranteed to be in any particular order, call :func:`sort` after
| this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k_by
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.select(
| ... pl.col("value").top_k().alias("top_k"),
| ... pl.col("value").bottom_k().alias("bottom_k"),
| ... )
| shape: (5, 2)
| ┌───────┬──────────┐
| │ top_k ┆ bottom_k │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪══════════╡
| │ 4 ┆ 1 │
| │ 98 ┆ 98 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| │ 99 ┆ 4 │
| └───────┴──────────┘
|
| bottom_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int | IntoExprColumn' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Expr'
| Return the elements corresponding to the `k` smallest elements of the `by` column(s).
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| by
| Column(s) used to determine the smallest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` largest elements of the `by` column(s) (instead of the `k`
| smallest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [6, 5, 4, 3, 2, 1],
| ... "c": ["Apple", "Orange", "Apple", "Apple", "Banana", "Banana"],
| ... }
| ... )
| >>> df
| shape: (6, 3)
| ┌─────┬─────┬────────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪════════╡
| │ 1 ┆ 6 ┆ Apple │
| │ 2 ┆ 5 ┆ Orange │
| │ 3 ┆ 4 ┆ Apple │
| │ 4 ┆ 3 ┆ Apple │
| │ 5 ┆ 2 ┆ Banana │
| │ 6 ┆ 1 ┆ Banana │
| └─────┴─────┴────────┘
|
| Get the bottom 2 rows by column `a` or `b`.
|
| >>> df.select(
| ... pl.all().bottom_k_by("a", 2).name.suffix("_btm_by_a"),
| ... pl.all().bottom_k_by("b", 2).name.suffix("_btm_by_b"),
| ... )
| shape: (2, 6)
| ┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
| │ a_btm_by_a ┆ b_btm_by_a ┆ c_btm_by_a ┆ a_btm_by_b ┆ b_btm_by_b ┆ c_btm_by_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞════════════╪════════════╪════════════╪════════════╪════════════╪════════════╡
| │ 1 ┆ 6 ┆ Apple ┆ 6 ┆ 1 ┆ Banana │
| │ 2 ┆ 5 ┆ Orange ┆ 5 ┆ 2 ┆ Banana │
| └────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
|
| Get the bottom 2 rows by multiple columns with given order.
|
| >>> df.select(
| ... pl.all()
| ... .bottom_k_by(["c", "a"], 2, reverse=[False, True])
| ... .name.suffix("_by_ca"),
| ... pl.all()
| ... .bottom_k_by(["c", "b"], 2, reverse=[False, True])
| ... .name.suffix("_by_cb"),
| ... )
| shape: (2, 6)
| ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
| │ a_by_ca ┆ b_by_ca ┆ c_by_ca ┆ a_by_cb ┆ b_by_cb ┆ c_by_cb │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
| │ 4 ┆ 3 ┆ Apple ┆ 1 ┆ 6 ┆ Apple │
| │ 3 ┆ 4 ┆ Apple ┆ 3 ┆ 4 ┆ Apple │
| └─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
|
| Get the bottom 2 rows by column `a` in each group.
|
| >>> (
| ... df.group_by("c", maintain_order=True)
| ... .agg(pl.all().bottom_k_by("a", 2))
| ... .explode(pl.all().exclude("c"))
| ... )
| shape: (5, 3)
| ┌────────┬─────┬─────┐
| │ c ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ Apple ┆ 1 ┆ 6 │
| │ Apple ┆ 3 ┆ 4 │
| │ Orange ┆ 2 ┆ 5 │
| │ Banana ┆ 5 ┆ 2 │
| │ Banana ┆ 6 ┆ 1 │
| └────────┴─────┴─────┘
|
| cast(self, dtype: 'PolarsDataType | pl.DataTypeExpr | type[Any]', *, strict: 'bool' = True, wrap_numerical: 'bool' = False) -> 'Expr'
| Cast between data types.
|
| Parameters
| ----------
| dtype
| DataType to cast to.
| strict
| Raise if cast is invalid on rows after predicates are pushed down.
| If `False`, invalid casts will produce null values.
| wrap_numerical
| If True numeric casts wrap overflowing values instead of
| marking the cast as invalid.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["4", "5", "6"],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("a").cast(pl.Float64),
| ... pl.col("b").cast(pl.Int32),
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ i32 │
| ╞═════╪═════╡
| │ 1.0 ┆ 4 │
| │ 2.0 ┆ 5 │
| │ 3.0 ┆ 6 │
| └─────┴─────┘
|
| cbrt(self) -> 'Expr'
| Compute the cube root of the elements.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").cbrt())
| shape: (3, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 1.259921 │
| │ 1.587401 │
| └──────────┘
|
| ceil(self) -> 'Expr'
| Rounds up to the nearest integer value.
|
| Only works on floating point Series.
|
| See Also
| --------
| floor : Round down to the nearest integer.
| round : Round to the nearest integer.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.3, 0.5, 1.0, 1.1]})
| >>> df.select(pl.col("a").ceil())
| shape: (4, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| │ 1.0 │
| │ 1.0 │
| │ 2.0 │
| └─────┘
|
| clip(self, lower_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None, upper_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None) -> 'Expr'
| Set values outside the given boundaries to the boundary value.
|
| Parameters
| ----------
| lower_bound
| Lower bound. Accepts expression input. Non-expression inputs are
| parsed as literals. Strings are parsed as column names.
| upper_bound
| Upper bound. Accepts expression input. Non-expression inputs are
| parsed as literals. Strings are parsed as column names.
|
| See Also
| --------
| when
|
| Notes
| -----
| This method only works for numeric and temporal columns. To clip other data
| types, consider writing a `when-then-otherwise` expression. See :func:`when`.
|
| Examples
| --------
| Specifying both a lower and upper bound:
|
| >>> df = pl.DataFrame({"a": [-50, 5, 50, None]})
| >>> df.with_columns(clip=pl.col("a").clip(1, 10))
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ clip │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ -50 ┆ 1 │
| │ 5 ┆ 5 │
| │ 50 ┆ 10 │
| │ null ┆ null │
| └──────┴──────┘
|
| Specifying only a single bound:
|
| >>> df.with_columns(clip=pl.col("a").clip(upper_bound=10))
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ clip │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ -50 ┆ -50 │
| │ 5 ┆ 5 │
| │ 50 ┆ 10 │
| │ null ┆ null │
| └──────┴──────┘
|
| Using columns as bounds:
|
| >>> df = pl.DataFrame(
| ... {"a": [-50, 5, 50, None], "low": [10, 1, 0, 0], "up": [20, 4, 3, 2]}
| ... )
| >>> df.with_columns(clip=pl.col("a").clip("low", "up"))
| shape: (4, 4)
| ┌──────┬─────┬─────┬──────┐
| │ a ┆ low ┆ up ┆ clip │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞══════╪═════╪═════╪══════╡
| │ -50 ┆ 10 ┆ 20 ┆ 10 │
| │ 5 ┆ 1 ┆ 4 ┆ 4 │
| │ 50 ┆ 0 ┆ 3 ┆ 3 │
| │ null ┆ 0 ┆ 2 ┆ null │
| └──────┴─────┴─────┴──────┘
|
| cos(self) -> 'Expr'
| Compute the element-wise value for the cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 2]})
| >>> df.select(pl.col("a").cos())
| shape: (2, 1)
| ┌────────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════════╡
| │ 1.0 │
| │ 6.1232e-17 │
| └────────────┘
| >>> df = pl.DataFrame({"a": [0.0, 90]})
| >>> df.select(pl.col("a").radians().cos())
| shape: (2, 1)
| ┌────────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════════╡
| │ 1.0 │
| │ 6.1232e-17 │
| └────────────┘
|
| cosh(self) -> 'Expr'
| Compute the element-wise value for the hyperbolic cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").cosh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.543081 │
| └──────────┘
|
| cot(self) -> 'Expr'
| Compute the element-wise value for the cotangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 4]})
| >>> df.select(pl.col("a").cot())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ inf │
| │ 1.0 │
| └─────┘
| >>> df = pl.DataFrame({"a": [0.0, 45]})
| >>> df.select(pl.col("a").radians().cot())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ inf │
| │ 1.0 │
| └─────┘
|
| count(self) -> 'Expr'
| Return the number of non-null elements in the column.
|
| Returns
| -------
| Expr
| Expression of data type :class:`UInt32`.
|
| See Also
| --------
| len
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [None, 4, 4]})
| >>> df.select(pl.all().count())
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 3 ┆ 2 │
| └─────┴─────┘
|
| cum_count(self, *, reverse: 'bool' = False) -> 'Expr'
| Return the cumulative count of the non-null values in the column.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": ["x", "k", None, "d"]})
| >>> df.with_columns(
| ... pl.col("a").cum_count().alias("cum_count"),
| ... pl.col("a").cum_count(reverse=True).alias("cum_count_reverse"),
| ... )
| shape: (4, 3)
| ┌──────┬───────────┬───────────────────┐
| │ a ┆ cum_count ┆ cum_count_reverse │
| │ --- ┆ --- ┆ --- │
| │ str ┆ u32 ┆ u32 │
| ╞══════╪═══════════╪═══════════════════╡
| │ x ┆ 1 ┆ 3 │
| │ k ┆ 2 ┆ 2 │
| │ null ┆ 2 ┆ 1 │
| │ d ┆ 3 ┆ 1 │
| └──────┴───────────┴───────────────────┘
|
| cum_max(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative max computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 2]})
| >>> df.with_columns(
| ... pl.col("a").cum_max().alias("cum_max"),
| ... pl.col("a").cum_max(reverse=True).alias("cum_max_reverse"),
| ... )
| shape: (3, 3)
| ┌─────┬─────────┬─────────────────┐
| │ a ┆ cum_max ┆ cum_max_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════════╪═════════════════╡
| │ 1 ┆ 1 ┆ 3 │
| │ 3 ┆ 3 ┆ 3 │
| │ 2 ┆ 3 ┆ 2 │
| └─────┴─────────┴─────────────────┘
|
|
| Null values are excluded, but can also be filled by calling
| `fill_null(strategy="forward")`.
|
| >>> df = pl.DataFrame({"values": [None, 10, None, 8, 9, None, 16, None]})
| >>> df.with_columns(
| ... pl.col("values").cum_max().alias("cum_max"),
| ... pl.col("values")
| ... .cum_max()
| ... .fill_null(strategy="forward")
| ... .alias("cum_max_all_filled"),
| ... )
| shape: (8, 3)
| ┌────────┬─────────┬────────────────────┐
| │ values ┆ cum_max ┆ cum_max_all_filled │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞════════╪═════════╪════════════════════╡
| │ null ┆ null ┆ null │
| │ 10 ┆ 10 ┆ 10 │
| │ null ┆ null ┆ 10 │
| │ 8 ┆ 10 ┆ 10 │
| │ 9 ┆ 10 ┆ 10 │
| │ null ┆ null ┆ 10 │
| │ 16 ┆ 16 ┆ 16 │
| │ null ┆ null ┆ 16 │
| └────────┴─────────┴────────────────────┘
|
| cum_min(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative min computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [3, 1, 2]})
| >>> df.with_columns(
| ... pl.col("a").cum_min().alias("cum_min"),
| ... pl.col("a").cum_min(reverse=True).alias("cum_min_reverse"),
| ... )
| shape: (3, 3)
| ┌─────┬─────────┬─────────────────┐
| │ a ┆ cum_min ┆ cum_min_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════════╪═════════════════╡
| │ 3 ┆ 3 ┆ 1 │
| │ 1 ┆ 1 ┆ 1 │
| │ 2 ┆ 1 ┆ 2 │
| └─────┴─────────┴─────────────────┘
|
| cum_prod(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative product computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.with_columns(
| ... pl.col("a").cum_prod().alias("cum_prod"),
| ... pl.col("a").cum_prod(reverse=True).alias("cum_prod_reverse"),
| ... )
| shape: (4, 3)
| ┌─────┬──────────┬──────────────────┐
| │ a ┆ cum_prod ┆ cum_prod_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪══════════╪══════════════════╡
| │ 1 ┆ 1 ┆ 24 │
| │ 2 ┆ 2 ┆ 24 │
| │ 3 ┆ 6 ┆ 12 │
| │ 4 ┆ 24 ┆ 4 │
| └─────┴──────────┴──────────────────┘
|
| cum_sum(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative sum computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.with_columns(
| ... pl.col("a").cum_sum().alias("cum_sum"),
| ... pl.col("a").cum_sum(reverse=True).alias("cum_sum_reverse"),
| ... )
| shape: (4, 3)
| ┌─────┬─────────┬─────────────────┐
| │ a ┆ cum_sum ┆ cum_sum_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════════╪═════════════════╡
| │ 1 ┆ 1 ┆ 10 │
| │ 2 ┆ 3 ┆ 9 │
| │ 3 ┆ 6 ┆ 7 │
| │ 4 ┆ 10 ┆ 4 │
| └─────┴─────────┴─────────────────┘
|
| Null values are excluded, but can also be filled by calling
| `fill_null(strategy="forward")`.
|
| >>> df = pl.DataFrame({"values": [None, 10, None, 8, 9, None, 16, None]})
| >>> df.with_columns(
| ... pl.col("values").cum_sum().alias("value_cum_sum"),
| ... pl.col("values")
| ... .cum_sum()
| ... .fill_null(strategy="forward")
| ... .alias("value_cum_sum_all_filled"),
| ... )
| shape: (8, 3)
| ┌────────┬───────────────┬──────────────────────────┐
| │ values ┆ value_cum_sum ┆ value_cum_sum_all_filled │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞════════╪═══════════════╪══════════════════════════╡
| │ null ┆ null ┆ null │
| │ 10 ┆ 10 ┆ 10 │
| │ null ┆ null ┆ 10 │
| │ 8 ┆ 18 ┆ 18 │
| │ 9 ┆ 27 ┆ 27 │
| │ null ┆ null ┆ 27 │
| │ 16 ┆ 43 ┆ 43 │
| │ null ┆ null ┆ 43 │
| └────────┴───────────────┴──────────────────────────┘
|
| cumulative_eval(self, expr: 'Expr', *, min_samples: 'int' = 1) -> 'Expr'
| Run an expression over a sliding window that increases `1` slot every iteration.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| expr
| Expression to evaluate
| min_samples
| Number of valid values there should be in the window before the expression
| is evaluated. valid values = `length - null_count`
|
| Warnings
| --------
| This can be really slow as it can have `O(n^2)` complexity. Don't use this
| for operations that visit all elements.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1, 2, 3, 4, 5]})
| >>> df.select(
| ... [
| ... pl.col("values").cumulative_eval(
| ... pl.element().first() - pl.element().last() ** 2
| ... )
| ... ]
| ... )
| shape: (5, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 0 │
| │ -3 │
| │ -8 │
| │ -15 │
| │ -24 │
| └────────┘
|
| cut(self, breaks: 'Sequence[float]', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, include_breaks: 'bool' = False) -> 'Expr'
| Bin continuous values into discrete categories.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| breaks
| List of unique cut points.
| labels
| Names of the categories. The number of labels must be equal to the number
| of cut points plus one.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from an
| :class:`Enum` to a :class:`Struct`.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Enum` if `include_breaks` is set to
| `False` (default), otherwise an expression of data type :class:`Struct`.
|
| See Also
| --------
| qcut
|
| Examples
| --------
| Divide a column into three categories.
|
| >>> df = pl.DataFrame({"foo": [-2, -1, 0, 1, 2]})
| >>> df.with_columns(
| ... pl.col("foo").cut([-1, 1], labels=["a", "b", "c"]).alias("cut")
| ... )
| shape: (5, 2)
| ┌─────┬──────┐
| │ foo ┆ cut │
| │ --- ┆ --- │
| │ i64 ┆ enum │
| ╞═════╪══════╡
| │ -2 ┆ a │
| │ -1 ┆ a │
| │ 0 ┆ b │
| │ 1 ┆ b │
| │ 2 ┆ c │
| └─────┴──────┘
|
| Add both the category and the breakpoint.
|
| >>> df.with_columns(
| ... pl.col("foo").cut([-1, 1], include_breaks=True).alias("cut")
| ... ).unnest("cut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ enum │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| degrees(self) -> 'Expr'
| Convert from radians to degrees.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> import math
| >>> df = pl.DataFrame({"a": [x * math.pi for x in range(-4, 5)]})
| >>> df.select(pl.col("a").degrees())
| shape: (9, 1)
| ┌────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════╡
| │ -720.0 │
| │ -540.0 │
| │ -360.0 │
| │ -180.0 │
| │ 0.0 │
| │ 180.0 │
| │ 360.0 │
| │ 540.0 │
| │ 720.0 │
| └────────┘
|
| diff(self, n: 'int | IntoExpr' = 1, null_behavior: 'NullBehavior' = 'ignore') -> 'Expr'
| Calculate the first discrete difference between shifted items.
|
| Parameters
| ----------
| n
| Number of slots to shift.
| null_behavior : {'ignore', 'drop'}
| How to handle null values.
|
| Examples
| --------
| >>> df = pl.DataFrame({"int": [20, 10, 30, 25, 35]})
| >>> df.with_columns(change=pl.col("int").diff())
| shape: (5, 2)
| ┌─────┬────────┐
| │ int ┆ change │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪════════╡
| │ 20 ┆ null │
| │ 10 ┆ -10 │
| │ 30 ┆ 20 │
| │ 25 ┆ -5 │
| │ 35 ┆ 10 │
| └─────┴────────┘
|
| >>> df.with_columns(change=pl.col("int").diff(n=2))
| shape: (5, 2)
| ┌─────┬────────┐
| │ int ┆ change │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪════════╡
| │ 20 ┆ null │
| │ 10 ┆ null │
| │ 30 ┆ 10 │
| │ 25 ┆ 15 │
| │ 35 ┆ 5 │
| └─────┴────────┘
|
| >>> df.select(pl.col("int").diff(n=2, null_behavior="drop").alias("diff"))
| shape: (3, 1)
| ┌──────┐
| │ diff │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 10 │
| │ 15 │
| │ 5 │
| └──────┘
|
| dot(self, other: 'Expr | str_') -> 'Expr'
| Compute the dot/inner product between two Expressions.
|
| Parameters
| ----------
| other
| Expression to compute dot product with.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 5], "b": [2, 4, 6]})
| >>> df.select(pl.col("a").dot(pl.col("b")))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 44 │
| └─────┘
|
| drop_nans(self) -> 'Expr'
| Drop all floating point NaN values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nulls
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To drop null values, use :func:`drop_nulls`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, None, 3.0, float("nan")]})
| >>> df.select(pl.col("a").drop_nans())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 1.0 │
| │ null │
| │ 3.0 │
| └──────┘
|
| drop_nulls(self) -> 'Expr'
| Drop all null values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nans
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To drop NaN values, use :func:`drop_nans`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, None, 3.0, float("nan")]})
| >>> df.select(pl.col("a").drop_nulls())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| │ 3.0 │
| │ NaN │
| └─────┘
|
| entropy(self, base: 'float' = 2.718281828459045, *, normalize: 'bool' = True) -> 'Expr'
| Computes the entropy.
|
| Uses the formula `-sum(pk * log(pk))` where `pk` are discrete probabilities.
|
| Parameters
| ----------
| base
| Given base, defaults to `e`
| normalize
| Normalize pk if it doesn't sum to 1.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").entropy(base=2))
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.459148 │
| └──────────┘
| >>> df.select(pl.col("a").entropy(base=2, normalize=False))
| shape: (1, 1)
| ┌───────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═══════════╡
| │ -6.754888 │
| └───────────┘
|
| eq(self, other: 'Any') -> 'Expr'
| Method equivalent of equality operator `expr == other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0],
| ... "y": [2.0, 2.0, float("nan"), 4.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").eq(pl.col("y")).alias("x == y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x == y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 1.0 ┆ 2.0 ┆ false │
| │ 2.0 ┆ 2.0 ┆ true │
| │ NaN ┆ NaN ┆ true │
| │ 4.0 ┆ 4.0 ┆ true │
| └─────┴─────┴────────┘
|
| eq_missing(self, other: 'Any') -> 'Expr'
| Method equivalent of equality operator `expr == other` where `None == None`.
|
| This differs from default `eq` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0, None, None],
| ... "y": [2.0, 2.0, float("nan"), 4.0, 5.0, None],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").eq(pl.col("y")).alias("x eq y"),
| ... pl.col("x").eq_missing(pl.col("y")).alias("x eq_missing y"),
| ... )
| shape: (6, 4)
| ┌──────┬──────┬────────┬────────────────┐
| │ x ┆ y ┆ x eq y ┆ x eq_missing y │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪══════╪════════╪════════════════╡
| │ 1.0 ┆ 2.0 ┆ false ┆ false │
| │ 2.0 ┆ 2.0 ┆ true ┆ true │
| │ NaN ┆ NaN ┆ true ┆ true │
| │ 4.0 ┆ 4.0 ┆ true ┆ true │
| │ null ┆ 5.0 ┆ null ┆ false │
| │ null ┆ null ┆ null ┆ true │
| └──────┴──────┴────────┴────────────────┘
|
| ewm_mean(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving average.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_mean(com=1, ignore_nulls=False))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 1.666667 │
| │ 2.428571 │
| └──────────┘
|
| ewm_mean_by(self, by: 'str_ | IntoExpr', *, half_life: 'str_ | timedelta') -> 'Expr'
| Compute time-based exponentially weighted moving average.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMA is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \alpha_i &= 1 - \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= \alpha_i x_i + (1 - \alpha_i) y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Times to calculate average by. Should be ``DateTime``, ``Date``, ``UInt64``,
| ``UInt32``, ``Int64``, or ``Int32`` data type.
| half_life
| Unit over which observation decays to half its value.
|
| Can be created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 day)
| - 1w (1 week)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| Note that `half_life` is treated as a constant duration - calendar
| durations such as months (or even days in the time-zone-aware case)
| are not supported, please express your duration in an approximately
| equivalent number of hours (e.g. '370h' instead of '1mo').
|
| Returns
| -------
| Expr
| :class:`.Float16` if input is `Float16`, class:`.Float32` if input is
| `Float32`, otherwise class:`.Float64`.
|
| Examples
| --------
| >>> from datetime import date, timedelta
| >>> df = pl.DataFrame(
| ... {
| ... "values": [0, 1, 2, None, 4],
| ... "times": [
| ... date(2020, 1, 1),
| ... date(2020, 1, 3),
| ... date(2020, 1, 10),
| ... date(2020, 1, 15),
| ... date(2020, 1, 17),
| ... ],
| ... }
| ... ).sort("times")
| >>> df.with_columns(
| ... result=pl.col("values").ewm_mean_by("times", half_life="4d"),
| ... )
| shape: (5, 3)
| ┌────────┬────────────┬──────────┐
| │ values ┆ times ┆ result │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ date ┆ f64 │
| ╞════════╪════════════╪══════════╡
| │ 0 ┆ 2020-01-01 ┆ 0.0 │
| │ 1 ┆ 2020-01-03 ┆ 0.292893 │
| │ 2 ┆ 2020-01-10 ┆ 1.492474 │
| │ null ┆ 2020-01-15 ┆ null │
| │ 4 ┆ 2020-01-17 ┆ 3.254508 │
| └────────┴────────────┴──────────┘
|
| ewm_std(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving standard deviation.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_std(com=1, ignore_nulls=False))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ null │
| │ 0.707107 │
| │ 0.963624 │
| └──────────┘
|
| ewm_sum(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_sum(alpha=0.5))
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 1.0 │
| │ 2.5 │
| │ 4.25 │
| └──────┘
|
| ewm_sum_by(self, by: 'str_ | IntoExpr', *, half_life: 'str_ | timedelta') -> 'Expr'
| Compute time-based exponentially-weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMS is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \lambda_i &= \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= x_i + \lambda_i y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Column to use as reference for the time decay.
| half_life
| Half-life of the exponential decay.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "values": [1, 2, 3, 4, 5],
| ... "times": [0, 1, 2, 5, 6],
| ... }
| ... )
| >>> df.select(
| ... pl.col("values").ewm_sum_by("times", half_life="1i"),
| ... )
| shape: (5, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 2.5 │
| │ 4.25 │
| │ 4.53125 │
| │ 7.265625 │
| └──────────┘
|
| ewm_var(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving variance.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_var(com=1, ignore_nulls=False))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ null │
| │ 0.5 │
| │ 0.928571 │
| └──────────┘
|
| exclude(self, columns: 'str_ | PolarsDataType | Collection[str_ | PolarsDataType]', *more_columns: 'str_ | PolarsDataType') -> 'Expr'
| Exclude columns from a multi-column expression.
|
| Only works after a wildcard or regex column selection, and you cannot provide
| both string column names *and* dtypes (you may prefer to use selectors instead).
|
| Parameters
| ----------
| columns
| The name or datatype of the column(s) to exclude. Accepts regular expression
| input. Regular expressions should start with `^` and end with `$`.
| *more_columns
| Additional names or datatypes of columns to exclude, specified as positional
| arguments.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "aa": [1, 2, 3],
| ... "ba": ["a", "b", None],
| ... "cc": [None, 2.5, 1.5],
| ... }
| ... )
| >>> df
| shape: (3, 3)
| ┌─────┬──────┬──────┐
| │ aa ┆ ba ┆ cc │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ f64 │
| ╞═════╪══════╪══════╡
| │ 1 ┆ a ┆ null │
| │ 2 ┆ b ┆ 2.5 │
| │ 3 ┆ null ┆ 1.5 │
| └─────┴──────┴──────┘
|
| Exclude by column name(s):
|
| >>> df.select(pl.all().exclude("ba"))
| shape: (3, 2)
| ┌─────┬──────┐
| │ aa ┆ cc │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ null │
| │ 2 ┆ 2.5 │
| │ 3 ┆ 1.5 │
| └─────┴──────┘
|
| Exclude by regex, e.g. removing all columns whose names end with the letter "a":
|
| >>> df.select(pl.all().exclude("^.*a$"))
| shape: (3, 1)
| ┌──────┐
| │ cc │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ 2.5 │
| │ 1.5 │
| └──────┘
|
| Exclude by dtype(s), e.g. removing all columns of type Int64 or Float64:
|
| >>> df.select(pl.all().exclude([pl.Int64, pl.Float64]))
| shape: (3, 1)
| ┌──────┐
| │ ba │
| │ --- │
| │ str │
| ╞══════╡
| │ a │
| │ b │
| │ null │
| └──────┘
|
| exp(self) -> 'Expr'
| Compute the exponential, element-wise.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").exp())
| shape: (3, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 2.718282 │
| │ 7.389056 │
| │ 54.59815 │
| └──────────┘
|
| explode(self, *, empty_as_null: 'bool' = <object object at 0x00000280D5528F20>, keep_nulls: 'bool' = True) -> 'Expr'
| Explode a list expression.
|
| This means that every item is expanded to a new row.
|
| Parameters
| ----------
| empty_as_null
| Explode an empty list/array into a `null`.
| keep_nulls
| Explode a `null` list/array into a `null`.
|
| Returns
| -------
| Expr
| Expression with the data type of the list elements.
|
| See Also
| --------
| Expr.list.explode : Explode a list column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["a", "b"],
| ... "values": [
| ... [1, 2],
| ... [3, 4],
| ... ],
| ... }
| ... )
| >>> df.select(pl.col("values").explode(empty_as_null=False))
| shape: (4, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 1 │
| │ 2 │
| │ 3 │
| │ 4 │
| └────────┘
|
| extend_constant(self, value: 'IntoExpr', n: 'int | IntoExprColumn') -> 'Expr'
| Extremely fast method for extending the Series with 'n' copies of a value.
|
| Parameters
| ----------
| value
| A constant literal value or a unit expression with which to extend the
| expression result Series; can pass None to extend with nulls.
| n
| The number of additional values that will be added.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1, 2, 3]})
| >>> df.select((pl.col("values") - 1).extend_constant(99, n=2))
| shape: (5, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 0 │
| │ 1 │
| │ 2 │
| │ 99 │
| │ 99 │
| └────────┘
|
| fill_nan(self, value: 'int | float | Expr | None') -> 'Expr'
| Fill floating point NaN value with a fill value.
|
| Parameters
| ----------
| value
| Value used to fill NaN values.
|
| See Also
| --------
| fill_null
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To fill null values, use :func:`fill_null`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1.0, None, float("nan")],
| ... "b": [4.0, float("nan"), 6],
| ... }
| ... )
| >>> df.with_columns(pl.col("b").fill_nan(0))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪═════╡
| │ 1.0 ┆ 4.0 │
| │ null ┆ 0.0 │
| │ NaN ┆ 6.0 │
| └──────┴─────┘
|
| fill_null(self, value: 'Any | Expr | None' = None, strategy: 'FillNullStrategy | None' = None, limit: 'int | None' = None) -> 'Expr'
| Fill null values using the specified value or strategy.
|
| To interpolate over null values see interpolate.
| See the examples below to fill nulls with an expression.
|
| Parameters
| ----------
| value
| Value used to fill null values.
| strategy : {None, 'forward', 'backward', 'min', 'max', 'mean', 'zero', 'one'}
| Strategy used to fill null values.
| limit
| Number of consecutive null values to fill when using the 'forward' or
| 'backward' strategy.
|
| See Also
| --------
| backward_fill
| fill_nan
| forward_fill
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To fill NaN values, use :func:`fill_nan`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": [4, None, 6],
| ... }
| ... )
| >>> df.with_columns(pl.col("b").fill_null(strategy="zero"))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 0 │
| │ null ┆ 6 │
| └──────┴─────┘
| >>> df.with_columns(pl.col("b").fill_null(99))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 99 │
| │ null ┆ 6 │
| └──────┴─────┘
| >>> df.with_columns(pl.col("b").fill_null(strategy="forward"))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 4 │
| │ null ┆ 6 │
| └──────┴─────┘
| >>> df.with_columns(pl.col("b").fill_null(pl.col("b").median()))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞══════╪═════╡
| │ 1 ┆ 4.0 │
| │ 2 ┆ 5.0 │
| │ null ┆ 6.0 │
| └──────┴─────┘
| >>> df.with_columns(pl.all().fill_null(pl.all().median()))
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 1.0 ┆ 4.0 │
| │ 2.0 ┆ 5.0 │
| │ 1.5 ┆ 6.0 │
| └─────┴─────┘
|
| filter(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn]', **constraints: 'Any') -> 'Expr'
| Filter the expression based on one or more predicate expressions.
|
| The original order of the remaining elements is preserved.
|
| Elements where the filter does not evaluate to True are discarded, including
| nulls.
|
| Mostly useful in an aggregation context. If you want to filter on a DataFrame
| level, use `LazyFrame.filter`.
|
| Parameters
| ----------
| predicates
| Expression(s) that evaluates to a boolean Series.
| constraints
| Column filters; use `name = value` to filter columns by the supplied value.
| Each constraint will behave the same as `pl.col(name).eq(value)`, and
| be implicitly joined with the other filter conditions using `&`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group_col": ["g1", "g1", "g2"],
| ... "b": [1, 2, 3],
| ... }
| ... )
| >>> df.group_by("group_col").agg(
| ... lt=pl.col("b").filter(pl.col("b") < 2).sum(),
| ... gte=pl.col("b").filter(pl.col("b") >= 2).sum(),
| ... ).sort("group_col")
| shape: (2, 3)
| ┌───────────┬─────┬─────┐
| │ group_col ┆ lt ┆ gte │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════════╪═════╪═════╡
| │ g1 ┆ 1 ┆ 2 │
| │ g2 ┆ 0 ┆ 3 │
| └───────────┴─────┴─────┘
|
| Filter expressions can also take constraints as keyword arguments.
|
| >>> df = pl.DataFrame(
| ... {
| ... "key": ["a", "a", "a", "a", "b", "b", "b", "b", "b"],
| ... "n": [1, 2, 2, 3, 1, 3, 3, 2, 3],
| ... },
| ... )
| >>> df.group_by("key").agg(
| ... n_1=pl.col("n").filter(n=1).sum(),
| ... n_2=pl.col("n").filter(n=2).sum(),
| ... n_3=pl.col("n").filter(n=3).sum(),
| ... ).sort(by="key")
| shape: (2, 4)
| ┌─────┬─────┬─────┬─────┐
| │ key ┆ n_1 ┆ n_2 ┆ n_3 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╡
| │ a ┆ 1 ┆ 4 ┆ 3 │
| │ b ┆ 1 ┆ 2 ┆ 9 │
| └─────┴─────┴─────┴─────┘
|
| first(self, *, ignore_nulls: 'bool' = False) -> 'Expr'
| Get the first value.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the first non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [None, 1, 2]})
| >>> df.select(pl.col("a").first())
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| └──────┘
| >>> df.select(pl.col("a").first(ignore_nulls=True))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| └─────┘
|
| flatten(self) -> 'Expr'
| Flatten a list or string column.
|
| Alias for :func:`Expr.list.explode`.
|
| .. deprecated:: 1.38
| `Expr.flatten()` is deprecated and will be removed in version 2.0.
| Use `Expr.list.explode(keep_nulls=False, empty_as_null=False)` instead,
| which provides the behavior you likely expect.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["a", "b", "b"],
| ... "values": [[1, 2], [2, 3], [4]],
| ... }
| ... )
| >>> df.group_by("group").agg(pl.col("values").flatten()) # doctest: +SKIP
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ values │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪═══════════╡
| │ a ┆ [1, 2] │
| │ b ┆ [2, 3, 4] │
| └───────┴───────────┘
|
| floor(self) -> 'Expr'
| Rounds down to the nearest integer value.
|
| Only works on floating point Series.
|
| See Also
| --------
| ceil : Round up to the nearest integer value.
| round : Round to the nearest integer.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.3, 0.5, 1.0, 1.1]})
| >>> df.select(pl.col("a").floor())
| shape: (4, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 0.0 │
| │ 1.0 │
| │ 1.0 │
| └─────┘
|
| floordiv(self, other: 'Any') -> 'Expr'
| Method equivalent of integer division operator `expr // other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| See Also
| --------
| truediv
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 3, 4, 5]})
| >>> df.with_columns(
| ... pl.col("x").truediv(2).alias("x/2"),
| ... pl.col("x").floordiv(2).alias("x//2"),
| ... )
| shape: (5, 3)
| ┌─────┬─────┬──────┐
| │ x ┆ x/2 ┆ x//2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ i64 │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 0.5 ┆ 0 │
| │ 2 ┆ 1.0 ┆ 1 │
| │ 3 ┆ 1.5 ┆ 1 │
| │ 4 ┆ 2.0 ┆ 2 │
| │ 5 ┆ 2.5 ┆ 2 │
| └─────┴─────┴──────┘
|
| Note that Polars' `floordiv` is subtly different from Python's floor division.
| For example, consider 6.0 floor-divided by 0.1.
| Python gives:
|
| >>> 6.0 // 0.1
| 59.0
|
| because `0.1` is not represented internally as that exact value,
| but a slightly larger value.
| So the result of the division is slightly less than 60,
| meaning the flooring operation returns 59.0.
|
| Polars instead first does the floating-point division,
| resulting in a floating-point value of 60.0,
| and then performs the flooring operation using :any:`floor`:
|
| >>> df = pl.DataFrame({"x": [6.0, 6.03]})
| >>> df.with_columns(
| ... pl.col("x").truediv(0.1).alias("x/0.1"),
| ... ).with_columns(
| ... pl.col("x/0.1").floor().alias("x/0.1 floor"),
| ... )
| shape: (2, 3)
| ┌──────┬───────┬─────────────┐
| │ x ┆ x/0.1 ┆ x/0.1 floor │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞══════╪═══════╪═════════════╡
| │ 6.0 ┆ 60.0 ┆ 60.0 │
| │ 6.03 ┆ 60.3 ┆ 60.0 │
| └──────┴───────┴─────────────┘
|
| yielding the more intuitive result 60.0.
| The row with x = 6.03 is included to demonstrate
| the effect of the flooring operation.
|
| `floordiv` combines those two steps
| to give the same result with one expression:
|
| >>> df.with_columns(
| ... pl.col("x").floordiv(0.1).alias("x//0.1"),
| ... )
| shape: (2, 2)
| ┌──────┬────────┐
| │ x ┆ x//0.1 │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪════════╡
| │ 6.0 ┆ 60.0 │
| │ 6.03 ┆ 60.0 │
| └──────┴────────┘
|
| forward_fill(self, limit: 'int | None' = None) -> 'Expr'
| Fill missing values with the last non-null value.
|
| This is an alias of `.fill_null(strategy="forward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to forward fill.
|
| See Also
| --------
| backward_fill
| fill_null
| shift
|
| gather(self, indices: 'int | Sequence[int] | IntoExpr | Series | np.ndarray[Any, Any]', *, null_on_oob: 'bool' = False) -> 'Expr'
| Take values by index.
|
| Parameters
| ----------
| indices
| An expression that leads to a UInt32 dtyped Series.
| null_on_oob
| Behavior if an index is out of bounds:
|
| - True -> set the result to null
| - False -> raise an error
|
| Returns
| -------
| Expr
| Expression of the same data type.
|
| See Also
| --------
| Expr.get : Take a single value
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": [
| ... "one",
| ... "one",
| ... "one",
| ... "two",
| ... "two",
| ... "two",
| ... ],
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.group_by("group", maintain_order=True).agg(
| ... pl.col("value").gather([2, 1])
| ... )
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪═══════════╡
| │ one ┆ [2, 98] │
| │ two ┆ [4, 99] │
| └───────┴───────────┘
|
| Use `null_on_oob=True` to return null for out-of-bounds indices.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").gather([0, 1, 10], null_on_oob=True))
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 1 │
| │ 2 │
| │ null │
| └──────┘
|
| gather_every(self, n: 'int', offset: 'int' = 0) -> 'Expr'
| Take every nth value in the Series and return as a new Series.
|
| Parameters
| ----------
| n
| Gather every *n*-th row.
| offset
| Starting index.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7, 8, 9]})
| >>> df.select(pl.col("foo").gather_every(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 4 │
| │ 7 │
| └─────┘
|
| >>> df.select(pl.col("foo").gather_every(3, offset=1))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 5 │
| │ 8 │
| └─────┘
|
| ge(self, other: 'Any') -> 'Expr'
| Method equivalent of "greater than or equal" operator `expr >= other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5.0, 4.0, float("nan"), 2.0],
| ... "y": [5.0, 3.0, float("nan"), 1.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").ge(pl.col("y")).alias("x >= y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x >= y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 5.0 ┆ 5.0 ┆ true │
| │ 4.0 ┆ 3.0 ┆ true │
| │ NaN ┆ NaN ┆ true │
| │ 2.0 ┆ 1.0 ┆ true │
| └─────┴─────┴────────┘
|
| get(self, index: 'int | Expr', *, null_on_oob: 'bool' = False) -> 'Expr'
| Return a single value by index.
|
| Parameters
| ----------
| index
| An expression that evaluates to an integer.
| Negative indexing is supported.
|
| null_on_oob
| Behavior if an index is out of bounds:
|
| - True -> set the result to null
| - False -> raise an error
|
| Returns
| -------
| Expr
| Expression of the same data type.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": [
| ... "one",
| ... "one",
| ... "one",
| ... "two",
| ... "two",
| ... "two",
| ... ],
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.group_by("group", maintain_order=True).agg(pl.col("value").get(1))
| shape: (2, 2)
| ┌───────┬───────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═══════╡
| │ one ┆ 98 │
| │ two ┆ 99 │
| └───────┴───────┘
|
| gt(self, other: 'Any') -> 'Expr'
| Method equivalent of "greater than" operator `expr > other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5.0, 4.0, float("nan"), 2.0],
| ... "y": [5.0, 3.0, float("nan"), 1.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").gt(pl.col("y")).alias("x > y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────┐
| │ x ┆ y ┆ x > y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪═══════╡
| │ 5.0 ┆ 5.0 ┆ false │
| │ 4.0 ┆ 3.0 ┆ true │
| │ NaN ┆ NaN ┆ false │
| │ 2.0 ┆ 1.0 ┆ true │
| └─────┴─────┴───────┘
|
| has_nulls(self) -> 'Expr'
| Check whether the expression contains one or more null values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 1, None],
| ... "b": [10, None, 300],
| ... "c": [350, 650, 850],
| ... }
| ... )
| >>> df.select(pl.all().has_nulls())
| shape: (1, 3)
| ┌──────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪══════╪═══════╡
| │ true ┆ true ┆ false │
| └──────┴──────┴───────┘
|
| hash(self, seed: 'int' = 0, seed_1: 'int | None' = None, seed_2: 'int | None' = None, seed_3: 'int | None' = None) -> 'Expr'
| Hash the elements in the selection.
|
| The hash value is of type `UInt64`.
|
| Parameters
| ----------
| seed
| Random seed parameter. Defaults to 0.
| seed_1
| Random seed parameter. Defaults to `seed` if not set.
| seed_2
| Random seed parameter. Defaults to `seed` if not set.
| seed_3
| Random seed parameter. Defaults to `seed` if not set.
|
| Notes
| -----
| This implementation of `hash` does not guarantee stable results
| across different Polars versions. Its stability is only guaranteed within a
| single version.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": ["x", None, "z"],
| ... }
| ... )
| >>> df.with_columns(pl.all().hash(10, 20, 30, 40)) # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌──────────────────────┬──────────────────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u64 ┆ u64 │
| ╞══════════════════════╪══════════════════════╡
| │ 9774092659964970114 ┆ 13614470193936745724 │
| │ 1101441246220388612 ┆ 11638928888656214026 │
| │ 11638928888656214026 ┆ 13382926553367784577 │
| └──────────────────────┴──────────────────────┘
|
| head(self, n: 'int | Expr' = 10) -> 'Expr'
| Get the first `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7]})
| >>> df.select(pl.col("foo").head(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| hist(self, bins: 'IntoExpr | None' = None, *, bin_count: 'int | None' = None, include_category: 'bool' = False, include_breakpoint: 'bool' = False) -> 'Expr'
| Bin values into buckets and count their occurrences.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| bins
| Bin edges. If None given, we determine the edges based on the data.
| bin_count
| If `bins` is not provided, `bin_count` uniform bins are created that fully
| encompass the data.
| include_breakpoint
| Include a column that indicates the upper breakpoint.
| include_category
| Include a column that shows the intervals as categories.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 8, 8, 2, 1, 3]})
| >>> df.select(pl.col("a").hist(bins=[1, 2, 3]))
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 3 │
| │ 2 │
| └─────┘
| >>> df.select(
| ... pl.col("a").hist(
| ... bins=[1, 2, 3], include_breakpoint=True, include_category=True
| ... )
| ... )
| shape: (2, 1)
| ┌──────────────────────┐
| │ a │
| │ --- │
| │ struct[3] │
| ╞══════════════════════╡
| │ {2.0,"[1.0, 2.0]",3} │
| │ {3.0,"(2.0, 3.0]",2} │
| └──────────────────────┘
|
| implode(self, *, maintain_order: 'bool' = True) -> 'Expr'
| Aggregate values into a list.
|
| The returned list itself is a scalar value of `list` dtype.
|
| Parameters
| ----------
| maintain_order
| Whether to preserve the order of elements in the list. Setting this
| to `False` can improve performance, especially within `group_by`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": [4, 5, 6],
| ... }
| ... )
| >>> df.select(pl.all().implode())
| shape: (1, 2)
| ┌───────────┬───────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ list[i64] ┆ list[i64] │
| ╞═══════════╪═══════════╡
| │ [1, 2, 3] ┆ [4, 5, 6] │
| └───────────┴───────────┘
|
| index_of(self, element: 'IntoExpr') -> 'Expr'
| Get the index of the first occurrence of a value, or ``None`` if it's not found.
|
| Parameters
| ----------
| element
| Value to find.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, None, 17]})
| >>> df.select(
| ... [
| ... pl.col("a").index_of(17).alias("seventeen"),
| ... pl.col("a").index_of(None).alias("null"),
| ... pl.col("a").index_of(55).alias("fiftyfive"),
| ... ]
| ... )
| shape: (1, 3)
| ┌───────────┬──────┬───────────┐
| │ seventeen ┆ null ┆ fiftyfive │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═══════════╪══════╪═══════════╡
| │ 2 ┆ 1 ┆ null │
| └───────────┴──────┴───────────┘
|
| inspect(self, fmt: 'str_' = '{}') -> 'Expr'
| Print the value that this expression evaluates to and pass on the value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 1, 2]})
| >>> df.select(pl.col("foo").cum_sum().inspect("value is: {}").alias("bar"))
| value is: shape: (3,)
| Series: 'foo' [i64]
| [
| 1
| 2
| 4
| ]
| shape: (3, 1)
| ┌─────┐
| │ bar │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 4 │
| └─────┘
|
| interpolate(self, method: 'InterpolationMethod' = 'linear') -> 'Expr'
| Interpolate intermediate values.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| method : {'linear', 'nearest'}
| Interpolation method.
|
| Examples
| --------
| Fill null values using linear interpolation.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, None, 3],
| ... "b": [1.0, float("nan"), 3.0],
| ... }
| ... )
| >>> df.select(pl.all().interpolate())
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 1.0 ┆ 1.0 │
| │ 2.0 ┆ NaN │
| │ 3.0 ┆ 3.0 │
| └─────┴─────┘
|
| Fill null values using nearest interpolation.
|
| >>> df.select(pl.all().interpolate("nearest"))
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 1.0 │
| │ 3 ┆ NaN │
| │ 3 ┆ 3.0 │
| └─────┴─────┘
|
| Regrid data to a new grid.
|
| >>> df_original_grid = pl.DataFrame(
| ... {
| ... "grid_points": [1, 3, 10],
| ... "values": [2.0, 6.0, 20.0],
| ... }
| ... ) # Interpolate from this to the new grid
| >>> df_new_grid = pl.DataFrame({"grid_points": range(1, 11)})
| >>> df_new_grid.join(
| ... df_original_grid, on="grid_points", how="left", coalesce=True
| ... ).with_columns(pl.col("values").interpolate())
| shape: (10, 2)
| ┌─────────────┬────────┐
| │ grid_points ┆ values │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════════════╪════════╡
| │ 1 ┆ 2.0 │
| │ 2 ┆ 4.0 │
| │ 3 ┆ 6.0 │
| │ 4 ┆ 8.0 │
| │ 5 ┆ 10.0 │
| │ 6 ┆ 12.0 │
| │ 7 ┆ 14.0 │
| │ 8 ┆ 16.0 │
| │ 9 ┆ 18.0 │
| │ 10 ┆ 20.0 │
| └─────────────┴────────┘
|
| interpolate_by(self, by: 'IntoExpr') -> 'Expr'
| Fill null values using interpolation based on another column.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| by
| Column to interpolate values based on.
|
| Examples
| --------
| Fill null values using linear interpolation.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, None, None, 3],
| ... "b": [1, 2, 7, 8],
| ... }
| ... )
| >>> df.with_columns(a_interpolated=pl.col("a").interpolate_by("b"))
| shape: (4, 3)
| ┌──────┬─────┬────────────────┐
| │ a ┆ b ┆ a_interpolated │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞══════╪═════╪════════════════╡
| │ 1 ┆ 1 ┆ 1.0 │
| │ null ┆ 2 ┆ 1.285714 │
| │ null ┆ 7 ┆ 2.714286 │
| │ 3 ┆ 8 ┆ 3.0 │
| └──────┴─────┴────────────────┘
|
| is_between(self, lower_bound: 'IntoExpr', upper_bound: 'IntoExpr', closed: 'ClosedInterval' = 'both') -> 'Expr'
| Check if this expression is between the given lower and upper bounds.
|
| Parameters
| ----------
| lower_bound
| Lower bound value. Accepts expression input. Strings are parsed as column
| names, other non-expression inputs are parsed as literals.
| upper_bound
| Upper bound value. Accepts expression input. Strings are parsed as column
| names, other non-expression inputs are parsed as literals.
| closed : {'both', 'left', 'right', 'none'}
| Define which sides of the interval are closed (inclusive).
|
| Notes
| -----
| If the value of the `lower_bound` is greater than that of the `upper_bound`
| then the result will be False, as no value can satisfy the condition.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"num": [1, 2, 3, 4, 5]})
| >>> df.with_columns(pl.col("num").is_between(2, 4).alias("is_between"))
| shape: (5, 2)
| ┌─────┬────────────┐
| │ num ┆ is_between │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪════════════╡
| │ 1 ┆ false │
| │ 2 ┆ true │
| │ 3 ┆ true │
| │ 4 ┆ true │
| │ 5 ┆ false │
| └─────┴────────────┘
|
| Use the `closed` argument to include or exclude the values at the bounds:
|
| >>> df.with_columns(
| ... pl.col("num").is_between(2, 4, closed="left").alias("is_between")
| ... )
| shape: (5, 2)
| ┌─────┬────────────┐
| │ num ┆ is_between │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪════════════╡
| │ 1 ┆ false │
| │ 2 ┆ true │
| │ 3 ┆ true │
| │ 4 ┆ false │
| │ 5 ┆ false │
| └─────┴────────────┘
|
| You can also use strings as well as numeric/temporal values (note: ensure that
| string literals are wrapped with `lit` so as not to conflate them with
| column names):
|
| >>> df = pl.DataFrame({"a": ["a", "b", "c", "d", "e"]})
| >>> df.with_columns(
| ... pl.col("a")
| ... .is_between(pl.lit("a"), pl.lit("c"), closed="both")
| ... .alias("is_between")
| ... )
| shape: (5, 2)
| ┌─────┬────────────┐
| │ a ┆ is_between │
| │ --- ┆ --- │
| │ str ┆ bool │
| ╞═════╪════════════╡
| │ a ┆ true │
| │ b ┆ true │
| │ c ┆ true │
| │ d ┆ false │
| │ e ┆ false │
| └─────┴────────────┘
|
| Use column expressions as lower/upper bounds, comparing to a literal value:
|
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5], "b": [5, 4, 3, 2, 1]})
| >>> df.with_columns(
| ... pl.lit(3).is_between(pl.col("a"), pl.col("b")).alias("between_ab")
| ... )
| shape: (5, 3)
| ┌─────┬─────┬────────────┐
| │ a ┆ b ┆ between_ab │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ bool │
| ╞═════╪═════╪════════════╡
| │ 1 ┆ 5 ┆ true │
| │ 2 ┆ 4 ┆ true │
| │ 3 ┆ 3 ┆ true │
| │ 4 ┆ 2 ┆ false │
| │ 5 ┆ 1 ┆ false │
| └─────┴─────┴────────────┘
|
| is_close(self, other: 'IntoExpr', *, abs_tol: 'float' = 0.0, rel_tol: 'float' = 1e-09, nans_equal: 'bool' = False) -> 'Expr'
| Check if this expression is close, i.e. almost equal, to the other expression.
|
| Two values `a` and `b` are considered close if the following condition holds:
|
| .. math::
| |a-b| \le max \{ \text{rel_tol} \cdot max \{ |a|, |b| \}, \text{abs_tol} \}
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
| abs_tol
| Absolute tolerance. This is the maximum allowed absolute difference between
| two values. Must be non-negative.
| rel_tol
| Relative tolerance. This is the maximum allowed difference between two
| values, relative to the larger absolute value. Must be non-negative.
| nans_equal
| Whether NaN values should be considered equal.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Notes
| -----
| The implementation of this method is symmetric and mirrors the behavior of
| :meth:`math.isclose`. Specifically note that this behavior is different to
| :meth:`numpy.isclose`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.5, 2.0, 2.5], "b": [1.55, 2.2, 3.0]})
| >>> df.with_columns(pl.col("a").is_close("b", abs_tol=0.1).alias("is_close"))
| shape: (3, 3)
| ┌─────┬──────┬──────────┐
| │ a ┆ b ┆ is_close │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪══════╪══════════╡
| │ 1.5 ┆ 1.55 ┆ true │
| │ 2.0 ┆ 2.2 ┆ false │
| │ 2.5 ┆ 3.0 ┆ false │
| └─────┴──────┴──────────┘
|
| is_duplicated(self) -> 'Expr'
| Return a boolean mask indicating duplicated values.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
| >>> df.select(pl.col("a").is_duplicated())
| shape: (3, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ true │
| │ true │
| │ false │
| └───────┘
|
| is_empty(self, *, ignore_nulls: 'bool' = False) -> 'Expr'
| Return whether the column is empty.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| ignore_nulls
| If true a column containing only nulls will also be considered empty.
| The default is false.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [None, None]})
| >>> df.select(
| ... a=pl.col.x.is_empty(),
| ... b=pl.col.x.drop_nulls().is_empty(),
| ... c=pl.col.x.is_empty(ignore_nulls=True),
| ... )
| shape: (1, 3)
| ┌───────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞═══════╪══════╪══════╡
| │ false ┆ true ┆ true │
| └───────┴──────┴──────┘
|
| is_finite(self) -> 'Expr'
| Returns a boolean Series indicating which values are finite.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1.0, 2],
| ... "B": [3.0, float("inf")],
| ... }
| ... )
| >>> df.select(pl.all().is_finite())
| shape: (2, 2)
| ┌──────┬───────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ bool ┆ bool │
| ╞══════╪═══════╡
| │ true ┆ true │
| │ true ┆ false │
| └──────┴───────┘
|
| is_first_distinct(self) -> 'Expr'
| Return a boolean mask indicating the first occurrence of each distinct value.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 3, 2]})
| >>> df.with_columns(pl.col("a").is_first_distinct().alias("first"))
| shape: (5, 2)
| ┌─────┬───────┐
| │ a ┆ first │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪═══════╡
| │ 1 ┆ true │
| │ 1 ┆ false │
| │ 2 ┆ true │
| │ 3 ┆ true │
| │ 2 ┆ false │
| └─────┴───────┘
|
| is_in(self, other: 'Expr | Collection[Any] | Series', *, nulls_equal: 'bool' = False) -> 'Expr'
| Check if elements of this expression are present in the other Series.
|
| Parameters
| ----------
| other
| Series or sequence of primitive type.
| nulls_equal : bool, default False
| If True, treat null as a distinct value. Null values will not propagate.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"sets": [[1, 2, 3], [1, 2], [9, 10]], "optional_members": [1, 2, 3]}
| ... )
| >>> df.with_columns(contains=pl.col("optional_members").is_in("sets"))
| shape: (3, 3)
| ┌───────────┬──────────────────┬──────────┐
| │ sets ┆ optional_members ┆ contains │
| │ --- ┆ --- ┆ --- │
| │ list[i64] ┆ i64 ┆ bool │
| ╞═══════════╪══════════════════╪══════════╡
| │ [1, 2, 3] ┆ 1 ┆ true │
| │ [1, 2] ┆ 2 ┆ true │
| │ [9, 10] ┆ 3 ┆ false │
| └───────────┴──────────────────┴──────────┘
|
| is_infinite(self) -> 'Expr'
| Returns a boolean Series indicating which values are infinite.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1.0, 2],
| ... "B": [3.0, float("inf")],
| ... }
| ... )
| >>> df.select(pl.all().is_infinite())
| shape: (2, 2)
| ┌───────┬───────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ bool ┆ bool │
| ╞═══════╪═══════╡
| │ false ┆ false │
| │ false ┆ true │
| └───────┴───────┘
|
| is_last_distinct(self) -> 'Expr'
| Return a boolean mask indicating the last occurrence of each distinct value.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 3, 2]})
| >>> df.with_columns(pl.col("a").is_last_distinct().alias("last"))
| shape: (5, 2)
| ┌─────┬───────┐
| │ a ┆ last │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪═══════╡
| │ 1 ┆ false │
| │ 1 ┆ true │
| │ 2 ┆ false │
| │ 3 ┆ true │
| │ 2 ┆ true │
| └─────┴───────┘
|
| is_nan(self) -> 'Expr'
| Returns a boolean Series indicating which values are NaN.
|
| Notes
| -----
| Floating point `NaN` (Not A Number) should not be confused
| with missing data represented as `Null/None`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(pl.col(pl.Float64).is_nan().name.suffix("_isnan"))
| shape: (5, 3)
| ┌──────┬─────┬─────────┐
| │ a ┆ b ┆ b_isnan │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪═════╪═════════╡
| │ 1 ┆ 1.0 ┆ false │
| │ 2 ┆ 2.0 ┆ false │
| │ null ┆ NaN ┆ true │
| │ 1 ┆ 1.0 ┆ false │
| │ 5 ┆ 5.0 ┆ false │
| └──────┴─────┴─────────┘
|
| is_not_nan(self) -> 'Expr'
| Returns a boolean Series indicating which values are not NaN.
|
| Notes
| -----
| Floating point `NaN` (Not A Number) should not be confused
| with missing data represented as `Null/None`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(pl.col(pl.Float64).is_not_nan().name.suffix("_is_not_nan"))
| shape: (5, 3)
| ┌──────┬─────┬──────────────┐
| │ a ┆ b ┆ b_is_not_nan │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪═════╪══════════════╡
| │ 1 ┆ 1.0 ┆ true │
| │ 2 ┆ 2.0 ┆ true │
| │ null ┆ NaN ┆ false │
| │ 1 ┆ 1.0 ┆ true │
| │ 5 ┆ 5.0 ┆ true │
| └──────┴─────┴──────────────┘
|
| is_not_null(self) -> 'Expr'
| Returns a boolean Series indicating which values are not null.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.all().is_not_null().name.suffix("_not_null") # nan != null
| ... )
| shape: (5, 4)
| ┌──────┬─────┬────────────┬────────────┐
| │ a ┆ b ┆ a_not_null ┆ b_not_null │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪═════╪════════════╪════════════╡
| │ 1 ┆ 1.0 ┆ true ┆ true │
| │ 2 ┆ 2.0 ┆ true ┆ true │
| │ null ┆ NaN ┆ false ┆ true │
| │ 1 ┆ 1.0 ┆ true ┆ true │
| │ 5 ┆ 5.0 ┆ true ┆ true │
| └──────┴─────┴────────────┴────────────┘
|
| is_null(self) -> 'Expr'
| Returns a boolean Series indicating which values are null.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(pl.all().is_null().name.suffix("_isnull")) # nan != null
| shape: (5, 4)
| ┌──────┬─────┬──────────┬──────────┐
| │ a ┆ b ┆ a_isnull ┆ b_isnull │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪═════╪══════════╪══════════╡
| │ 1 ┆ 1.0 ┆ false ┆ false │
| │ 2 ┆ 2.0 ┆ false ┆ false │
| │ null ┆ NaN ┆ true ┆ false │
| │ 1 ┆ 1.0 ┆ false ┆ false │
| │ 5 ┆ 5.0 ┆ false ┆ false │
| └──────┴─────┴──────────┴──────────┘
|
| is_sorted(self, *, descending: 'bool | None' = False, nulls_last: 'bool | None' = False) -> 'Expr'
| Checks if an expression is sorted.
|
| If `descending` and/or `nulls_last` are None, it will check `True` and `False`
| for the unspecified option(s), and return `True` if the expression is sorted
| under any combination of those settings.
|
| Parameters
| ----------
| descending
| Checks if the expression is sorted in descending order.
| Defaults to False.
| nulls_last
| Consider null values as being ordered last when checking sortedness.
| Defaults to False.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| Check if a column is sorted in ascending order.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.select(pl.col("a").is_sorted())
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ bool │
| ╞══════╡
| │ true │
| └──────┘
|
| Check if a column is sorted in descending order.
|
| >>> df = pl.DataFrame({"a": [4, 3, 2, 1]})
| >>> df.select(pl.col("a").is_sorted(descending=True))
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ bool │
| ╞══════╡
| │ true │
| └──────┘
|
| Check if a column is sorted in either direction.
|
| >>> df = pl.DataFrame({"a": [4, 3, 2, 1]})
| >>> df.select(pl.col("a").is_sorted(descending=None))
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ bool │
| ╞══════╡
| │ true │
| └──────┘
|
| is_unique(self) -> 'Expr'
| Get mask of unique values.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
| >>> df.select(pl.col("a").is_unique())
| shape: (3, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ false │
| │ true │
| └───────┘
|
| item(self, *, allow_empty: 'bool' = False) -> 'Expr'
| Get the single value.
|
| This raises an error if there is not exactly one value.
|
| Parameters
| ----------
| allow_empty
| Allow having no values to return `null`.
|
| See Also
| --------
| :meth:`Expr.get` : Get a single value by index.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1]})
| >>> df.select(pl.col("a").item())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| └─────┘
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").item())
| Traceback (most recent call last):
| ...
| polars.exceptions.ComputeError: aggregation 'item' expected a single value, got 3 values
| ...
| >>> df.head(0).select(pl.col("a").item(allow_empty=True))
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| └──────┘
|
| kurtosis(self, *, fisher: 'bool' = True, bias: 'bool' = True) -> 'Expr'
| Compute the kurtosis (Fisher or Pearson) of a dataset.
|
| Kurtosis is the fourth central moment divided by the square of the
| variance. If Fisher's definition is used, then 3.0 is subtracted from
| the result to give 0.0 for a normal distribution.
| If bias is False then the kurtosis is calculated using k statistics to
| eliminate bias coming from biased moment estimators.
|
| See scipy.stats for more information
|
| Parameters
| ----------
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").kurtosis())
| shape: (1, 1)
| ┌───────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═══════════╡
| │ -1.153061 │
| └───────────┘
|
| last(self, *, ignore_nulls: 'bool' = False) -> 'Expr'
| Get the last value.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the last non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 2]})
| >>> df.select(pl.col("a").last())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| └─────┘
|
| le(self, other: 'Any') -> 'Expr'
| Method equivalent of "less than or equal" operator `expr <= other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5.0, 4.0, float("nan"), 0.5],
| ... "y": [5.0, 3.5, float("nan"), 2.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").le(pl.col("y")).alias("x <= y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x <= y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 5.0 ┆ 5.0 ┆ true │
| │ 4.0 ┆ 3.5 ┆ false │
| │ NaN ┆ NaN ┆ true │
| │ 0.5 ┆ 2.0 ┆ true │
| └─────┴─────┴────────┘
|
| len(self) -> 'Expr'
| Return the number of elements in the column.
|
| Null values count towards the total.
|
| Returns
| -------
| Expr
| Expression of data type :class:`UInt32`.
|
| See Also
| --------
| count
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [None, 4, 4]})
| >>> df.select(pl.all().len())
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 3 ┆ 3 │
| └─────┴─────┘
|
| limit(self, n: 'int | Expr' = 10) -> 'Expr'
| Get the first `n` rows (alias for :func:`Expr.head`).
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7]})
| >>> df.select(pl.col("foo").limit(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| log(self, base: 'float | IntoExpr' = 2.718281828459045) -> 'Expr'
| Compute the logarithm to a given base.
|
| Parameters
| ----------
| base
| Given base, defaults to `e`
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").log(base=2))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.0 │
| │ 1.0 │
| │ 1.584963 │
| └──────────┘
|
| log10(self) -> 'Expr'
| Compute the base 10 logarithm of the input array, element-wise.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").log10())
| shape: (3, 1)
| ┌─────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞═════════╡
| │ 0.0 │
| │ 0.30103 │
| │ 0.60206 │
| └─────────┘
|
| log1p(self) -> 'Expr'
| Compute the natural logarithm of each element plus one.
|
| This computes `log(1 + x)` but is more numerically stable for `x` close to zero.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").log1p())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.693147 │
| │ 1.098612 │
| │ 1.386294 │
| └──────────┘
|
| lower_bound(self) -> 'Expr'
| Calculate the lower bound.
|
| Returns a unit Series with the lowest value possible for the dtype of this
| expression.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").lower_bound())
| shape: (1, 1)
| ┌──────────────────────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════════════════════╡
| │ -9223372036854775808 │
| └──────────────────────┘
|
| lt(self, other: 'Any') -> 'Expr'
| Method equivalent of "less than" operator `expr < other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 3.0],
| ... "y": [2.0, 2.0, float("nan"), 4.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").lt(pl.col("y")).alias("x < y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────┐
| │ x ┆ y ┆ x < y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪═══════╡
| │ 1.0 ┆ 2.0 ┆ true │
| │ 2.0 ┆ 2.0 ┆ false │
| │ NaN ┆ NaN ┆ false │
| │ 3.0 ┆ 4.0 ┆ true │
| └─────┴─────┴───────┘
|
| map_batches(self, function: 'Callable[[Series], Series | Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, agg_list: 'bool' = False, is_elementwise: 'bool' = False, returns_scalar: 'bool' = False) -> 'Expr'
| Apply a custom python function to a whole Series or sequence of Series.
|
| The output of this custom function is presumed to be either a Series,
| or a NumPy array (in which case it will be automatically converted into
| a Series), or a scalar that will be converted into a Series. If the
| result is a scalar and you want it to stay as a scalar, pass in
| ``returns_scalar=True``. If you want to apply a
| custom function elementwise over single values, see :func:`map_elements`.
| A reasonable use case for `map` functions is transforming the values
| represented by an expression using a third-party library.
|
| Parameters
| ----------
| function
| Lambda/function to apply.
| return_dtype
| Datatype of the output Series.
|
| It is recommended to set this whenever possible. If this is `None`, it tries
| to infer the datatype by calling the function with dummy data and looking at
| the output.
| agg_list
| First implode when in a group-by aggregation.
|
| .. deprecated:: 1.32.0
|
| Use `expr.implode().map_batches(..)` instead.
| is_elementwise
| Set to true if the operations is elementwise for better performance
| and optimization.
|
| An elementwise operations has unit or equal length for all inputs
| and can be ran sequentially on slices without results being affected.
| returns_scalar
| If the function returns a scalar, by default it will be wrapped in
| a list in the output, since the assumption is that the function
| always returns something Series-like. If you want to keep the
| result as a scalar, set this argument to True.
|
| Notes
| -----
| A UDF passed to `map_batches` must be pure, meaning that it cannot modify
| or depend on state other than its arguments. Polars may call the function
| with arbitrary input data.
|
| See Also
| --------
| map_elements
| replace
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "sine": [0.0, 1.0, 0.0, -1.0],
| ... "cosine": [1.0, 0.0, -1.0, 0.0],
| ... }
| ... )
| >>> df.select(
| ... pl.all().map_batches(
| ... lambda x: x.to_numpy().argmax(),
| ... returns_scalar=True,
| ... )
| ... )
| shape: (1, 2)
| ┌──────┬────────┐
| │ sine ┆ cosine │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪════════╡
| │ 1 ┆ 0 │
| └──────┴────────┘
|
| Here's an example of a function that returns a scalar, where we want it
| to stay as a scalar:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [0, 1, 0, 1],
| ... "b": [1, 2, 3, 4],
| ... }
| ... )
| >>> df.group_by("a").agg(
| ... pl.col("b").map_batches(
| ... lambda x: x.max(), returns_scalar=True, return_dtype=pl.self_dtype()
| ... )
| ... ) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 0 ┆ 3 │
| └─────┴─────┘
|
| Call a function that takes multiple arguments by creating a `struct` and
| referencing its fields inside the function call.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [5, 1, 0, 3],
| ... "b": [4, 2, 3, 4],
| ... }
| ... )
| >>> df.with_columns(
| ... a_times_b=pl.struct("a", "b").map_batches(
| ... lambda x: np.multiply(x.struct.field("a"), x.struct.field("b")),
| ... return_dtype=pl.Int64,
| ... )
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────────┐
| │ a ┆ b ┆ a_times_b │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═══════════╡
| │ 5 ┆ 4 ┆ 20 │
| │ 1 ┆ 2 ┆ 2 │
| │ 0 ┆ 3 ┆ 0 │
| │ 3 ┆ 4 ┆ 12 │
| └─────┴─────┴───────────┘
|
| map_elements(self, function: 'Callable[[Any], Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, skip_nulls: 'bool' = True, pass_name: 'bool' = False, strategy: 'MapElementsStrategy' = 'thread_local', returns_scalar: 'bool' = False) -> 'Expr'
| Map a custom/user-defined function (UDF) to each element of a column.
|
| .. warning::
| This method is much slower than the native expressions API.
| Only use it if you cannot implement your logic otherwise.
|
| Suppose that the function is: `x ↦ sqrt(x)`:
|
| - For mapping elements of a series, consider:
| `pl.col("col_name").sqrt()`.
| - For mapping inner elements of lists, consider:
| `pl.col("col_name").list.eval(pl.element().sqrt())`.
| - For mapping elements of struct fields, consider:
| `pl.col("col_name").struct.field("field_name").sqrt()`.
|
| If you want to replace the original column or field,
| consider :meth:`.with_columns <polars.DataFrame.with_columns>`
| and :meth:`.with_fields <polars.Expr.struct.with_fields>`.
|
| Parameters
| ----------
| function
| Lambda/function to map.
| return_dtype
| Datatype of the output Series.
|
| It is recommended to set this whenever possible. If this is `None`, it tries
| to infer the datatype by calling the function with dummy data and looking at
| the output.
| skip_nulls
| Don't map the function over values that contain nulls (this is faster).
| pass_name
| Pass the Series name to the custom function (this is more expensive).
| returns_scalar
| .. deprecated:: 1.32.0
| Is ignored and will be removed in 2.0.
| strategy : {'thread_local', 'threading'}
| The threading strategy to use.
|
| - 'thread_local': run the python function on a single thread.
| - 'threading': run the python function on separate threads. Use with
| care as this can slow performance. This might only speed up
| your code if the amount of work per element is significant
| and the python function releases the GIL (e.g. via calling
| a c function)
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Notes
| -----
| * Using `map_elements` is strongly discouraged as you will be effectively
| running python "for" loops, which will be very slow. Wherever possible you
| should prefer the native expression API to achieve the best performance.
|
| * If your function is expensive and you don't want it to be called more than
| once for a given input, consider applying an `@lru_cache` decorator to it.
| If your data is suitable you may achieve *significant* speedups.
|
| * Window function application using `over` is considered a GroupBy context
| here, so `map_elements` can be used to map functions over window groups.
|
| * A UDF passed to `map_elements` must be pure, meaning that it cannot modify or
| depend on state other than its arguments. Polars may call the function
| with arbitrary input data.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 1],
| ... "b": ["a", "b", "c", "c"],
| ... }
| ... )
|
| The function is applied to each element of column `'a'`:
|
| >>> df.with_columns( # doctest: +SKIP
| ... pl.col("a")
| ... .map_elements(lambda x: x * 2, return_dtype=pl.self_dtype())
| ... .alias("a_times_2"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────────┐
| │ a ┆ b ┆ a_times_2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ i64 │
| ╞═════╪═════╪═══════════╡
| │ 1 ┆ a ┆ 2 │
| │ 2 ┆ b ┆ 4 │
| │ 3 ┆ c ┆ 6 │
| │ 1 ┆ c ┆ 2 │
| └─────┴─────┴───────────┘
|
| Tip: it is better to implement this with an expression:
|
| >>> df.with_columns(
| ... (pl.col("a") * 2).alias("a_times_2"),
| ... ) # doctest: +IGNORE_RESULT
|
| >>> (
| ... df.lazy()
| ... .group_by("b")
| ... .agg(
| ... pl.col("a")
| ... .implode()
| ... .map_elements(lambda x: x.sum(), return_dtype=pl.Int64)
| ... )
| ... .collect()
| ... ) # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌─────┬─────┐
| │ b ┆ a │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ b ┆ 2 │
| │ c ┆ 4 │
| └─────┴─────┘
|
| Tip: again, it is better to implement this with an expression:
|
| >>> (
| ... df.lazy()
| ... .group_by("b", maintain_order=True)
| ... .agg(pl.col("a").sum())
| ... .collect()
| ... ) # doctest: +IGNORE_RESULT
|
| Window function application using `over` will behave as a GroupBy
| context, with your function receiving individual window groups:
|
| >>> df = pl.DataFrame(
| ... {
| ... "key": ["x", "x", "y", "x", "y", "z"],
| ... "val": [1, 1, 1, 1, 1, 1],
| ... }
| ... )
| >>> df.with_columns(
| ... scaled=pl.col("val")
| ... .implode()
| ... .map_elements(lambda s: s * len(s), return_dtype=pl.List(pl.Int64))
| ... .explode(empty_as_null=False)
| ... .over("key"),
| ... ).sort("key")
| shape: (6, 3)
| ┌─────┬─────┬────────┐
| │ key ┆ val ┆ scaled │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪════════╡
| │ x ┆ 1 ┆ 3 │
| │ x ┆ 1 ┆ 3 │
| │ x ┆ 1 ┆ 3 │
| │ y ┆ 1 ┆ 2 │
| │ y ┆ 1 ┆ 2 │
| │ z ┆ 1 ┆ 1 │
| └─────┴─────┴────────┘
|
| Note that this function would *also* be better-implemented natively:
|
| >>> df.with_columns(
| ... scaled=(pl.col("val") * pl.col("val").count()).over("key"),
| ... ).sort("key") # doctest: +IGNORE_RESULT
|
| max(self) -> 'Expr'
| Get maximum value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0]})
| >>> df.select(pl.col("a").max())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
|
| max_by(self, by: 'IntoExpr') -> 'Expr'
| Get maximum value, ordered by another expression.
|
| If the by expression has multiple values equal to the maximum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the largest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0], "b": ["x", "y", "z"]})
| >>> df.select(pl.col("b").max_by("a"))
| shape: (1, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ str │
| ╞═════╡
| │ z │
| └─────┘
|
| mean(self) -> 'Expr'
| Get mean value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").mean())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| └─────┘
|
| median(self) -> 'Expr'
| Get median value using linear interpolation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").median())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| └─────┘
|
| min(self) -> 'Expr'
| Get minimum value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0]})
| >>> df.select(pl.col("a").min())
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ -1.0 │
| └──────┘
|
| min_by(self, by: 'IntoExpr') -> 'Expr'
| Get minimum value, ordered by another expression.
|
| If the by expression has multiple values equal to the minimum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the smallest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0], "b": ["x", "y", "z"]})
| >>> df.select(pl.col("b").min_by("a"))
| shape: (1, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ str │
| ╞═════╡
| │ x │
| └─────┘
|
| mod(self, other: 'Any') -> 'Expr'
| Method equivalent of modulus operator `expr % other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [0, 1, 2, 3, 4]})
| >>> df.with_columns(pl.col("x").mod(2).alias("x%2"))
| shape: (5, 2)
| ┌─────┬─────┐
| │ x ┆ x%2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 0 ┆ 0 │
| │ 1 ┆ 1 │
| │ 2 ┆ 0 │
| │ 3 ┆ 1 │
| │ 4 ┆ 0 │
| └─────┴─────┘
|
| mode(self, *, maintain_order: 'bool' = False) -> 'Expr'
| Compute the most occurring value(s).
|
| Can return multiple Values.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 1, 2, 3],
| ... "b": [1, 1, 2, 2],
| ... }
| ... )
| >>> df.select(pl.all().mode().first()) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| └─────┴─────┘
|
| mul(self, other: 'Any') -> 'Expr'
| Method equivalent of multiplication operator `expr * other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 4, 8, 16]})
| >>> df.with_columns(
| ... pl.col("x").mul(2).alias("x*2"),
| ... pl.col("x").mul(pl.col("x").log(2)).alias("x * xlog2"),
| ... )
| shape: (5, 3)
| ┌─────┬─────┬───────────┐
| │ x ┆ x*2 ┆ x * xlog2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪═════╪═══════════╡
| │ 1 ┆ 2 ┆ 0.0 │
| │ 2 ┆ 4 ┆ 2.0 │
| │ 4 ┆ 8 ┆ 8.0 │
| │ 8 ┆ 16 ┆ 24.0 │
| │ 16 ┆ 32 ┆ 64.0 │
| └─────┴─────┴───────────┘
|
| n_unique(self) -> 'Expr'
| Count unique values.
|
| Notes
| -----
| `null` is considered to be a unique value for the purposes of this operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 1, 2, 2, 3], "y": [1, 1, 1, None, None]})
| >>> df.select(
| ... x_unique=pl.col("x").n_unique(),
| ... y_unique=pl.col("y").n_unique(),
| ... )
| shape: (1, 2)
| ┌──────────┬──────────┐
| │ x_unique ┆ y_unique │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞══════════╪══════════╡
| │ 3 ┆ 2 │
| └──────────┴──────────┘
|
| nan_max(self) -> 'Expr'
| Get maximum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.0, float("nan")]})
| >>> df.select(pl.col("a").nan_max())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ NaN │
| └─────┘
|
| nan_min(self) -> 'Expr'
| Get minimum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.0, float("nan")]})
| >>> df.select(pl.col("a").nan_min())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ NaN │
| └─────┘
|
| ne(self, other: 'Any') -> 'Expr'
| Method equivalent of inequality operator `expr != other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0],
| ... "y": [2.0, 2.0, float("nan"), 4.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").ne(pl.col("y")).alias("x != y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x != y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 1.0 ┆ 2.0 ┆ true │
| │ 2.0 ┆ 2.0 ┆ false │
| │ NaN ┆ NaN ┆ false │
| │ 4.0 ┆ 4.0 ┆ false │
| └─────┴─────┴────────┘
|
| ne_missing(self, other: 'Any') -> 'Expr'
| Method equivalent of equality operator `expr != other` where `None == None`.
|
| This differs from default `ne` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0, None, None],
| ... "y": [2.0, 2.0, float("nan"), 4.0, 5.0, None],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").ne(pl.col("y")).alias("x ne y"),
| ... pl.col("x").ne_missing(pl.col("y")).alias("x ne_missing y"),
| ... )
| shape: (6, 4)
| ┌──────┬──────┬────────┬────────────────┐
| │ x ┆ y ┆ x ne y ┆ x ne_missing y │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪══════╪════════╪════════════════╡
| │ 1.0 ┆ 2.0 ┆ true ┆ true │
| │ 2.0 ┆ 2.0 ┆ false ┆ false │
| │ NaN ┆ NaN ┆ false ┆ false │
| │ 4.0 ┆ 4.0 ┆ false ┆ false │
| │ null ┆ 5.0 ┆ null ┆ true │
| │ null ┆ null ┆ null ┆ false │
| └──────┴──────┴────────┴────────────────┘
|
| neg(self) -> 'Expr'
| Method equivalent of unary minus operator `-expr`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 2, None]})
| >>> df.with_columns(pl.col("a").neg())
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 1 │
| │ 0 │
| │ -2 │
| │ null │
| └──────┘
|
| not_(self) -> 'Expr'
| Method equivalent of bitwise "not" operator `~expr`.
|
| This has the effect of negating logical boolean expressions,
| but operates bitwise on integers.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "label": ["aa", "bb", "cc", "dd", "ee"],
| ... "valid": [True, False, None, False, True],
| ... "int_code": [1, 0, 2, None, -1],
| ... }
| ... )
|
| Apply "not" to boolean expression (negates the value) and
| integer expression (operates bitwise):
|
| >>> df.with_columns(
| ... not_valid=pl.col("valid").not_(),
| ... not_int_code=pl.col("int_code").not_(),
| ... )
| shape: (5, 5)
| ┌───────┬───────┬──────────┬───────────┬──────────────┐
| │ label ┆ valid ┆ int_code ┆ not_valid ┆ not_int_code │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ bool ┆ i64 ┆ bool ┆ i64 │
| ╞═══════╪═══════╪══════════╪═══════════╪══════════════╡
| │ aa ┆ true ┆ 1 ┆ false ┆ -2 │
| │ bb ┆ false ┆ 0 ┆ true ┆ -1 │
| │ cc ┆ null ┆ 2 ┆ null ┆ -3 │
| │ dd ┆ false ┆ null ┆ true ┆ null │
| │ ee ┆ true ┆ -1 ┆ false ┆ 0 │
| └───────┴───────┴──────────┴───────────┴──────────────┘
|
| null_count(self) -> 'Expr'
| Count null values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 1, None],
| ... "b": [10, None, 300],
| ... "c": [350, 650, 850],
| ... }
| ... )
| >>> df.select(pl.all().null_count())
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 2 ┆ 1 ┆ 0 │
| └─────┴─────┴─────┘
|
| or_(self, *others: 'Any') -> 'Expr'
| Method equivalent of bitwise "or" operator `expr | other | ...`.
|
| This has the effect of combining logical boolean expressions,
| but operates bitwise on integers.
|
| Parameters
| ----------
| *others
| One or more integer or boolean expressions to evaluate/combine.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5, 6, 7, 4, 8],
| ... "y": [1.5, 2.5, 1.0, 4.0, -5.75],
| ... "z": [-9, 2, -1, 4, 8],
| ... }
| ... )
|
| Combine logical "or" conditions:
|
| >>> df.select(
| ... (pl.col("x") == pl.col("y"))
| ... .or_(
| ... pl.col("x") == pl.col("y"),
| ... pl.col("y") == pl.col("z"),
| ... pl.col("y").cast(int) == pl.col("z"),
| ... )
| ... .alias("any")
| ... )
| shape: (5, 1)
| ┌───────┐
| │ any │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ true │
| │ false │
| │ true │
| │ false │
| └───────┘
|
| Bitwise "or" operation on integer columns:
|
| >>> df.select("x", "z", x_or_z=pl.col("x").or_(pl.col("z")))
| shape: (5, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ z ┆ x_or_z │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪════════╡
| │ 5 ┆ -9 ┆ -9 │
| │ 6 ┆ 2 ┆ 6 │
| │ 7 ┆ -1 ┆ -1 │
| │ 4 ┆ 4 ┆ 4 │
| │ 8 ┆ 8 ┆ 8 │
| └─────┴─────┴────────┘
|
| over(self, partition_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, *more_exprs: 'IntoExpr', order_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, descending: 'bool' = False, nulls_last: 'bool' = False, mapping_strategy: 'WindowMappingStrategy' = 'group_to_rows') -> 'Expr'
| Compute expressions over the given groups.
|
| This expression is similar to performing a group by aggregation and joining the
| result back into the original DataFrame.
|
| The outcome is similar to how `window functions
| <https://www.postgresql.org/docs/current/tutorial-window.html>`_
| work in PostgreSQL.
|
| Parameters
| ----------
| partition_by
| Column(s) to group by. Accepts expression input. Strings are parsed as
| column names.
| *more_exprs
| Additional columns to group by, specified as positional arguments.
| order_by
| Order rows within each partition group before evaluating the expression.
| Useful for order-sensitive operations such as
| :func:`cum_sum` or :func:`diff`.
| descending
| In case 'order_by' is given, indicate whether to order in
| ascending or descending order.
| nulls_last
| In case 'order_by' is given, indicate whether to order
| the nulls in last position.
| mapping_strategy: {'group_to_rows', 'join', 'explode'}
| - group_to_rows
| If the aggregation results in multiple values per group, map them back
| to their row position in the DataFrame. This can only be done if each
| group yields the same number of elements before aggregation as after. If
| the aggregation results in one scalar value per group, this value will
| be mapped to every row.
| - join
| If the aggregation may result in multiple values per group, join the
| values as 'List<group_dtype>' to each row position. Warning: this can be
| memory intensive. If the aggregation always results in one scalar value
| per group, join this value as '<group_dtype>' to each row position.
| - explode
| If the aggregation may result in multiple values per group, map each
| value to a new row, similar to the results of `group_by` + `agg` +
| `explode`. If the aggregation always results in one scalar value per
| group, map this value to one row position. Sorting of the given groups
| is required if the groups are not part of the window operation for the
| operation, otherwise the result would not make sense. This operation
| changes the number of rows.
|
| Examples
| --------
| Pass the name of a column to compute the expression over that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "a", "b", "b", "b"],
| ... "b": [1, 2, 3, 5, 3],
| ... "c": [5, 4, 3, 2, 1],
| ... }
| ... )
| >>> df.with_columns(c_max=pl.col("c").max().over("a"))
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────┐
| │ a ┆ b ┆ c ┆ c_max │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ a ┆ 1 ┆ 5 ┆ 5 │
| │ a ┆ 2 ┆ 4 ┆ 5 │
| │ b ┆ 3 ┆ 3 ┆ 3 │
| │ b ┆ 5 ┆ 2 ┆ 3 │
| │ b ┆ 3 ┆ 1 ┆ 3 │
| └─────┴─────┴─────┴───────┘
|
| Expression input is also supported.
|
| >>> df.with_columns(c_max=pl.col("c").max().over(pl.col("b") // 2))
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────┐
| │ a ┆ b ┆ c ┆ c_max │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ a ┆ 1 ┆ 5 ┆ 5 │
| │ a ┆ 2 ┆ 4 ┆ 4 │
| │ b ┆ 3 ┆ 3 ┆ 4 │
| │ b ┆ 5 ┆ 2 ┆ 2 │
| │ b ┆ 3 ┆ 1 ┆ 4 │
| └─────┴─────┴─────┴───────┘
|
| Group by multiple columns by passing multiple column names or expressions.
|
| >>> df.with_columns(c_min=pl.col("c").min().over("a", pl.col("b") % 2))
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────┐
| │ a ┆ b ┆ c ┆ c_min │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ a ┆ 1 ┆ 5 ┆ 5 │
| │ a ┆ 2 ┆ 4 ┆ 4 │
| │ b ┆ 3 ┆ 3 ┆ 1 │
| │ b ┆ 5 ┆ 2 ┆ 1 │
| │ b ┆ 3 ┆ 1 ┆ 1 │
| └─────┴─────┴─────┴───────┘
|
| Mapping strategy `join` joins the values by group.
|
| >>> df.with_columns(
| ... c_pairs=pl.col("c").head(2).over("a", mapping_strategy="join")
| ... )
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────────┐
| │ a ┆ b ┆ c ┆ c_pairs │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ list[i64] │
| ╞═════╪═════╪═════╪═══════════╡
| │ a ┆ 1 ┆ 5 ┆ [5, 4] │
| │ a ┆ 2 ┆ 4 ┆ [5, 4] │
| │ b ┆ 3 ┆ 3 ┆ [3, 2] │
| │ b ┆ 5 ┆ 2 ┆ [3, 2] │
| │ b ┆ 3 ┆ 1 ┆ [3, 2] │
| └─────┴─────┴─────┴───────────┘
|
| Mapping strategy `explode` maps the values to new rows, changing the shape.
|
| >>> df.select(
| ... c_first_2=pl.col("c").head(2).over("a", mapping_strategy="explode")
| ... )
| shape: (4, 1)
| ┌───────────┐
| │ c_first_2 │
| │ --- │
| │ i64 │
| ╞═══════════╡
| │ 5 │
| │ 4 │
| │ 3 │
| │ 2 │
| └───────────┘
|
| You can use non-elementwise expressions with `over` too. By default they are
| evaluated using row-order, but you can specify a different one using `order_by`.
|
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "store_id": ["a", "a", "b", "b"],
| ... "date": [
| ... date(2024, 9, 18),
| ... date(2024, 9, 17),
| ... date(2024, 9, 18),
| ... date(2024, 9, 16),
| ... ],
| ... "sales": [7, 9, 8, 10],
| ... }
| ... )
| >>> df.with_columns(
| ... cumulative_sales=pl.col("sales")
| ... .cum_sum()
| ... .over("store_id", order_by="date")
| ... )
| shape: (4, 4)
| ┌──────────┬────────────┬───────┬──────────────────┐
| │ store_id ┆ date ┆ sales ┆ cumulative_sales │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 ┆ i64 │
| ╞══════════╪════════════╪═══════╪══════════════════╡
| │ a ┆ 2024-09-18 ┆ 7 ┆ 16 │
| │ a ┆ 2024-09-17 ┆ 9 ┆ 9 │
| │ b ┆ 2024-09-18 ┆ 8 ┆ 18 │
| │ b ┆ 2024-09-16 ┆ 10 ┆ 10 │
| └──────────┴────────────┴───────┴──────────────────┘
|
| If you don't require that the group order be preserved, then the more performant
| option is to use `mapping_strategy='explode'` - be careful however to only ever
| use this in a `select` statement, not a `with_columns` one.
|
| >>> window = {
| ... "partition_by": "store_id",
| ... "order_by": "date",
| ... "mapping_strategy": "explode",
| ... }
| >>> df.select(
| ... pl.all().over(**window),
| ... cumulative_sales=pl.col("sales").cum_sum().over(**window),
| ... )
| shape: (4, 4)
| ┌──────────┬────────────┬───────┬──────────────────┐
| │ store_id ┆ date ┆ sales ┆ cumulative_sales │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 ┆ i64 │
| ╞══════════╪════════════╪═══════╪══════════════════╡
| │ a ┆ 2024-09-17 ┆ 9 ┆ 9 │
| │ a ┆ 2024-09-18 ┆ 7 ┆ 16 │
| │ b ┆ 2024-09-16 ┆ 10 ┆ 10 │
| │ b ┆ 2024-09-18 ┆ 8 ┆ 18 │
| └──────────┴────────────┴───────┴──────────────────┘
|
| pct_change(self, n: 'int | IntoExprColumn' = 1) -> 'Expr'
| Computes percentage change between values.
|
| Percentage change (as fraction) between current element and most-recent
| non-null element at least `n` period(s) before the current element.
|
| Computes the change from the previous row by default.
|
| Parameters
| ----------
| n
| periods to shift for forming percent change.
|
| Notes
| -----
| Null values are preserved. If you're coming from pandas, this matches
| their ``fill_method=None`` behaviour.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [10, 11, 12, None, 12],
| ... }
| ... )
| >>> df.with_columns(pl.col("a").pct_change().alias("pct_change"))
| shape: (5, 2)
| ┌──────┬────────────┐
| │ a ┆ pct_change │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞══════╪════════════╡
| │ 10 ┆ null │
| │ 11 ┆ 0.1 │
| │ 12 ┆ 0.090909 │
| │ null ┆ null │
| │ 12 ┆ null │
| └──────┴────────────┘
|
| peak_max(self) -> 'Expr'
| Get a boolean mask of the local maximum peaks.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5]})
| >>> df.select(pl.col("a").peak_max())
| shape: (5, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ false │
| │ false │
| │ false │
| │ true │
| └───────┘
|
| peak_min(self) -> 'Expr'
| Get a boolean mask of the local minimum peaks.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [4, 1, 3, 2, 5]})
| >>> df.select(pl.col("a").peak_min())
| shape: (5, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ true │
| │ false │
| │ true │
| │ false │
| └───────┘
|
| pipe(self, function: 'Callable[Concatenate[Expr, P], T]', *args: 'P.args', **kwargs: 'P.kwargs') -> 'T'
| Offers a structured way to apply a sequence of user-defined functions (UDFs).
|
| Parameters
| ----------
| function
| Callable; will receive the expression as the first parameter,
| followed by any given args/kwargs.
| *args
| Arguments to pass to the UDF.
| **kwargs
| Keyword arguments to pass to the UDF.
|
| Examples
| --------
| >>> def extract_number(expr: pl.Expr) -> pl.Expr:
| ... """Extract the digits from a string."""
| ... return expr.str.extract(r"\d+", 0).cast(pl.Int64)
| >>>
| >>> def scale_negative_even(expr: pl.Expr, *, n: int = 1) -> pl.Expr:
| ... """Set even numbers negative, and scale by a user-supplied value."""
| ... expr = pl.when(expr % 2 == 0).then(-expr).otherwise(expr)
| ... return expr * n
| >>>
| >>> df = pl.DataFrame({"val": ["a: 1", "b: 2", "c: 3", "d: 4"]})
| >>> df.with_columns(
| ... udfs=(
| ... pl.col("val").pipe(extract_number).pipe(scale_negative_even, n=5)
| ... ),
| ... )
| shape: (4, 2)
| ┌──────┬──────┐
| │ val ┆ udfs │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞══════╪══════╡
| │ a: 1 ┆ 5 │
| │ b: 2 ┆ -10 │
| │ c: 3 ┆ 15 │
| │ d: 4 ┆ -20 │
| └──────┴──────┘
|
| pow(self, exponent: 'IntoExprColumn | int | float') -> 'Expr'
| Method equivalent of exponentiation operator `expr ** exponent`.
|
| If the exponent is float, the result follows the dtype of exponent.
| Otherwise, it follows dtype of base.
|
| Parameters
| ----------
| exponent
| Numeric literal or expression exponent value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 4, 8]})
| >>> df.with_columns(
| ... pl.col("x").pow(3).alias("cube"),
| ... pl.col("x").pow(pl.col("x").log(2)).alias("x ** xlog2"),
| ... )
| shape: (4, 3)
| ┌─────┬──────┬────────────┐
| │ x ┆ cube ┆ x ** xlog2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪══════╪════════════╡
| │ 1 ┆ 1 ┆ 1.0 │
| │ 2 ┆ 8 ┆ 2.0 │
| │ 4 ┆ 64 ┆ 16.0 │
| │ 8 ┆ 512 ┆ 512.0 │
| └─────┴──────┴────────────┘
|
| Raising an integer to a positive integer results in an integer - in order
| to raise to a negative integer, you can cast either the base or the exponent
| to float first:
|
| >>> df.with_columns(
| ... x_squared=pl.col("x").pow(2),
| ... x_inverse=pl.col("x").pow(-1.0),
| ... )
| shape: (4, 3)
| ┌─────┬───────────┬───────────┐
| │ x ┆ x_squared ┆ x_inverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪═══════════╪═══════════╡
| │ 1 ┆ 1 ┆ 1.0 │
| │ 2 ┆ 4 ┆ 0.5 │
| │ 4 ┆ 16 ┆ 0.25 │
| │ 8 ┆ 64 ┆ 0.125 │
| └─────┴───────────┴───────────┘
|
| product(self) -> 'Expr'
| Compute the product of an expression.
|
| Notes
| -----
| If there are no non-null values, then the output is `1`.
| If you would prefer empty products to return `None`, you can
| use `pl.when(expr.count()>0).then(expr.product())` instead
| of `expr.product()`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").product())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 6 │
| └─────┘
|
| qcut(self, quantiles: 'Sequence[float] | int', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, allow_duplicates: 'bool' = False, include_breaks: 'bool' = False) -> 'Expr'
| Bin continuous values into discrete categories based on their quantiles.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| quantiles
| Either a list of quantile probabilities between 0 and 1 or a positive
| integer determining the number of bins with uniform probability.
| labels
| Names of the categories. The number of labels must be equal to the number
| of categories.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| allow_duplicates
| If set to `True`, duplicates in the resulting quantiles are dropped,
| rather than raising a `DuplicateError`. This can happen even with unique
| probabilities, depending on the data.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from a
| :class:`Categorical` to a :class:`Struct`.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Categorical` if `include_breaks` is set to
| `False` (default), otherwise an expression of data type :class:`Struct`.
|
| See Also
| --------
| cut
|
| Examples
| --------
| Divide a column into three categories according to pre-defined quantile
| probabilities.
|
| >>> df = pl.DataFrame({"foo": [-2, -1, 0, 1, 2]})
| >>> df.with_columns(
| ... pl.col("foo").qcut([0.25, 0.75], labels=["a", "b", "c"]).alias("qcut")
| ... )
| shape: (5, 2)
| ┌─────┬──────┐
| │ foo ┆ qcut │
| │ --- ┆ --- │
| │ i64 ┆ cat │
| ╞═════╪══════╡
| │ -2 ┆ a │
| │ -1 ┆ a │
| │ 0 ┆ b │
| │ 1 ┆ b │
| │ 2 ┆ c │
| └─────┴──────┘
|
| Divide a column into two categories using uniform quantile probabilities.
|
| >>> df.with_columns(
| ... pl.col("foo")
| ... .qcut(2, labels=["low", "high"], left_closed=True)
| ... .alias("qcut")
| ... )
| shape: (5, 2)
| ┌─────┬──────┐
| │ foo ┆ qcut │
| │ --- ┆ --- │
| │ i64 ┆ cat │
| ╞═════╪══════╡
| │ -2 ┆ low │
| │ -1 ┆ low │
| │ 0 ┆ high │
| │ 1 ┆ high │
| │ 2 ┆ high │
| └─────┴──────┘
|
| Add both the category and the breakpoint.
|
| >>> df.with_columns(
| ... pl.col("foo").qcut([0.25, 0.75], include_breaks=True).alias("qcut")
| ... ).unnest("qcut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ cat │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| quantile(self, quantile: 'float | list_[float] | Expr', interpolation: 'QuantileMethod' = 'nearest') -> 'Expr'
| Get quantile value.
|
| Parameters
| ----------
| quantile
| Quantile(s) between 0.0 and 1.0. Can be a single float or a list of floats.
|
| - If a single float, returns a single f64 value per row.
| - If a list of floats, returns a list of f64 values per row (one value per quantile).
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0, 1, 2, 3, 4, 5]})
| >>> df.select(pl.col("a").quantile(0.3))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 2.0 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="higher"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 2.0 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="lower"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="midpoint"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.5 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="linear"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.5 │
| └─────┘
| >>> df.select(pl.col("a").quantile([0.25, 0.75], interpolation="linear"))
| shape: (1, 1)
| ┌──────────────┐
| │ a │
| │ --- │
| │ list[f64] │
| ╞══════════════╡
| │ [1.25, 3.75] │
| └──────────────┘
|
| radians(self) -> 'Expr'
| Convert from degrees to radians.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-720, -540, -360, -180, 0, 180, 360, 540, 720]})
| >>> df.select(pl.col("a").radians())
| shape: (9, 1)
| ┌────────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════════╡
| │ -12.566371 │
| │ -9.424778 │
| │ -6.283185 │
| │ -3.141593 │
| │ 0.0 │
| │ 3.141593 │
| │ 6.283185 │
| │ 9.424778 │
| │ 12.566371 │
| └────────────┘
|
| rank(self, method: 'RankMethod' = 'average', *, descending: 'bool' = False, seed: 'int | None' = None) -> 'Expr'
| Assign ranks to data, dealing with ties appropriately.
|
| Parameters
| ----------
| method : {'average', 'min', 'max', 'dense', 'ordinal', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'ordinal' : All values are given a distinct rank, corresponding to
| the order that the values occur in the Series.
| - 'random' : Like 'ordinal', but the rank for ties is not dependent
| on the order that the values occur in the Series.
| descending
| Rank in descending order.
| seed
| If `method="random"`, use this as seed.
|
| Notes
| -----
| If you're coming from SQL, you may be expecting null values to be ranked last.
| Polars, however, only ranks non-null values and preserves the null ones.
|
| Examples
| --------
| The 'average' method:
|
| >>> df = pl.DataFrame({"a": [3, 6, 1, 1, 6]})
| >>> df.select(pl.col("a").rank())
| shape: (5, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 3.0 │
| │ 4.5 │
| │ 1.5 │
| │ 1.5 │
| │ 4.5 │
| └─────┘
|
| The 'ordinal' method:
|
| >>> df = pl.DataFrame({"a": [3, 6, 1, 1, 6]})
| >>> df.select(pl.col("a").rank("ordinal"))
| shape: (5, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 3 │
| │ 4 │
| │ 1 │
| │ 2 │
| │ 5 │
| └─────┘
|
| Use 'rank' with 'over' to rank within groups:
|
| >>> df = pl.DataFrame({"a": [1, 1, 2, 2, 2], "b": [6, 7, 5, 14, 11]})
| >>> df.with_columns(pl.col("b").rank().over("a").alias("rank"))
| shape: (5, 3)
| ┌─────┬─────┬──────┐
| │ a ┆ b ┆ rank │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 6 ┆ 1.0 │
| │ 1 ┆ 7 ┆ 2.0 │
| │ 2 ┆ 5 ┆ 1.0 │
| │ 2 ┆ 14 ┆ 3.0 │
| │ 2 ┆ 11 ┆ 2.0 │
| └─────┴─────┴──────┘
|
| Divide by the length or number of non-null values
| to compute the percentile rank.
|
| >>> df = pl.DataFrame({"a": [6, 7, None, 14, 11]})
| >>> df.with_columns(
| ... pct=pl.col("a").rank() / pl.len(),
| ... pct_valid=pl.col("a").rank() / pl.count("a"),
| ... )
| shape: (5, 3)
| ┌──────┬──────┬───────────┐
| │ a ┆ pct ┆ pct_valid │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪═══════════╡
| │ 6 ┆ 0.2 ┆ 0.25 │
| │ 7 ┆ 0.4 ┆ 0.5 │
| │ null ┆ null ┆ null │
| │ 14 ┆ 0.8 ┆ 1.0 │
| │ 11 ┆ 0.6 ┆ 0.75 │
| └──────┴──────┴───────────┘
|
| rechunk(self) -> 'Expr'
| Create a single chunk of memory for this Series.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
|
| Create a Series with 3 nulls, append column `a`, then rechunk.
|
| >>> df.select(pl.repeat(None, 3).append(pl.col("a")).rechunk())
| shape: (6, 1)
| ┌────────┐
| │ repeat │
| │ --- │
| │ i64 │
| ╞════════╡
| │ null │
| │ null │
| │ null │
| │ 1 │
| │ 1 │
| │ 2 │
| └────────┘
|
| register_plugin(self, *, lib: 'str_', symbol: 'str_', args: 'list_[IntoExpr] | None' = None, kwargs: 'dict[Any, Any] | None' = None, is_elementwise: 'bool' = False, input_wildcard_expansion: 'bool' = False, returns_scalar: 'bool' = False, cast_to_supertypes: 'bool' = False, pass_name_to_apply: 'bool' = False, changes_length: 'bool' = False) -> 'Expr'
| Register a plugin function.
|
| .. deprecated:: 0.20.16
| Use :func:`polars.plugins.register_plugin_function` instead.
|
| See the `user guide <https://docs.pola.rs/user-guide/plugins/>`_
| for more information about plugins.
|
| Warnings
| --------
| This method is deprecated. Use the new `polars.plugins.register_plugin_function`
| function instead.
|
| This is highly unsafe as this will call the C function loaded by
| `lib::symbol`.
|
| The parameters you set dictate how Polars will handle the function.
| Make sure they are correct!
|
| Parameters
| ----------
| lib
| Library to load.
| symbol
| Function to load.
| args
| Arguments (other than self) passed to this function.
| These arguments have to be of type Expression.
| kwargs
| Non-expression arguments. They must be JSON serializable.
| is_elementwise
| If the function only operates on scalars
| this will trigger fast paths.
| input_wildcard_expansion
| Expand expressions as input of this function.
| returns_scalar
| Automatically explode on unit length if it ran as final aggregation.
| this is the case for aggregations like `sum`, `min`, `covariance` etc.
| cast_to_supertypes
| Cast the input datatypes to their supertype.
| pass_name_to_apply
| if set, then the `Series` passed to the function in the group_by operation
| will ensure the name is set. This is an extra heap allocation per group.
| changes_length
| For example a `unique` or a `slice`
|
| reinterpret(self, *, signed: 'bool | None' = None, dtype: 'PolarsDataType | None' = None) -> 'Expr'
| Reinterpret the underlying bits as a signed/unsigned integer or float.
|
| This operation is only allowed for numeric types of the same size.
| For lower bits numbers, you can safely use the cast operation.
|
| Either `signed` or `dtype` can be specified.
| Defaults to `signed=True` otherwise.
|
| Parameters
| ----------
| signed
| If True, reinterpret as signed integer. Otherwise, reinterpret
| as unsigned integer.
| dtype
| DataType to reinterpret to.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 1, 2], dtype=pl.UInt64)
| >>> df = pl.DataFrame([s])
| >>> df.select(
| ... [
| ... pl.col("a").reinterpret(dtype=pl.Int64).alias("reinterpreted"),
| ... pl.col("a").alias("original"),
| ... ]
| ... )
| shape: (3, 2)
| ┌───────────────┬──────────┐
| │ reinterpreted ┆ original │
| │ --- ┆ --- │
| │ i64 ┆ u64 │
| ╞═══════════════╪══════════╡
| │ 1 ┆ 1 │
| │ 1 ┆ 1 │
| │ 2 ┆ 2 │
| └───────────────┴──────────┘
|
| repeat_by(self, by: 'pl.Series | Expr | str_ | int') -> 'Expr'
| Repeat the elements in this Series as specified in the given expression.
|
| The repeated elements are expanded into a `List`.
|
| Parameters
| ----------
| by
| Numeric column that determines how often the values will be repeated.
| The column will be coerced to UInt32. Give this dtype to make the coercion a
| no-op.
|
| Returns
| -------
| Expr
| Expression of data type :class:`List`, where the inner data type is equal
| to the original data type.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["x", "y", "z"],
| ... "n": [1, 2, 3],
| ... }
| ... )
| >>> df.select(pl.col("a").repeat_by("n"))
| shape: (3, 1)
| ┌─────────────────┐
| │ a │
| │ --- │
| │ list[str] │
| ╞═════════════════╡
| │ ["x"] │
| │ ["y", "y"] │
| │ ["z", "z", "z"] │
| └─────────────────┘
|
| replace(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | None' = None) -> 'Expr'
| Replace the given values by different values of the same data type.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Length must match the length of `old` or have length 1.
|
| default
| Set values that were not replaced to this value.
| Defaults to keeping the original value.
| Accepts expression input. Non-expression inputs are parsed as literals.
|
| .. deprecated:: 1.0.0
| Use :meth:`replace_strict` instead to set a default while replacing
| values.
|
| return_dtype
| The data type of the resulting expression. If set to `None` (default),
| the data type of the original column is preserved.
|
| .. deprecated:: 1.0.0
| Use :meth:`replace_strict` instead to set a return data type while
| replacing values, or explicitly call :meth:`cast` on the output.
|
| See Also
| --------
| replace_strict
| str.replace
|
| Examples
| --------
| Replace a single value by another value. Values that were not replaced remain
| unchanged.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3]})
| >>> df.with_columns(replaced=pl.col("a").replace(2, 100))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 100 │
| │ 2 ┆ 100 │
| │ 3 ┆ 3 │
| └─────┴──────────┘
|
| Replace multiple values by passing sequences to the `old` and `new` parameters.
|
| >>> df.with_columns(replaced=pl.col("a").replace([2, 3], [100, 200]))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 100 │
| │ 2 ┆ 100 │
| │ 3 ┆ 200 │
| └─────┴──────────┘
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {2: 100, 3: 200}
| >>> df.with_columns(replaced=pl.col("a").replace(mapping))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 100 │
| │ 2 ┆ 100 │
| │ 3 ┆ 200 │
| └─────┴──────────┘
|
| The original data type is preserved when replacing by values of a different
| data type. Use :meth:`replace_strict` to replace and change the return data
| type.
|
| >>> df = pl.DataFrame({"a": ["x", "y", "z"]})
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> df.with_columns(replaced=pl.col("a").replace(mapping))
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
|
| Expression input is supported.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3], "b": [1.5, 2.5, 5.0, 1.0]})
| >>> df.with_columns(
| ... replaced=pl.col("a").replace(
| ... old=pl.col("a").max(),
| ... new=pl.col("b").sum(),
| ... )
| ... )
| shape: (4, 3)
| ┌─────┬─────┬──────────┐
| │ a ┆ b ┆ replaced │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ i64 │
| ╞═════╪═════╪══════════╡
| │ 1 ┆ 1.5 ┆ 1 │
| │ 2 ┆ 2.5 ┆ 2 │
| │ 2 ┆ 5.0 ┆ 2 │
| │ 3 ┆ 1.0 ┆ 10 │
| └─────┴─────┴──────────┘
|
| replace_strict(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None) -> 'Expr'
| Replace all values by different values.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace_strict(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Length must match the length of `old` or have length 1.
| default
| Set values that were not replaced to this value. If no default is specified,
| (default), an error is raised if any values were not replaced.
| Accepts expression input. Non-expression inputs are parsed as literals.
| return_dtype
| The data type of the resulting expression. If set to `None` (default),
| the data type is determined automatically based on the other inputs.
|
| Raises
| ------
| InvalidOperationError
| If any non-null values in the original column were not replaced, and no
| `default` was specified.
|
| See Also
| --------
| replace
| str.replace
|
| Examples
| --------
| Replace values by passing sequences to the `old` and `new` parameters.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3]})
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict([1, 2, 3], [100, 200, 300])
| ... )
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 100 │
| │ 2 ┆ 200 │
| │ 2 ┆ 200 │
| │ 3 ┆ 300 │
| └─────┴──────────┘
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {1: 100, 2: 200, 3: 300}
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 100 │
| │ 2 ┆ 200 │
| │ 2 ┆ 200 │
| │ 3 ┆ 300 │
| └─────┴──────────┘
|
| By default, an error is raised if any non-null values were not replaced.
| Specify a default to set all values that were not matched.
|
| >>> mapping = {2: 200, 3: 300}
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict(mapping)
| ... ) # doctest: +SKIP
| Traceback (most recent call last):
| ...
| polars.exceptions.InvalidOperationError: incomplete mapping specified for `replace_strict`
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping, default=-1))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ -1 │
| │ 2 ┆ 200 │
| │ 2 ┆ 200 │
| │ 3 ┆ 300 │
| └─────┴──────────┘
|
| Replacing by values of a different data type sets the return type based on
| a combination of the `new` data type and the `default` data type.
|
| >>> df = pl.DataFrame({"a": ["x", "y", "z"]})
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping))
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping, default="x"))
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
|
| Set the `return_dtype` parameter to control the resulting data type directly.
|
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict(mapping, return_dtype=pl.UInt8)
| ... )
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ u8 │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
|
| Expression input is supported for all parameters.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3], "b": [1.5, 2.5, 5.0, 1.0]})
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict(
| ... old=pl.col("a").max(),
| ... new=pl.col("b").sum(),
| ... default=pl.col("b"),
| ... )
| ... )
| shape: (4, 3)
| ┌─────┬─────┬──────────┐
| │ a ┆ b ┆ replaced │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 │
| ╞═════╪═════╪══════════╡
| │ 1 ┆ 1.5 ┆ 1.5 │
| │ 2 ┆ 2.5 ┆ 2.5 │
| │ 2 ┆ 5.0 ┆ 5.0 │
| │ 3 ┆ 1.0 ┆ 10.0 │
| └─────┴─────┴──────────┘
|
| reshape(self, dimensions: 'tuple[int, ...]') -> 'Expr'
| Reshape this Expr to a flat column or an Array column.
|
| Parameters
| ----------
| dimensions
| Tuple of the dimension sizes. If -1 is used as the value for the
| first dimension, that dimension is inferred.
| Because the size of the Column may not be known in advance, it is
| only possible to use -1 for the first dimension.
|
| Returns
| -------
| Expr
| If a single dimension is given, results in an expression of the original
| data type.
| If a multiple dimensions are given, results in an expression of data type
| :class:`Array` with shape `dimensions`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7, 8, 9]})
| >>> square = df.select(pl.col("foo").reshape((3, 3)))
| >>> square
| shape: (3, 1)
| ┌───────────────┐
| │ foo │
| │ --- │
| │ array[i64, 3] │
| ╞═══════════════╡
| │ [1, 2, 3] │
| │ [4, 5, 6] │
| │ [7, 8, 9] │
| └───────────────┘
| >>> square.select(pl.col("foo").reshape((9,)))
| shape: (9, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| │ 4 │
| │ 5 │
| │ 6 │
| │ 7 │
| │ 8 │
| │ 9 │
| └─────┘
|
| See Also
| --------
| Expr.list.explode : Explode a list column.
|
| reverse(self) -> 'Expr'
| Reverse the selection.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1, 2, 3, 4, 5],
| ... "fruits": ["banana", "banana", "apple", "apple", "banana"],
| ... "B": [5, 4, 3, 2, 1],
| ... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"],
| ... }
| ... )
| >>> df.select(
| ... [
| ... pl.all(),
| ... pl.all().reverse().name.suffix("_reverse"),
| ... ]
| ... )
| shape: (5, 8)
| ┌─────┬────────┬─────┬────────┬───────────┬────────────────┬───────────┬──────────────┐
| │ A ┆ fruits ┆ B ┆ cars ┆ A_reverse ┆ fruits_reverse ┆ B_reverse ┆ cars_reverse │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ i64 ┆ str ┆ i64 ┆ str ┆ i64 ┆ str │
| ╞═════╪════════╪═════╪════════╪═══════════╪════════════════╪═══════════╪══════════════╡
| │ 1 ┆ banana ┆ 5 ┆ beetle ┆ 5 ┆ banana ┆ 1 ┆ beetle │
| │ 2 ┆ banana ┆ 4 ┆ audi ┆ 4 ┆ apple ┆ 2 ┆ beetle │
| │ 3 ┆ apple ┆ 3 ┆ beetle ┆ 3 ┆ apple ┆ 3 ┆ beetle │
| │ 4 ┆ apple ┆ 2 ┆ beetle ┆ 2 ┆ banana ┆ 4 ┆ audi │
| │ 5 ┆ banana ┆ 1 ┆ beetle ┆ 1 ┆ banana ┆ 5 ┆ beetle │
| └─────┴────────┴─────┴────────┴───────────┴────────────────┴───────────┴──────────────┘
|
| rle(self) -> 'Expr'
| Compress the column data using run-length encoding.
|
| Run-length encoding (RLE) encodes data by storing each *run* of identical values
| as a single value and its length.
|
| Returns
| -------
| Expr
| Expression of data type `Struct` with fields `len` of data type `UInt32`
| and `value` of the original data type.
|
| See Also
| --------
| rle_id
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 1, None, 1, 3, 3]})
| >>> df.select(pl.col("a").rle()).unnest("a")
| shape: (6, 2)
| ┌─────┬───────┐
| │ len ┆ value │
| │ --- ┆ --- │
| │ u32 ┆ i64 │
| ╞═════╪═══════╡
| │ 2 ┆ 1 │
| │ 1 ┆ 2 │
| │ 1 ┆ 1 │
| │ 1 ┆ null │
| │ 1 ┆ 1 │
| │ 2 ┆ 3 │
| └─────┴───────┘
|
| rle_id(self) -> 'Expr'
| Get a distinct integer ID for each run of identical values.
|
| The ID starts at 0 and increases by one each time the value of the column
| changes.
|
| Returns
| -------
| Expr
| Expression of data type `UInt32`.
|
| See Also
| --------
| rle
|
| Notes
| -----
| This functionality is especially useful for defining a new group for every time
| a column's value changes, rather than for every distinct value of that column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 1, 1, 1],
| ... "b": ["x", "x", None, "y", "y"],
| ... }
| ... )
| >>> df.with_columns(
| ... rle_id_a=pl.col("a").rle_id(),
| ... rle_id_ab=pl.struct("a", "b").rle_id(),
| ... )
| shape: (5, 4)
| ┌─────┬──────┬──────────┬───────────┐
| │ a ┆ b ┆ rle_id_a ┆ rle_id_ab │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ u32 ┆ u32 │
| ╞═════╪══════╪══════════╪═══════════╡
| │ 1 ┆ x ┆ 0 ┆ 0 │
| │ 2 ┆ x ┆ 1 ┆ 1 │
| │ 1 ┆ null ┆ 2 ┆ 2 │
| │ 1 ┆ y ┆ 2 ┆ 3 │
| │ 1 ┆ y ┆ 2 ┆ 3 │
| └─────┴──────┴──────────┴───────────┘
|
| rolling(self, index_column: 'IntoExprColumn', *, period: 'str_ | timedelta', offset: 'str_ | timedelta | None' = None, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Create rolling groups based on a temporal or integer column.
|
| If you have a time series `<t_0, t_1, ..., t_n>`, then by default the
| windows created will be
|
| * (t_0 - period, t_0]
| * (t_1 - period, t_1]
| * ...
| * (t_n - period, t_n]
|
| whereas if you pass a non-default `offset`, then the windows will be
|
| * (t_0 + offset, t_0 + offset + period]
| * (t_1 + offset, t_1 + offset + period]
| * ...
| * (t_n + offset, t_n + offset + period]
|
| The `period` and `offset` arguments are created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order.
| In case of a rolling group by on indices, dtype needs to be one of
| {UInt32, UInt64, Int32, Int64}. Note that the first three get temporarily
| cast to Int64, so if performance matters use an Int64 column.
| period
| Length of the window - must be non-negative.
| offset
| Offset of the window. Default is `-period`.
| closed : {'right', 'left', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
|
| Examples
| --------
| >>> dates = [
| ... "2020-01-01 13:45:48",
| ... "2020-01-01 16:42:13",
| ... "2020-01-01 16:45:09",
| ... "2020-01-02 18:12:48",
| ... "2020-01-03 19:45:32",
| ... "2020-01-08 23:16:43",
| ... ]
| >>> df = pl.DataFrame({"dt": dates, "a": [3, 7, 5, 9, 2, 1]}).with_columns(
| ... pl.col("dt").str.strptime(pl.Datetime).set_sorted()
| ... )
| >>> df.with_columns(
| ... sum_a=pl.sum("a").rolling(index_column="dt", period="2d"),
| ... min_a=pl.min("a").rolling(index_column="dt", period="2d"),
| ... max_a=pl.max("a").rolling(index_column="dt", period="2d"),
| ... )
| shape: (6, 5)
| ┌─────────────────────┬─────┬───────┬───────┬───────┐
| │ dt ┆ a ┆ sum_a ┆ min_a ┆ max_a │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════════════════════╪═════╪═══════╪═══════╪═══════╡
| │ 2020-01-01 13:45:48 ┆ 3 ┆ 3 ┆ 3 ┆ 3 │
| │ 2020-01-01 16:42:13 ┆ 7 ┆ 10 ┆ 3 ┆ 7 │
| │ 2020-01-01 16:45:09 ┆ 5 ┆ 15 ┆ 3 ┆ 7 │
| │ 2020-01-02 18:12:48 ┆ 9 ┆ 24 ┆ 3 ┆ 9 │
| │ 2020-01-03 19:45:32 ┆ 2 ┆ 11 ┆ 2 ┆ 9 │
| │ 2020-01-08 23:16:43 ┆ 1 ┆ 1 ┆ 1 ┆ 1 │
| └─────────────────────┴─────┴───────┴───────┴───────┘
|
| rolling_kurtosis(self, window_size: 'int', *, fisher: 'bool' = True, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling kurtosis.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Expr.kurtosis
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 4, 2, 9]})
| >>> df.select(pl.col("a").rolling_kurtosis(3))
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ null │
| │ -1.5 │
| │ -1.5 │
| └──────┘
|
| rolling_map(self, function: 'Callable[[Series], Any]', window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a custom rolling window function.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| function
| Custom aggregation function.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Warnings
| --------
| Computing custom functions is extremely slow. Use specialized rolling
| functions such as :func:`Expr.rolling_sum` if at all possible.
|
| Examples
| --------
| >>> from numpy import nansum
| >>> df = pl.DataFrame({"a": [11.0, 2.0, 9.0, float("nan"), 8.0]})
| >>> df.select(pl.col("a").rolling_map(nansum, window_size=3))
| shape: (5, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ null │
| │ 22.0 │
| │ 11.0 │
| │ 17.0 │
| └──────┘
|
| rolling_max(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling max (moving max) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their max.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_max=pl.col("A").rolling_max(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_max │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 2.0 │
| │ 3.0 ┆ 3.0 │
| │ 4.0 ┆ 4.0 │
| │ 5.0 ┆ 5.0 │
| │ 6.0 ┆ 6.0 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_max=pl.col("A").rolling_max(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_max │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.25 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ 3.75 │
| │ 6.0 ┆ 4.5 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_max=pl.col("A").rolling_max(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_max │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 3.0 │
| │ 3.0 ┆ 4.0 │
| │ 4.0 ┆ 5.0 │
| │ 5.0 ┆ 6.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_max_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling max based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling max with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_max=pl.col("index").rolling_max_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_max │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 2 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 3 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 4 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 20 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 21 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 22 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 23 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 24 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling max with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_max=pl.col("index").rolling_max_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_max │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 2 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 3 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 4 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 20 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 21 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 22 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 23 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 24 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_mean(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling mean (moving mean) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their mean. Weights
| are normalized to sum to 1.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window, after being normalized to sum to
| 1.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_mean=pl.col("A").rolling_mean(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────┐
| │ A ┆ rolling_mean │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.5 │
| │ 4.0 ┆ 3.5 │
| │ 5.0 ┆ 4.5 │
| │ 6.0 ┆ 5.5 │
| └─────┴──────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_mean=pl.col("A").rolling_mean(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────┐
| │ A ┆ rolling_mean │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.75 │
| │ 3.0 ┆ 2.75 │
| │ 4.0 ┆ 3.75 │
| │ 5.0 ┆ 4.75 │
| │ 6.0 ┆ 5.75 │
| └─────┴──────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_mean=pl.col("A").rolling_mean(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────┐
| │ A ┆ rolling_mean │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 2.0 │
| │ 3.0 ┆ 3.0 │
| │ 4.0 ┆ 4.0 │
| │ 5.0 ┆ 5.0 │
| │ 6.0 ┆ null │
| └─────┴──────────────┘
|
| rolling_mean_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling mean based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling mean with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_mean=pl.col("index").rolling_mean_by(
| ... "date", window_size="2h"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬──────────────────┐
| │ index ┆ date ┆ rolling_row_mean │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪══════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.5 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.5 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.5 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.5 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.5 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.5 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.5 │
| └───────┴─────────────────────┴──────────────────┘
|
| Compute the rolling mean with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_mean=pl.col("index").rolling_mean_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬──────────────────┐
| │ index ┆ date ┆ rolling_row_mean │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪══════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.0 │
| └───────┴─────────────────────┴──────────────────┘
|
| rolling_median(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling median.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their median.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_median=pl.col("A").rolling_median(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬────────────────┐
| │ A ┆ rolling_median │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.5 │
| │ 4.0 ┆ 3.5 │
| │ 5.0 ┆ 4.5 │
| │ 6.0 ┆ 5.5 │
| └─────┴────────────────┘
|
| Specify weights for the values in each window:
|
| >>> df.with_columns(
| ... rolling_median=pl.col("A").rolling_median(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬────────────────┐
| │ A ┆ rolling_median │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.5 │
| │ 4.0 ┆ 3.5 │
| │ 5.0 ┆ 4.5 │
| │ 6.0 ┆ 5.5 │
| └─────┴────────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_median=pl.col("A").rolling_median(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬────────────────┐
| │ A ┆ rolling_median │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 2.0 │
| │ 3.0 ┆ 3.0 │
| │ 4.0 ┆ 4.0 │
| │ 5.0 ┆ 5.0 │
| │ 6.0 ┆ null │
| └─────┴────────────────┘
|
| rolling_median_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Compute a rolling median based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling median with the temporal windows closed on the right:
|
| >>> df_temporal.with_columns(
| ... rolling_row_median=pl.col("index").rolling_median_by(
| ... "date", window_size="2h"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬────────────────────┐
| │ index ┆ date ┆ rolling_row_median │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.5 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.5 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.5 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.5 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.5 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.5 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.5 │
| └───────┴─────────────────────┴────────────────────┘
|
| rolling_min(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling min (moving min) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their min.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_min=pl.col("A").rolling_min(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_min │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 2.0 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ 4.0 │
| │ 6.0 ┆ 5.0 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_min=pl.col("A").rolling_min(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_min │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.25 │
| │ 3.0 ┆ 0.5 │
| │ 4.0 ┆ 0.75 │
| │ 5.0 ┆ 1.0 │
| │ 6.0 ┆ 1.25 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_min=pl.col("A").rolling_min(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_min │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 2.0 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ 4.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_min_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling min based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling min with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_min=pl.col("index").rolling_min_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_min │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_quantile(self, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', window_size: 'int' = 2, weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling quantile.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their quantile.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.25, window_size=4
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ null │
| │ 4.0 ┆ 2.0 │
| │ 5.0 ┆ 3.0 │
| │ 6.0 ┆ 4.0 │
| └─────┴──────────────────┘
|
| Specify weights for the values in each window:
|
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.25, window_size=4, weights=[0.2, 0.4, 0.4, 0.2]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ null │
| │ 4.0 ┆ 2.0 │
| │ 5.0 ┆ 3.0 │
| │ 6.0 ┆ 4.0 │
| └─────┴──────────────────┘
|
| Specify weights and interpolation method
|
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.25,
| ... window_size=4,
| ... weights=[0.2, 0.4, 0.4, 0.2],
| ... interpolation="linear",
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ null │
| │ 4.0 ┆ 1.625 │
| │ 5.0 ┆ 2.625 │
| │ 6.0 ┆ 3.625 │
| └─────┴──────────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.2, window_size=5, center=True
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ 2.0 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ null │
| │ 6.0 ┆ null │
| └─────┴──────────────────┘
|
| rolling_quantile_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Compute a rolling quantile based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling quantile with the temporal windows closed on the right:
|
| >>> df_temporal.with_columns(
| ... rolling_row_quantile=pl.col("index").rolling_quantile_by(
| ... "date", window_size="2h", quantile=0.3
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬──────────────────────┐
| │ index ┆ date ┆ rolling_row_quantile │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪══════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.0 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.0 │
| └───────┴─────────────────────┴──────────────────────┘
|
| rolling_rank(self, window_size: 'int', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling rank.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| A window of length `window_size` will traverse the array. The values
| that fill this window will be ranked according to the `method`
| parameter. The resulting values will be the rank of the value that is
| at the end of the sliding window.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Returns
| -------
| Expr
| An Expr of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 4, 4, 1, 9]})
| >>> df.select(pl.col("a").rolling_rank(3, method="average"))
| shape: (5, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ null │
| │ 2.5 │
| │ 1.0 │
| │ 3.0 │
| └──────┘
|
| rolling_rank_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Compute a rolling rank based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Returns
| -------
| Expr
| An Expr of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| rolling_skew(self, window_size: 'int', *, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling skew.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| bias
| If False, the calculations are corrected for statistical bias.
| bias: bool = True,
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Expr.skew
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 4, 2, 9]})
| >>> df.select(pl.col("a").rolling_skew(3))
| shape: (4, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ null │
| │ null │
| │ 0.381802 │
| │ 0.47033 │
| └──────────┘
|
| Note how the values match the following:
|
| >>> pl.Series([1, 4, 2]).skew(), pl.Series([4, 2, 9]).skew()
| (0.38180177416060584, 0.47033046033698594)
|
| rolling_std(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Expr'
| Compute a rolling standard deviation.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their std. Weights
| are normalized to sum to 1.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window after being normalized to sum to
| 1.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_std=pl.col("A").rolling_std(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_std │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.707107 │
| │ 3.0 ┆ 0.707107 │
| │ 4.0 ┆ 0.707107 │
| │ 5.0 ┆ 0.707107 │
| │ 6.0 ┆ 0.707107 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_std=pl.col("A").rolling_std(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_std │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.433013 │
| │ 3.0 ┆ 0.433013 │
| │ 4.0 ┆ 0.433013 │
| │ 5.0 ┆ 0.433013 │
| │ 6.0 ┆ 0.433013 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_std=pl.col("A").rolling_std(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_std │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 1.0 │
| │ 4.0 ┆ 1.0 │
| │ 5.0 ┆ 1.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_std_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Expr'
| Compute a rolling standard deviation based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling std with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_std=pl.col("index").rolling_std_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_std │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.707107 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 0.707107 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 0.707107 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 0.707107 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 0.707107 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 0.707107 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 0.707107 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 0.707107 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 0.707107 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling std with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_std=pl.col("index").rolling_std_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_std │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.707107 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 1.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 1.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 1.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 1.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 1.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 1.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 1.0 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_sum(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling sum (moving sum) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their sum.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_sum=pl.col("A").rolling_sum(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_sum │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 3.0 │
| │ 3.0 ┆ 5.0 │
| │ 4.0 ┆ 7.0 │
| │ 5.0 ┆ 9.0 │
| │ 6.0 ┆ 11.0 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_sum=pl.col("A").rolling_sum(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_sum │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.75 │
| │ 3.0 ┆ 2.75 │
| │ 4.0 ┆ 3.75 │
| │ 5.0 ┆ 4.75 │
| │ 6.0 ┆ 5.75 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_sum=pl.col("A").rolling_sum(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_sum │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 6.0 │
| │ 3.0 ┆ 9.0 │
| │ 4.0 ┆ 12.0 │
| │ 5.0 ┆ 15.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_sum_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 0, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling sum based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling sum with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_sum=pl.col("index").rolling_sum_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_sum │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 3 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 7 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 39 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 41 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 43 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 45 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 47 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling sum with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_sum=pl.col("index").rolling_sum_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_sum │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 3 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 6 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 9 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 57 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 60 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 63 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 66 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 69 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_var(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Expr'
| Compute a rolling variance.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their var. Weights
| are normalized to sum to 1.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window after being normalized to sum to
| 1.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_var=pl.col("A").rolling_var(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_var │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.5 │
| │ 3.0 ┆ 0.5 │
| │ 4.0 ┆ 0.5 │
| │ 5.0 ┆ 0.5 │
| │ 6.0 ┆ 0.5 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_var=pl.col("A").rolling_var(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_var │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.1875 │
| │ 3.0 ┆ 0.1875 │
| │ 4.0 ┆ 0.1875 │
| │ 5.0 ┆ 0.1875 │
| │ 6.0 ┆ 0.1875 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_var=pl.col("A").rolling_var(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_var │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 1.0 │
| │ 4.0 ┆ 1.0 │
| │ 5.0 ┆ 1.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_var_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Expr'
| Compute a rolling variance based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling var with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_var=pl.col("index").rolling_var_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_var │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 0.5 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 0.5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 0.5 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 0.5 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 0.5 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 0.5 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 0.5 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 0.5 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling var with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_var=pl.col("index").rolling_var_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_var │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 1.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 1.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 1.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 1.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 1.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 1.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 1.0 │
| └───────┴─────────────────────┴─────────────────┘
|
| round(self, decimals: 'int' = 0, mode: 'RoundMode' = 'half_to_even') -> 'Expr'
| Round underlying floating point data by `decimals` digits.
|
| Parameters
| ----------
| decimals
| Number of decimals to round by.
| mode : {'half_to_even', 'half_away_from_zero', 'to_zero'}
| The rounding strategy used. A "rounded value" is a value with at most
| `decimals` decimal places (e.g. integers when ``decimals=0``, multiples
| of 0.1 when ``decimals=1``, 0.01 when ``decimals=2``, and so on).
|
| Strategies that start with ``half_`` round all values to the *nearest*
| rounded value, only using the strategy to break ties when a value falls
| exactly between two rounded values (e.g. 0.5 when ``decimals=0``, 0.05
| when ``decimals=1``). Other rounding strategies specify explicitly
| which rounded value is chosen and always apply (not just for tiebreaks).
|
| * *half_to_even* (default)
| Round to the nearest value; break ties by choosing the nearest
| **even** value. For example, 0.5 rounds to 0, 1.5 rounds to 2,
| 2.5 rounds to 2. Also known as "banker's rounding"; this is the
| default because it tends to minimise cumulative rounding bias.
| * *half_away_from_zero*
| Round to the nearest value; break ties by rounding **away from
| zero**. For example, 0.5 rounds to 1, -0.5 rounds to -1, 2.5
| rounds to 3. Also known as "commercial rounding".
| * *to_zero*
| Always round (truncate) **towards zero**, discarding the fractional
| part beyond `decimals`. For example, 0.9 rounds to 0, -0.9 rounds
| to 0, 1.29 rounds to 1.2 (with ``decimals=1``). Equivalent to the
| :meth:`truncate` method.
|
| See Also
| --------
| ceil : Round up to the nearest integer.
| floor : Round down to the nearest integer.
| round_sig_figs : Round to a given number of significant figures.
| truncate : Truncate to a given number of decimals.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.33, 0.52, 1.02, 1.17]})
| >>> df.select(pl.col("a").round(1))
| shape: (4, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.3 │
| │ 0.5 │
| │ 1.0 │
| │ 1.2 │
| └─────┘
|
| >>> df = pl.DataFrame(
| ... {
| ... "f64": [-3.5, -2.5, -1.5, -0.5, 0.5, 1.5, 2.5, 3.5],
| ... "d": ["-3.5", "-2.5", "-1.5", "-0.5", "0.5", "1.5", "2.5", "3.5"],
| ... },
| ... schema_overrides={"d": pl.Decimal(scale=1)},
| ... )
| >>> df.with_columns(
| ... pl.all().round(mode="half_away_from_zero").name.suffix("_away"),
| ... pl.all().round(mode="half_to_even").name.suffix("_to_even"),
| ... )
| shape: (8, 6)
| ┌──────┬───────────────┬──────────┬───────────────┬─────────────┬───────────────┐
| │ f64 ┆ d ┆ f64_away ┆ d_away ┆ f64_to_even ┆ d_to_even │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ decimal[38,1] ┆ f64 ┆ decimal[38,1] ┆ f64 ┆ decimal[38,1] │
| ╞══════╪═══════════════╪══════════╪═══════════════╪═════════════╪═══════════════╡
| │ -3.5 ┆ -3.5 ┆ -4.0 ┆ -4.0 ┆ -4.0 ┆ -4.0 │
| │ -2.5 ┆ -2.5 ┆ -3.0 ┆ -3.0 ┆ -2.0 ┆ -2.0 │
| │ -1.5 ┆ -1.5 ┆ -2.0 ┆ -2.0 ┆ -2.0 ┆ -2.0 │
| │ -0.5 ┆ -0.5 ┆ -1.0 ┆ -1.0 ┆ -0.0 ┆ 0.0 │
| │ 0.5 ┆ 0.5 ┆ 1.0 ┆ 1.0 ┆ 0.0 ┆ 0.0 │
| │ 1.5 ┆ 1.5 ┆ 2.0 ┆ 2.0 ┆ 2.0 ┆ 2.0 │
| │ 2.5 ┆ 2.5 ┆ 3.0 ┆ 3.0 ┆ 2.0 ┆ 2.0 │
| │ 3.5 ┆ 3.5 ┆ 4.0 ┆ 4.0 ┆ 4.0 ┆ 4.0 │
| └──────┴───────────────┴──────────┴───────────────┴─────────────┴───────────────┘
|
| round_sig_figs(self, digits: 'int') -> 'Expr'
| Round to a number of significant figures.
|
| Parameters
| ----------
| digits
| Number of significant figures to round to.
|
| See Also
| --------
| ceil : Round up to the nearest integer.
| floor : Round down to the nearest integer.
| round : Round to a given number of decimals.
| truncate : Truncate to a given number of decimals.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.01234, 3.333, 1234.0]})
| >>> df.with_columns(pl.col("a").round_sig_figs(2).alias("round_sig_figs"))
| shape: (3, 2)
| ┌─────────┬────────────────┐
| │ a ┆ round_sig_figs │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════════╪════════════════╡
| │ 0.01234 ┆ 0.012 │
| │ 3.333 ┆ 3.3 │
| │ 1234.0 ┆ 1200.0 │
| └─────────┴────────────────┘
|
| sample(self, n: 'int | IntoExprColumn | None' = None, *, fraction: 'float | IntoExprColumn | None' = None, with_replacement: 'bool' = False, shuffle: 'bool | None' = None, seed: 'int | None' = None) -> 'Expr'
| Sample from this expression.
|
| Parameters
| ----------
| n
| Number of items to return. Cannot be used with `fraction`. Defaults to 1 if
| `fraction` is None.
| fraction
| Fraction of items to return. Cannot be used with `n`.
| with_replacement
| Allow values to be sampled more than once.
| shuffle
| Determines the order of the sampled elements.
| If True, sampled elements are explicitly shuffled.
| If False, the relative order of the sampled elements is preserved.
| (i.e. they appear in the same order as the original input).
| If None (default), no ordering guarantee; uses the most performant
| algorithm.
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated for each sample operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(
| ... pl.col("a").sample(
| ... fraction=1.0, with_replacement=True, shuffle=False, seed=1
| ... )
| ... )
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 3 │
| │ 3 │
| │ 1 │
| └─────┘
|
| search_sorted(self, element: 'IntoExpr | np.ndarray[Any, Any]', side: 'SearchSortedSide' = 'any', *, descending: 'bool' = False) -> 'Expr'
| Find indices where elements should be inserted to maintain order.
|
| .. math:: a[i-1] < v <= a[i]
|
| Parameters
| ----------
| element
| Expression or scalar value.
| side : {'any', 'left', 'right'}
| If 'any', the index of the first suitable location found is given.
| If 'left', the index of the leftmost suitable location found is given.
| If 'right', return the rightmost suitable location found is given.
| descending
| Boolean indicating whether the values are descending or not (they
| are required to be sorted either way).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "values": [1, 2, 3, 5],
| ... }
| ... )
| >>> df.select(
| ... [
| ... pl.col("values").search_sorted(0).alias("zero"),
| ... pl.col("values").search_sorted(3).alias("three"),
| ... pl.col("values").search_sorted(6).alias("six"),
| ... ]
| ... )
| shape: (1, 3)
| ┌──────┬───────┬─────┐
| │ zero ┆ three ┆ six │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞══════╪═══════╪═════╡
| │ 0 ┆ 2 ┆ 4 │
| └──────┴───────┴─────┘
|
| set_sorted(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Expr'
| Flags the expression as 'sorted'.
|
| Enables downstream code to user fast paths for sorted arrays. It is
| recommended to also set whether `nulls_last` is `True` or `False`, as
| this enables many internal optimizations.
|
| Parameters
| ----------
| descending
| Whether the `Series` order is descending.
| nulls_last
| Whether the nulls are at the end.
|
| Warnings
| --------
| This can lead to incorrect results if the data is NOT sorted!!
| Use with care!
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1, 2, 3]})
| >>> df.select(pl.col("values").set_sorted().max())
| shape: (1, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 3 │
| └────────┘
|
| shift(self, n: 'int | IntoExprColumn' = 1, *, fill_value: 'IntoExpr | None' = None) -> 'Expr'
| Shift values by the given number of indices.
|
| Parameters
| ----------
| n
| Number of indices to shift forward. If a negative value is passed, values
| are shifted in the opposite direction instead.
| fill_value
| Fill the resulting null values with this scalar value.
|
| Notes
| -----
| This method is similar to the `LAG` operation in SQL when the value for `n`
| is positive. With a negative value for `n`, it is similar to `LEAD`.
|
| See Also
| --------
| fill_null
|
| Examples
| --------
| By default, values are shifted forward by one index.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.with_columns(shift=pl.col("a").shift())
| shape: (4, 2)
| ┌─────┬───────┐
| │ a ┆ shift │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═══════╡
| │ 1 ┆ null │
| │ 2 ┆ 1 │
| │ 3 ┆ 2 │
| │ 4 ┆ 3 │
| └─────┴───────┘
|
| Pass a negative value to shift in the opposite direction instead.
|
| >>> df.with_columns(shift=pl.col("a").shift(-2))
| shape: (4, 2)
| ┌─────┬───────┐
| │ a ┆ shift │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═══════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| │ 3 ┆ null │
| │ 4 ┆ null │
| └─────┴───────┘
|
| Specify `fill_value` to fill the resulting null values.
|
| >>> df.with_columns(shift=pl.col("a").shift(-2, fill_value=100))
| shape: (4, 2)
| ┌─────┬───────┐
| │ a ┆ shift │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═══════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| │ 3 ┆ 100 │
| │ 4 ┆ 100 │
| └─────┴───────┘
|
| shrink_dtype(self) -> 'Expr'
| Shrink numeric columns to the minimal required datatype.
|
| Shrink to the dtype needed to fit the extrema of this [`Series`].
| This can be used to reduce memory pressure.
|
| .. versionchanged:: 1.33.0
| Deprecated and turned into a no-op. The operation does not match the
| Polars data-model during lazy execution since the output datatype
| cannot be known without inspecting the data.
|
| Use `Series.shrink_dtype` instead.
|
| Examples
| --------
| >>> pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": [1, 2, 2 << 32],
| ... "c": [-1, 2, 1 << 30],
| ... "d": [-112, 2, 112],
| ... "e": [-112, 2, 129],
| ... "f": ["a", "b", "c"],
| ... "g": [0.1, 1.32, 0.12],
| ... "h": [True, None, False],
| ... }
| ... ).select(pl.all().shrink_dtype()) # doctest: +SKIP
| shape: (3, 8)
| ┌─────┬────────────┬────────────┬──────┬──────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ d ┆ e ┆ f ┆ g ┆ h │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i8 ┆ i64 ┆ i32 ┆ i8 ┆ i16 ┆ str ┆ f32 ┆ bool │
| ╞═════╪════════════╪════════════╪══════╪══════╪═════╪══════╪═══════╡
| │ 1 ┆ 1 ┆ -1 ┆ -112 ┆ -112 ┆ a ┆ 0.1 ┆ true │
| │ 2 ┆ 2 ┆ 2 ┆ 2 ┆ 2 ┆ b ┆ 1.32 ┆ null │
| │ 3 ┆ 8589934592 ┆ 1073741824 ┆ 112 ┆ 129 ┆ c ┆ 0.12 ┆ false │
| └─────┴────────────┴────────────┴──────┴──────┴─────┴──────┴───────┘
|
| shuffle(self, seed: 'int | None' = None) -> 'Expr'
| Shuffle the contents of this expression.
|
| Note this is shuffled independently of any other column or Expression. If you
| want each row to stay the same use df.sample(shuffle=True)
|
| Parameters
| ----------
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated each time the shuffle is called.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").shuffle(seed=1))
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 3 │
| │ 1 │
| └─────┘
|
| sign(self) -> 'Expr'
| Compute the element-wise sign function on numeric types.
|
| The returned value is computed as follows:
|
| * -1 if x < 0.
| * 1 if x > 0.
| * x otherwise (typically 0, but could be NaN if the input is).
|
| Null values are preserved as-is, and the dtype of the input is preserved.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-9.0, -0.0, 0.0, 4.0, float("nan"), None]})
| >>> df.select(pl.col.a.sign())
| shape: (6, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ -1.0 │
| │ -0.0 │
| │ 0.0 │
| │ 1.0 │
| │ NaN │
| │ null │
| └──────┘
|
| sin(self) -> 'Expr'
| Compute the element-wise value for the sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 2]})
| >>> df.select(pl.col("a").sin())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
| >>> df = pl.DataFrame({"a": [0.0, 90]})
| >>> df.select(pl.col("a").radians().sin())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
|
| sinh(self) -> 'Expr'
| Compute the element-wise value for the hyperbolic sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").sinh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.175201 │
| └──────────┘
|
| skew(self, *, bias: 'bool' = True) -> 'Expr'
| Compute the sample skewness of a data set.
|
| For normally distributed data, the skewness should be about zero. For
| unimodal continuous distributions, a skewness value greater than zero means
| that there is more weight in the right tail of the distribution. The
| function `skewtest` can be used to determine if the skewness value
| is close enough to zero, statistically speaking.
|
|
| See scipy.stats for more information.
|
| Parameters
| ----------
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Notes
| -----
| The sample skewness is computed as the Fisher-Pearson coefficient
| of skewness, i.e.
|
| .. math:: g_1=\frac{m_3}{m_2^{3/2}}
|
| where
|
| .. math:: m_i=\frac{1}{N}\sum_{n=1}^N(x[n]-\bar{x})^i
|
| is the biased sample :math:`i\texttt{th}` central moment, and
| :math:`\bar{x}` is
| the sample mean. If `bias` is False, the calculations are
| corrected for bias and the value computed is the adjusted
| Fisher-Pearson standardized moment coefficient, i.e.
|
| .. math::
| G_1 = \frac{k_3}{k_2^{3/2}} = \frac{\sqrt{N(N-1)}}{N-2}\frac{m_3}{m_2^{3/2}}
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").skew())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.343622 │
| └──────────┘
|
| slice(self, offset: 'int | Expr', length: 'int | Expr | None' = None) -> 'Expr'
| Get a slice of this expression.
|
| Parameters
| ----------
| offset
| Start index. Negative indexing is supported.
| length
| Length of the slice. If set to `None`, all rows starting at the offset
| will be selected.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [8, 9, 10, 11],
| ... "b": [None, 4, 4, 4],
| ... }
| ... )
| >>> df.select(pl.all().slice(1, 2))
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 9 ┆ 4 │
| │ 10 ┆ 4 │
| └─────┴─────┘
|
| sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Expr'
| Sort this column.
|
| When used in a projection/selection context, the whole column is sorted.
| When used in a group by context, the groups are sorted.
|
| Parameters
| ----------
| descending
| Sort in descending order.
| nulls_last
| Place null values last.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, None, 3, 2],
| ... }
| ... )
| >>> df.select(pl.col("a").sort())
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| │ 1 │
| │ 2 │
| │ 3 │
| └──────┘
| >>> df.select(pl.col("a").sort(descending=True))
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| │ 3 │
| │ 2 │
| │ 1 │
| └──────┘
| >>> df.select(pl.col("a").sort(nulls_last=True))
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 1 │
| │ 2 │
| │ 3 │
| │ null │
| └──────┘
|
| When sorting in a group by context, the groups are sorted.
|
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["one", "one", "one", "two", "two", "two"],
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.group_by("group").agg(pl.col("value").sort()) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌───────┬────────────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪════════════╡
| │ two ┆ [3, 4, 99] │
| │ one ┆ [1, 2, 98] │
| └───────┴────────────┘
|
| sort_by(self, by: 'IntoExpr | Iterable[IntoExpr]', *more_by: 'IntoExpr', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False, multithreaded: 'bool' = True, maintain_order: 'bool' = False) -> 'Expr'
| Sort this column by the ordering of other columns.
|
| When used in a projection/selection context, the whole column is sorted.
| When used in a group by context, the groups are sorted.
|
| Parameters
| ----------
| by
| Column(s) to sort by. Accepts expression input. Strings are parsed as column
| names.
| *more_by
| Additional columns to sort by, specified as positional arguments.
| descending
| Sort in descending order. When sorting by multiple columns, can be specified
| per column by passing a sequence of booleans.
| nulls_last
| Place null values last; can specify a single boolean applying to all columns
| or a sequence of booleans for per-column control.
| multithreaded
| Sort using multiple threads.
| maintain_order
| Whether the order should be maintained if elements are equal.
|
| Examples
| --------
| Pass a single column name to sort by that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["a", "a", "b", "b"],
| ... "value1": [1, 3, 4, 2],
| ... "value2": [8, 7, 6, 5],
| ... }
| ... )
| >>> df.select(pl.col("group").sort_by("value1"))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ a │
| │ b │
| │ a │
| │ b │
| └───────┘
|
| Sorting by expressions is also supported.
|
| >>> df.select(pl.col("group").sort_by(pl.col("value1") + pl.col("value2")))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ b │
| │ a │
| │ a │
| │ b │
| └───────┘
|
| Sort by multiple columns by passing a list of columns.
|
| >>> df.select(pl.col("group").sort_by(["value1", "value2"], descending=True))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ b │
| │ a │
| │ b │
| │ a │
| └───────┘
|
| Or use positional arguments to sort by multiple columns in the same way.
|
| >>> df.select(pl.col("group").sort_by("value1", "value2"))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ a │
| │ b │
| │ a │
| │ b │
| └───────┘
|
| When sorting in a group by context, the groups are sorted.
|
| >>> df.group_by("group").agg(
| ... pl.col("value1").sort_by("value2")
| ... ) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ value1 │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪═══════════╡
| │ a ┆ [3, 1] │
| │ b ┆ [2, 4] │
| └───────┴───────────┘
|
| Take a single row from each group where a column attains its minimal value
| within that group.
|
| >>> df.group_by("group").agg(
| ... pl.all().sort_by("value2").first()
| ... ) # doctest: +IGNORE_RESULT
| shape: (2, 3)
| ┌───────┬────────┬────────┐
| │ group ┆ value1 ┆ value2 |
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 |
| ╞═══════╪════════╪════════╡
| │ a ┆ 3 ┆ 7 |
| │ b ┆ 2 ┆ 5 |
| └───────┴────────┴────────┘
|
| sqrt(self) -> 'Expr'
| Compute the square root of the elements.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").sqrt())
| shape: (3, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 1.414214 │
| │ 2.0 │
| └──────────┘
|
| std(self, ddof: 'int' = 1) -> 'Expr'
| Get standard deviation.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").std())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
|
| sub(self, other: 'Any') -> 'Expr'
| Method equivalent of subtraction operator `expr - other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [0, 1, 2, 3, 4]})
| >>> df.with_columns(
| ... pl.col("x").sub(2).alias("x-2"),
| ... pl.col("x").sub(pl.col("x").cum_sum()).alias("x-expr"),
| ... )
| shape: (5, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ x-2 ┆ x-expr │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪════════╡
| │ 0 ┆ -2 ┆ 0 │
| │ 1 ┆ -1 ┆ 0 │
| │ 2 ┆ 0 ┆ -1 │
| │ 3 ┆ 1 ┆ -3 │
| │ 4 ┆ 2 ┆ -6 │
| └─────┴─────┴────────┘
|
| sum(self) -> 'Expr'
| Get sum value.
|
| Notes
| -----
| * Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
| * If there are no non-null values, then the output is `0`.
| If you would prefer empty sums to return `None`, you can
| use `pl.when(expr.count()>0).then(expr.sum())` instead
| of `expr.sum()`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").sum())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 0 │
| └─────┘
|
| tail(self, n: 'int | Expr' = 10) -> 'Expr'
| Get the last `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7]})
| >>> df.select(pl.col("foo").tail(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 5 │
| │ 6 │
| │ 7 │
| └─────┘
|
| tan(self) -> 'Expr'
| Compute the element-wise value for the tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 4]})
| >>> df.select(pl.col("a").tan())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
| >>> df = pl.DataFrame({"a": [0.0, 45]})
| >>> df.select(pl.col("a").radians().tan())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
|
| tanh(self) -> 'Expr'
| Compute the element-wise value for the hyperbolic tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").tanh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.761594 │
| └──────────┘
|
| to_physical(self) -> 'Expr'
| Cast to physical representation of the logical dtype.
|
| - :func:`polars.datatypes.Date` -> :func:`polars.datatypes.Int32`
| - :func:`polars.datatypes.Datetime` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Time` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Duration` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Categorical` -> :func:`polars.datatypes.UInt32`
| - `List(inner)` -> `List(physical of inner)`
| - `Array(inner)` -> `Struct(physical of inner)`
| - `Struct(fields)` -> `Array(physical of fields)`
|
| Other data types will be left unchanged.
|
| Warnings
| --------
| The physical representations are an implementation detail
| and not guaranteed to be stable.
|
| Examples
| --------
| Replicating the pandas
| `pd.factorize
| <https://pandas.pydata.org/docs/reference/api/pandas.factorize.html>`_
| function.
|
| >>> pl.DataFrame({"vals": ["a", "x", None, "a"]}).with_columns(
| ... pl.col("vals").cast(pl.Categorical),
| ... pl.col("vals")
| ... .cast(pl.Categorical)
| ... .to_physical()
| ... .alias("vals_physical"),
| ... )
| shape: (4, 2)
| ┌──────┬───────────────┐
| │ vals ┆ vals_physical │
| │ --- ┆ --- │
| │ cat ┆ u32 │
| ╞══════╪═══════════════╡
| │ a ┆ 0 │
| │ x ┆ 1 │
| │ null ┆ null │
| │ a ┆ 0 │
| └──────┴───────────────┘
|
| top_k(self, k: 'int | IntoExprColumn' = 5) -> 'Expr'
| Return the `k` largest elements.
|
| Non-null elements are always preferred over null elements. The output
| is not guaranteed to be in any particular order, call :func:`sort`
| after this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k_by
| bottom_k
| bottom_k_by
|
| Examples
| --------
| Get the 5 largest values in series.
|
| >>> df = pl.DataFrame({"value": [1, 98, 2, 3, 99, 4]})
| >>> df.select(
| ... pl.col("value").top_k().alias("top_k"),
| ... pl.col("value").bottom_k().alias("bottom_k"),
| ... )
| shape: (5, 2)
| ┌───────┬──────────┐
| │ top_k ┆ bottom_k │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪══════════╡
| │ 4 ┆ 1 │
| │ 98 ┆ 98 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| │ 99 ┆ 4 │
| └───────┴──────────┘
|
| top_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int | IntoExprColumn' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Expr'
| Return the elements corresponding to the `k` largest elements of the `by` column(s).
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed to `reverse`.
|
| Parameters
| ----------
| by
| Column(s) used to determine the largest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` smallest elements of the `by` column(s) (instead of the `k`
| largest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| bottom_k
| bottom_k_by
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [6, 5, 4, 3, 2, 1],
| ... "c": ["Apple", "Orange", "Apple", "Apple", "Banana", "Banana"],
| ... }
| ... )
| >>> df
| shape: (6, 3)
| ┌─────┬─────┬────────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪════════╡
| │ 1 ┆ 6 ┆ Apple │
| │ 2 ┆ 5 ┆ Orange │
| │ 3 ┆ 4 ┆ Apple │
| │ 4 ┆ 3 ┆ Apple │
| │ 5 ┆ 2 ┆ Banana │
| │ 6 ┆ 1 ┆ Banana │
| └─────┴─────┴────────┘
|
| Get the top 2 rows by column `a` or `b`.
|
| >>> df.select(
| ... pl.all().top_k_by("a", 2).name.suffix("_top_by_a"),
| ... pl.all().top_k_by("b", 2).name.suffix("_top_by_b"),
| ... )
| shape: (2, 6)
| ┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
| │ a_top_by_a ┆ b_top_by_a ┆ c_top_by_a ┆ a_top_by_b ┆ b_top_by_b ┆ c_top_by_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞════════════╪════════════╪════════════╪════════════╪════════════╪════════════╡
| │ 6 ┆ 1 ┆ Banana ┆ 1 ┆ 6 ┆ Apple │
| │ 5 ┆ 2 ┆ Banana ┆ 2 ┆ 5 ┆ Orange │
| └────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
|
| Get the top 2 rows by multiple columns with given order.
|
| >>> df.select(
| ... pl.all()
| ... .top_k_by(["c", "a"], 2, reverse=[False, True])
| ... .name.suffix("_by_ca"),
| ... pl.all()
| ... .top_k_by(["c", "b"], 2, reverse=[False, True])
| ... .name.suffix("_by_cb"),
| ... )
| shape: (2, 6)
| ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
| │ a_by_ca ┆ b_by_ca ┆ c_by_ca ┆ a_by_cb ┆ b_by_cb ┆ c_by_cb │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
| │ 2 ┆ 5 ┆ Orange ┆ 2 ┆ 5 ┆ Orange │
| │ 5 ┆ 2 ┆ Banana ┆ 6 ┆ 1 ┆ Banana │
| └─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
|
| Get the top 2 rows by column `a` in each group.
|
| >>> (
| ... df.group_by("c", maintain_order=True)
| ... .agg(pl.all().top_k_by("a", 2))
| ... .explode(pl.all().exclude("c"))
| ... )
| shape: (5, 3)
| ┌────────┬─────┬─────┐
| │ c ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ Apple ┆ 4 ┆ 3 │
| │ Apple ┆ 3 ┆ 4 │
| │ Orange ┆ 2 ┆ 5 │
| │ Banana ┆ 6 ┆ 1 │
| │ Banana ┆ 5 ┆ 2 │
| └────────┴─────┴─────┘
|
| truediv(self, other: 'Any') -> 'Expr'
| Method equivalent of float division operator `expr / other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Notes
| -----
| Zero-division behaviour follows IEEE-754:
|
| 0/0: Invalid operation - mathematically undefined, returns NaN.
| n/0: On finite operands gives an exact infinite result, eg: ±infinity.
|
| See Also
| --------
| floordiv
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={"x": [-2, -1, 0, 1, 2], "y": [0.5, 0.0, 0.0, -4.0, -0.5]}
| ... )
| >>> df.with_columns(
| ... pl.col("x").truediv(2).alias("x/2"),
| ... pl.col("x").truediv(pl.col("y")).alias("x/y"),
| ... )
| shape: (5, 4)
| ┌─────┬──────┬──────┬───────┐
| │ x ┆ y ┆ x/2 ┆ x/y │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 ┆ f64 │
| ╞═════╪══════╪══════╪═══════╡
| │ -2 ┆ 0.5 ┆ -1.0 ┆ -4.0 │
| │ -1 ┆ 0.0 ┆ -0.5 ┆ -inf │
| │ 0 ┆ 0.0 ┆ 0.0 ┆ NaN │
| │ 1 ┆ -4.0 ┆ 0.5 ┆ -0.25 │
| │ 2 ┆ -0.5 ┆ 1.0 ┆ -4.0 │
| └─────┴──────┴──────┴───────┘
|
| truncate(self, decimals: 'int' = 0) -> 'Expr'
| Truncate numeric data toward zero to `decimals` number of decimal places.
|
| Parameters
| ----------
| decimals
| Number of decimal places to truncate to.
|
| Notes
| -----
| * Truncation discards the fractional part beyond the given number of decimals.
| For example, when rounding to 0 decimals 0.25, -0.25, 0.99, and -0.99 will
| all round to 0. When rounding to 1 decimal 1.9999 rounds to 1.9 and -1.9999
| rounds to -1.9. There is no tiebreak behaviour at midpoint values as there
| is with :meth:`round` so 0.5 and -0.5 will also round to 0 when decimals=1.
|
| * This method performs numeric truncation. For truncating temporal
| data (dates/datetimes), use :func:`Expr.dt.truncate` instead.
|
| See Also
| --------
| ceil : Round up to the nearest integer.
| floor : Round down to the nearest integer.
| round : Round to a given number of decimals.
| round_sig_figs : Round to a given number of significant figures.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-9.9999, 0.12345, 1.0251, 8.8765]})
| >>> df.with_columns(
| ... t0=pl.col("n").truncate(0),
| ... t1=pl.col("n").truncate(1),
| ... t2=pl.col("n").truncate(2),
| ... t3=pl.col("n").truncate(3),
| ... t4=pl.col("n").truncate(4),
| ... )
| shape: (4, 6)
| ┌─────────┬──────┬──────┬───────┬────────┬─────────┐
| │ n ┆ t0 ┆ t1 ┆ t2 ┆ t3 ┆ t4 │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
| ╞═════════╪══════╪══════╪═══════╪════════╪═════════╡
| │ -9.9999 ┆ -9.0 ┆ -9.9 ┆ -9.99 ┆ -9.999 ┆ -9.9999 │
| │ 0.12345 ┆ 0.0 ┆ 0.1 ┆ 0.12 ┆ 0.123 ┆ 0.1234 │
| │ 1.0251 ┆ 1.0 ┆ 1.0 ┆ 1.02 ┆ 1.025 ┆ 1.025 │
| │ 8.8765 ┆ 8.0 ┆ 8.8 ┆ 8.87 ┆ 8.876 ┆ 8.8765 │
| └─────────┴──────┴──────┴───────┴────────┴─────────┘
|
| unique(self, *, maintain_order: 'bool' = False) -> 'Expr'
| Get unique values of this expression.
|
| `null` is considered to be a unique value for the purposes of this operation.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
| >>> df.select(pl.col("a").unique()) # doctest: +IGNORE_RESULT
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 1 │
| └─────┘
| >>> df.select(pl.col("a").unique(maintain_order=True))
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| └─────┘
|
| unique_counts(self) -> 'Expr'
| Return a count of the unique values in the order of appearance.
|
| This method differs from `value_counts` in that it does not return the
| values, only the counts and might be faster
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "id": ["a", "b", "b", "c", "c", "c"],
| ... }
| ... )
| >>> df.select(pl.col("id").unique_counts())
| shape: (3, 1)
| ┌─────┐
| │ id │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Note that `group_by` can be used to generate counts.
|
| >>> df.group_by("id", maintain_order=True).len().select("len")
| shape: (3, 1)
| ┌─────┐
| │ len │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| To add counts as a new column `pl.len()` can be used as a window function.
|
| >>> df.with_columns(pl.len().over("id"))
| shape: (6, 2)
| ┌─────┬─────┐
| │ id ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ b ┆ 2 │
| │ b ┆ 2 │
| │ c ┆ 3 │
| │ c ┆ 3 │
| │ c ┆ 3 │
| └─────┴─────┘
|
| upper_bound(self) -> 'Expr'
| Calculate the upper bound.
|
| Returns a unit Series with the highest value possible for the dtype of this
| expression.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").upper_bound())
| shape: (1, 1)
| ┌─────────────────────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════════════════════╡
| │ 9223372036854775807 │
| └─────────────────────┘
|
| value_counts(self, *, sort: 'bool' = False, parallel: 'bool' = False, name: 'str_ | None' = None, normalize: 'bool' = False) -> 'Expr'
| Count the occurrence of unique values.
|
| Parameters
| ----------
| sort
| Sort the output by count, in descending order.
| If set to `False` (default), the order is non-deterministic.
| parallel
| Execute the computation in parallel.
|
| .. note::
| This option should likely *not* be enabled in a `group_by` context,
| as the computation will already be parallelized per group.
| name
| Give the resulting count column a specific name; if `normalize` is
| True this defaults to "proportion", otherwise defaults to "count".
| normalize
| If True, the count is returned as the relative frequency of unique
| values normalized to 1.0.
|
| Returns
| -------
| Expr
| Expression of type :class:`Struct`, mapping unique values to their
| count (or proportion).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"color": ["red", "blue", "red", "green", "blue", "blue"]}
| ... )
| >>> df_count = df.select(pl.col("color").value_counts())
| >>> df_count # doctest: +IGNORE_RESULT
| shape: (3, 1)
| ┌─────────────┐
| │ color │
| │ --- │
| │ struct[2] │
| ╞═════════════╡
| │ {"green",1} │
| │ {"blue",3} │
| │ {"red",2} │
| └─────────────┘
|
| >>> df_count.unnest("color") # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌───────┬───────┐
| │ color ┆ count │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═══════╡
| │ green ┆ 1 │
| │ blue ┆ 3 │
| │ red ┆ 2 │
| └───────┴───────┘
|
| Sort the output by (descending) count, customize the field name,
| and normalize the count to its relative proportion (of 1.0).
|
| >>> df_count = df.select(
| ... pl.col("color").value_counts(
| ... name="fraction",
| ... normalize=True,
| ... sort=True,
| ... )
| ... )
| >>> df_count
| shape: (3, 1)
| ┌────────────────────┐
| │ color │
| │ --- │
| │ struct[2] │
| ╞════════════════════╡
| │ {"blue",0.5} │
| │ {"red",0.333333} │
| │ {"green",0.166667} │
| └────────────────────┘
|
| >>> df_count.unnest("color")
| shape: (3, 2)
| ┌───────┬──────────┐
| │ color ┆ fraction │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪══════════╡
| │ blue ┆ 0.5 │
| │ red ┆ 0.333333 │
| │ green ┆ 0.166667 │
| └───────┴──────────┘
|
| Note that `group_by` can be used to generate counts.
|
| >>> df.group_by("color").len() # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌───────┬─────┐
| │ color ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═════╡
| │ red ┆ 2 │
| │ green ┆ 1 │
| │ blue ┆ 3 │
| └───────┴─────┘
|
| To add counts as a new column `pl.len()` can be used as a window function.
|
| >>> df.with_columns(pl.len().over("color"))
| shape: (6, 2)
| ┌───────┬─────┐
| │ color ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═════╡
| │ red ┆ 2 │
| │ blue ┆ 3 │
| │ red ┆ 2 │
| │ green ┆ 1 │
| │ blue ┆ 3 │
| │ blue ┆ 3 │
| └───────┴─────┘
|
| >>> df.with_columns((pl.len().over("color") / pl.len()).alias("fraction"))
| shape: (6, 2)
| ┌───────┬──────────┐
| │ color ┆ fraction │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪══════════╡
| │ red ┆ 0.333333 │
| │ blue ┆ 0.5 │
| │ red ┆ 0.333333 │
| │ green ┆ 0.166667 │
| │ blue ┆ 0.5 │
| │ blue ┆ 0.5 │
| └───────┴──────────┘
|
| var(self, ddof: 'int' = 1) -> 'Expr'
| Get variance.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").var())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
|
| where(self, predicate: 'Expr') -> 'Expr'
| Filter a single column.
|
| .. deprecated:: 0.20.4
| Use the :func:`filter` method instead.
|
| Alias for :func:`filter`.
|
| Parameters
| ----------
| predicate
| Boolean expression.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group_col": ["g1", "g1", "g2"],
| ... "b": [1, 2, 3],
| ... }
| ... )
| >>> df.group_by("group_col").agg( # doctest: +SKIP
| ... [
| ... pl.col("b").where(pl.col("b") < 2).sum().alias("lt"),
| ... pl.col("b").where(pl.col("b") >= 2).sum().alias("gte"),
| ... ]
| ... ).sort("group_col")
| shape: (2, 3)
| ┌───────────┬─────┬─────┐
| │ group_col ┆ lt ┆ gte │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════════╪═════╪═════╡
| │ g1 ┆ 1 ┆ 2 │
| │ g2 ┆ 0 ┆ 3 │
| └───────────┴─────┴─────┘
|
| xor(self, other: 'Any') -> 'Expr'
| Method equivalent of bitwise exclusive-or operator `expr ^ other`.
|
| Parameters
| ----------
| other
| Integer or boolean value; accepts expression input.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"x": [True, False, True, False], "y": [True, True, False, False]}
| ... )
| >>> df.with_columns(pl.col("x").xor(pl.col("y")).alias("x ^ y"))
| shape: (4, 3)
| ┌───────┬───────┬───────┐
| │ x ┆ y ┆ x ^ y │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞═══════╪═══════╪═══════╡
| │ true ┆ true ┆ false │
| │ false ┆ true ┆ true │
| │ true ┆ false ┆ true │
| │ false ┆ false ┆ false │
| └───────┴───────┴───────┘
|
| >>> def binary_string(n: int) -> str:
| ... return bin(n)[2:].zfill(8)
| >>>
| >>> df = pl.DataFrame(
| ... data={"x": [10, 8, 250, 66], "y": [1, 2, 3, 4]},
| ... schema={"x": pl.UInt8, "y": pl.UInt8},
| ... )
| >>> df.with_columns(
| ... pl.col("x")
| ... .map_elements(binary_string, return_dtype=pl.String)
| ... .alias("bin_x"),
| ... pl.col("y")
| ... .map_elements(binary_string, return_dtype=pl.String)
| ... .alias("bin_y"),
| ... pl.col("x").xor(pl.col("y")).alias("xor_xy"),
| ... pl.col("x")
| ... .xor(pl.col("y"))
| ... .map_elements(binary_string, return_dtype=pl.String)
| ... .alias("bin_xor_xy"),
| ... )
| shape: (4, 6)
| ┌─────┬─────┬──────────┬──────────┬────────┬────────────┐
| │ x ┆ y ┆ bin_x ┆ bin_y ┆ xor_xy ┆ bin_xor_xy │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ str ┆ str ┆ u8 ┆ str │
| ╞═════╪═════╪══════════╪══════════╪════════╪════════════╡
| │ 10 ┆ 1 ┆ 00001010 ┆ 00000001 ┆ 11 ┆ 00001011 │
| │ 8 ┆ 2 ┆ 00001000 ┆ 00000010 ┆ 10 ┆ 00001010 │
| │ 250 ┆ 3 ┆ 11111010 ┆ 00000011 ┆ 249 ┆ 11111001 │
| │ 66 ┆ 4 ┆ 01000010 ┆ 00000100 ┆ 70 ┆ 01000110 │
| └─────┴─────┴──────────┴──────────┴────────┴────────────┘
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| deserialize(source: 'str_ | Path | IOBase | bytes', *, format: 'SerializationFormat' = 'binary') -> 'Expr'
| Read a serialized expression from a file.
|
| Parameters
| ----------
| source
| Path to a file or a file-like object (by file-like object, we refer to
| objects that have a `read()` method, such as a file handler (e.g.
| via builtin `open` function) or `BytesIO`).
| format
| The format with which the Expr was serialized. Options:
|
| - `"binary"`: Deserialize from binary format (bytes). This is the default.
| - `"json"`: Deserialize from JSON format (string).
|
| Warnings
| --------
| This function uses :mod:`pickle` if the logical plan contains Python UDFs,
| and as such inherits the security implications. Deserializing can execute
| arbitrary code, so it should only be attempted on trusted data.
|
| See Also
| --------
| Expr.meta.serialize
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| >>> import io
| >>> expr = pl.col("foo").sum().over("bar")
| >>> bytes = expr.meta.serialize()
| >>> pl.Expr.deserialize(io.BytesIO(bytes))
| <Expr ['col("foo").sum().over([col("ba…'] at ...>
|
| from_json(value: 'str_') -> 'Expr'
| Read an expression from a JSON encoded string to construct an Expression.
|
| .. deprecated:: 0.20.11
| This method has been renamed to :meth:`deserialize`.
| Note that the new method operates on file-like inputs rather than strings.
| Enclose your input in `io.StringIO` to keep the same behavior.
|
| Parameters
| ----------
| value
| JSON encoded string value
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| arr
| Create an object namespace of all array related methods.
|
| See the individual method pages for full details.
|
| bin
| Create an object namespace of all binary related methods.
|
| See the individual method pages for full details
|
| cat
| Create an object namespace of all categorical related methods.
|
| See the individual method pages for full details
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": ["a", "b"]}).select(
| ... pl.col("values").cast(pl.Categorical)
| ... )
| >>> df.select(pl.col("values").cat.get_categories())
| shape: (2, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ str │
| ╞════════╡
| │ a │
| │ b │
| └────────┘
|
| dt
| Create an object namespace of all datetime related methods.
|
| ext
| Create an object namespace of all extension type related expressions.
|
| See the individual method pages for full details.
|
| list
| Create an object namespace of all list related methods.
|
| See the individual method pages for full details.
|
| meta
| Create an object namespace of all meta related expression methods.
|
| This can be used to modify and traverse existing expressions.
|
| name
| Create an object namespace of all expressions that modify expression names.
|
| See the individual method pages for full details.
|
| str
| Create an object namespace of all string related methods.
|
| See the individual method pages for full details.
|
| Examples
| --------
| >>> df = pl.DataFrame({"letters": ["a", "b"]})
| >>> df.select(pl.col("letters").str.to_uppercase())
| shape: (2, 1)
| ┌─────────┐
| │ letters │
| │ --- │
| │ str │
| ╞═════════╡
| │ A │
| │ B │
| └─────────┘
|
| struct
| Create an object namespace of all struct related methods.
|
| See the individual method pages for full details.
|
| Examples
| --------
| >>> df = (
| ... pl.DataFrame(
| ... {
| ... "int": [1, 2],
| ... "str": ["a", "b"],
| ... "bool": [True, None],
| ... "list": [[1, 2], [3]],
| ... }
| ... )
| ... .to_struct("my_struct")
| ... .to_frame()
| ... )
| >>> df.select(pl.col("my_struct").struct.field("str"))
| shape: (2, 1)
| ┌─────┐
| │ str │
| │ --- │
| │ str │
| ╞═════╡
| │ a │
| │ b │
| └─────┘
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set[str_]]', '_pyexpr': 'Py...
class When(builtins.object)
| When(when: 'Any') -> 'None'
|
| Utility class for the `when-then-otherwise` expression.
|
| Represents the initial state of the expression after `pl.when(...)` is called.
|
| In this state, `then` must be called to continue to finish the expression.
|
| Methods defined here:
|
| __init__(self, when: 'Any') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| then(self, statement: 'IntoExpr') -> 'Then'
| Attach a statement to the corresponding condition.
|
| Parameters
| ----------
| statement
| The statement to apply if the corresponding condition is true.
| Accepts expression input. Strings are parsed as column names, other
| non-expression inputs are parsed as literals.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
DATA
__all__ = ['Expr', 'When']
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\expr\__init__.py
polars.field
Python Library Documentation: function field in module polars.functions.lazy
field(name: 'str | list[str]') -> 'Expr'
Select a field in the current `struct.with_fields` scope.
Parameters
----------
name
Name of the field(s) to select.
Examples
--------
>>> df = pl.DataFrame({"a": [{"x": 5, "y": 2}, {"x": 3, "y": 4}]})
>>> df.select(pl.col("a").struct.with_fields(pl.field("x") ** 2))
shape: (2, 1)
┌───────────┐
│ a │
│ --- │
│ struct[2] │
╞═══════════╡
│ {25,2} │
│ {9,4} │
└───────────┘
polars.first
Python Library Documentation: function first in module polars.functions.lazy
first(*columns: 'str') -> 'Expr'
Get the first column or value.
This function has different behavior depending on the presence of `columns`
values. If none given (the default), returns an expression that takes the first
column of the context; otherwise, takes the first value of the given column(s).
Parameters
----------
*columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "baz"],
... }
... )
Return the first column:
>>> df.select(pl.first())
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 8 │
│ 3 │
└─────┘
Return the first value for the given column(s):
>>> df.select(pl.first("b"))
shape: (1, 1)
┌─────┐
│ b │
│ --- │
│ i64 │
╞═════╡
│ 4 │
└─────┘
>>> df.select(pl.first("a", "c"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ c │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1 ┆ foo │
└─────┴─────┘
polars.fold
Python Library Documentation: function fold in module polars.functions.lazy
fold(acc: 'IntoExpr', function: 'Callable[[Series, Series], Series]', exprs: 'Sequence[Expr | str] | Expr', *, returns_scalar: 'bool' = False, return_dtype: 'pl.DataTypeExpr | PolarsDataType | None' = None) -> 'Expr'
Accumulate over multiple columns horizontally/ row wise with a left fold.
Parameters
----------
acc
Accumulator Expression. This is the value that will be initialized when the fold
starts. For a sum this could for instance be lit(0).
function
Function to apply over the accumulator and the value.
Fn(acc, value) -> new_value
exprs
Expressions to aggregate over. May also be a wildcard expression.
returns_scalar
Whether or not `function` applied returns a scalar. This must be set correctly
by the user.
return_dtype
Output datatype.
If not set, the dtype will be inferred based on the dtype
of the accumulator.
Notes
-----
If you simply want the first encountered expression as accumulator,
consider using `reduce`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [3, 4, 5],
... "c": [5, 6, 7],
... }
... )
>>> df
shape: (3, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 5 │
│ 2 ┆ 4 ┆ 6 │
│ 3 ┆ 5 ┆ 7 │
└─────┴─────┴─────┘
Horizontally sum over all columns and add 1.
>>> df.select(
... pl.fold(
... acc=pl.lit(1), function=lambda acc, x: acc + x, exprs=pl.col("*")
... ).alias("sum"),
... )
shape: (3, 1)
┌─────┐
│ sum │
│ --- │
│ i32 │
╞═════╡
│ 10 │
│ 13 │
│ 16 │
└─────┘
You can also apply a condition/predicate on all columns:
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [0, 1, 2],
... }
... )
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 0 │
│ 2 ┆ 1 │
│ 3 ┆ 2 │
└─────┴─────┘
>>> df.filter(
... pl.fold(
... acc=pl.lit(True),
... function=lambda acc, x: acc & x,
... exprs=pl.col("*") > 1,
... )
... )
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 3 ┆ 2 │
└─────┴─────┘
polars.format
Python Library Documentation: function format in module polars.functions.as_datatype
format(f_string: 'str', *args: 'IntoExpr') -> 'Expr'
Format expressions as a string.
Parameters
----------
f_string
A string with placeholders of the form `{}`, `{index}` or `{name}`.
A placeholder can be empty, in which case it consumes the next argument
to pl.format. It can also be an index in which case it addresses the
nth argument to pl.format. Finally a placeholder can also be an ASCII
identifier, in which case it directly refers to a column name.
If you wish to use the characters `{}` literally you must escape them
by doubling, e.g. `'{{"json": 42}}'`.
args
Expression(s) that fill the placeholders
Notes
-----
If any input expression evaluates to null for a row, the output of
``pl.format`` is null for that row.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": ["a", "b", "c"],
... "b": [1, 2, 3],
... }
... )
>>> df.select(
... [
... pl.format("foo_{}_bar_{}", pl.col("a"), "b").alias("fmt"),
... ]
... )
shape: (3, 1)
┌─────────────┐
│ fmt │
│ --- │
│ str │
╞═════════════╡
│ foo_a_bar_1 │
│ foo_b_bar_2 │
│ foo_c_bar_3 │
└─────────────┘
>>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, None, 6]})
>>> df.select(pl.format("{}_{}", "a", "b").alias("fmt"))
shape: (3, 1)
┌──────┐
│ fmt │
│ --- │
│ str │
╞══════╡
│ 1_4 │
│ null │
│ 3_6 │
└──────┘
polars.from_arrow
Python Library Documentation: function from_arrow in module polars.convert.general
from_arrow(data: 'pa.Table | pa.Array | pa.ChunkedArray | pa.RecordBatch | Iterable[pa.RecordBatch | pa.Table] | ArrowArrayExportable | ArrowStreamExportable', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, rechunk: 'bool' = True) -> 'DataFrame | Series'
Create a DataFrame or Series from an Arrow Table or Array.
This operation will be zero copy for the most part. Types that are not
supported by Polars may be cast to the closest supported type.
Hint: You can also directly pass arrow tables to `pl.DataFrame()` / arrow
arrays to `pl.Series()` if the output type is known to avoid typing issues.
Parameters
----------
data : :class:`pyarrow.Table`, :class:`pyarrow.Array`, one or more :class:`pyarrow.RecordBatch`
Data representing an Arrow Table, Array, sequence of RecordBatches or Tables, or other
object that supports the Arrow PyCapsule interface.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
Schema entries are applied positionally, following the order of the
Python dictionary. The provided schema takes precedence over the schema
of the underlying Arrow data. As such, if the provided schema names do
not match the underlying Arrow data, the column names of the Arrow data
will be discarded in favor of the names set in this argument.
The number of names given in the schema must match the underlying data
dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the schema param will be overridden.
rechunk : bool, default True
Make sure that all data is in contiguous memory.
Returns
-------
DataFrame or Series
Examples
--------
Constructing a DataFrame from an Arrow Table:
>>> import pyarrow as pa
>>> data = pa.table({"a": [1, 2, 3], "b": [4, 5, 6]})
>>> pl.from_arrow(data)
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 4 │
│ 2 ┆ 5 │
│ 3 ┆ 6 │
└─────┴─────┘
Constructing a Series from an Arrow Array:
>>> import pyarrow as pa
>>> data = pa.array([1, 2, 3])
>>> pl.from_arrow(data, schema={"s": pl.Int32})
shape: (3,)
Series: 's' [i32]
[
1
2
3
]
polars.from_dataframe
Python Library Documentation: function from_dataframe in module polars.convert.general
from_dataframe(df: 'SupportsInterchange | ArrowArrayExportable | ArrowStreamExportable', *, allow_copy: 'bool | None' = None, rechunk: 'bool' = True) -> 'DataFrame'
Build a Polars DataFrame from any dataframe supporting the PyCapsule Interface.
.. versionchanged:: 1.23.0
`from_dataframe` uses the PyCapsule Interface instead of the Dataframe
Interchange Protocol for conversion, only using the latter as a fallback.
Parameters
----------
df
Object supporting the dataframe PyCapsule Interface.
allow_copy
Allow memory to be copied to perform the conversion. If set to False, may cause
conversions that are not zero-copy to fail.
.. deprecated: 1.23.0
`allow_copy` is deprecated and will be removed in a future version.
rechunk : bool, default True
Make sure that all data is in contiguous memory.
Notes
-----
- Details on the PyCapsule Interface:
https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html.
- Details on the Python dataframe interchange protocol:
https://data-apis.org/dataframe-protocol/latest/index.html.
Using a dedicated function like :func:`from_pandas` or :func:`from_arrow` is
a more efficient method of conversion.
Examples
--------
Convert a pandas dataframe to Polars.
>>> import pandas as pd
>>> df_pd = pd.DataFrame({"a": [1, 2], "b": [3.0, 4.0], "c": ["x", "y"]})
>>> pl.from_dataframe(df_pd)
shape: (2, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ f64 ┆ str │
╞═════╪═════╪═════╡
│ 1 ┆ 3.0 ┆ x │
│ 2 ┆ 4.0 ┆ y │
└─────┴─────┴─────┘
polars.from_dict
Python Library Documentation: function from_dict in module polars.convert.general
from_dict(data: 'Mapping[str, Sequence[object] | Mapping[str, Sequence[object]] | Series]', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True) -> 'DataFrame'
Construct a DataFrame from a dictionary of sequences.
This operation clones data, unless you pass a `{str: pl.Series,}` dict.
Parameters
----------
data : dict of sequences
Two-dimensional data represented as a dictionary. dict must contain
Sequences.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the columns param will be overridden.
strict : bool, default True
Throw an error if any `data` value does not exactly match the given or inferred
data type for that column. If set to `False`, values that do not match the data
type are cast to that data type or, if casting is not possible, set to null
instead.
Returns
-------
DataFrame
Examples
--------
>>> df = pl.from_dict({"a": [1, 2], "b": [3, 4]})
>>> df
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 3 │
│ 2 ┆ 4 │
└─────┴─────┘
polars.from_dicts
Python Library Documentation: function from_dicts in module polars.convert.general
from_dicts(data: 'Iterable[Mapping[str, Any]]', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, infer_schema_length: 'int | None' = 100) -> 'DataFrame'
Construct a DataFrame from a sequence of dictionaries. This operation clones data.
Parameters
----------
data
Sequence with dictionaries mapping column name to value
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If a list of column names is supplied that does NOT match the names in the
underlying data, the names given here will overwrite the actual fields in
the order that they appear - however, in this case it is typically clearer
to rename after loading the frame.
If you want to drop some of the fields found in the input dictionaries, a
*partial* schema can be declared, in which case omitted fields will not be
loaded. Similarly, you can extend the loaded frame with empty columns by
adding them to the schema.
schema_overrides : dict, default None
Support override of inferred types for one or more columns.
strict : bool, default True
Throw an error if any `data` value does not exactly match the given or inferred
data type for that column. If set to `False`, values that do not match the data
type are cast to that data type or, if casting is not possible, set to null
instead.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
Returns
-------
DataFrame
Examples
--------
>>> data = [{"a": 1, "b": 4}, {"a": 2, "b": 5}, {"a": 3, "b": 6}]
>>> df = pl.from_dicts(data)
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 4 │
│ 2 ┆ 5 │
│ 3 ┆ 6 │
└─────┴─────┘
Declaring a partial `schema` will drop the omitted columns.
>>> df = pl.from_dicts(data, schema={"a": pl.Int32})
>>> df
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i32 │
╞═════╡
│ 1 │
│ 2 │
│ 3 │
└─────┘
Can also use the `schema` param to extend the loaded columns with one
or more additional (empty) columns that are not present in the input dicts:
>>> pl.from_dicts(
... data,
... schema=["a", "b", "c", "d"],
... schema_overrides={"c": pl.Float64, "d": pl.String},
... )
shape: (3, 4)
┌─────┬─────┬──────┬──────┐
│ a ┆ b ┆ c ┆ d │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ f64 ┆ str │
╞═════╪═════╪══════╪══════╡
│ 1 ┆ 4 ┆ null ┆ null │
│ 2 ┆ 5 ┆ null ┆ null │
│ 3 ┆ 6 ┆ null ┆ null │
└─────┴─────┴──────┴──────┘
polars.from_epoch
Python Library Documentation: function from_epoch in module polars.functions.lazy
from_epoch(column: 'str | Expr | Series | Sequence[int | float]', time_unit: 'EpochTimeUnit' = 's') -> 'Expr | Series'
Utility function that parses an epoch timestamp (or Unix time) to Polars Date(time).
Depending on the `time_unit` provided, this function will return a different dtype:
- time_unit="d" returns pl.Date
- time_unit="ns" returns pl.Datetime["ns"]
- otherwise returns pl.Datetime["us"] (Polars' default time unit)
Parameters
----------
column
Series or Expression to parse to Datetime.
time_unit
The unit of time of the timesteps since epoch time.
Examples
--------
Convert from integer seconds:
>>> df = pl.LazyFrame({"ts": [1666683077, 1666683099]})
>>> df.select(pl.from_epoch(pl.col("ts"), time_unit="s")).collect()
shape: (2, 1)
┌─────────────────────┐
│ ts │
│ --- │
│ datetime[μs] │
╞═════════════════════╡
│ 2022-10-25 07:31:17 │
│ 2022-10-25 07:31:39 │
└─────────────────────┘
Convert from fractional seconds:
>>> df = pl.LazyFrame({"ts": [-609066.723456, 1066445333.8888, 3405071999.987654]})
>>> df.select(pl.from_epoch(pl.col("ts"), time_unit="s")).collect()
shape: (3, 1)
┌────────────────────────────┐
│ ts │
│ --- │
│ datetime[μs] │
╞════════════════════════════╡
│ 1969-12-24 22:48:53.276544 │
│ 2003-10-18 02:48:53.888800 │
│ 2077-11-25 13:19:59.987654 │
└────────────────────────────┘
The function can also be used in an eager context by passing a Series
or sequence of int/float values.
>>> s = pl.Series([12345, 12346])
>>> pl.from_epoch(s, time_unit="d")
shape: (2,)
Series: '' [date]
[
2003-10-20
2003-10-21
]
polars.from_numpy
Python Library Documentation: function from_numpy in module polars.convert.general
from_numpy(data: 'np.ndarray[Any, Any]', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, orient: 'Orientation | None' = None) -> 'DataFrame'
Construct a DataFrame from a NumPy ndarray. This operation clones data.
Note that this is slower than creating from columnar memory.
Parameters
----------
data : :class:`numpy.ndarray`
Two-dimensional data represented as a NumPy ndarray.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the columns param will be overridden.
orient : {None, 'col', 'row'}
Whether to interpret two-dimensional data as columns or as rows. If None,
the orientation is inferred by matching the columns and data dimensions. If
this does not yield conclusive results, column orientation is used.
Returns
-------
DataFrame
Examples
--------
>>> import numpy as np
>>> data = np.array([[1, 2, 3], [4, 5, 6]])
>>> df = pl.from_numpy(data, schema=["a", "b"], orient="col")
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 4 │
│ 2 ┆ 5 │
│ 3 ┆ 6 │
└─────┴─────┘
polars.from_pandas
Python Library Documentation: function from_pandas in module polars.convert.general
from_pandas(data: 'pd.DataFrame | pd.Series[Any] | pd.Index[Any] | pd.DatetimeIndex', *, schema_overrides: 'SchemaDict | None' = None, rechunk: 'bool' = True, nan_to_null: 'bool' = True, include_index: 'bool' = False) -> 'DataFrame | Series'
Construct a Polars DataFrame or Series from a pandas DataFrame, Series, or Index.
This operation may clone data. If you want to ensure that in-place modifications
of the output don't affect the input, you may want to consider one of the following:
- Enable `Copy-On-Write <https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html>`_
in pandas.
- Call :meth:`DataFrame.clone` on the output of `from_pandas`.
This requires that :mod:`pandas` and :mod:`pyarrow` are installed.
Parameters
----------
data : :class:`pandas.DataFrame` or :class:`pandas.Series` or :class:`pandas.Index`
Data represented as a pandas DataFrame, Series, or Index.
schema_overrides : dict, default None
Support override of inferred types for one or more columns.
rechunk : bool, default True
Make sure that all data is in contiguous memory.
nan_to_null : bool, default True
If data contains `NaN` values PyArrow will convert the `NaN` to `None`
include_index : bool, default False
Load any non-default pandas indexes as columns.
.. note::
If the input is a pandas ``DataFrame`` and has a nameless index
which just enumerates the rows, then it will not be included in the
result, regardless of this parameter. If you want to be sure to include it,
please call ``.reset_index()`` prior to calling this function.
Returns
-------
DataFrame
Examples
--------
Constructing a :class:`DataFrame` from a :class:`pandas.DataFrame`:
>>> import pandas as pd
>>> pd_df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns=["a", "b", "c"])
>>> df = pl.from_pandas(pd_df)
>>> df
shape: (2, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 2 ┆ 3 │
│ 4 ┆ 5 ┆ 6 │
└─────┴─────┴─────┘
Constructing a Series from a :class:`pandas.Series`:
>>> import pandas as pd
>>> pd_series = pd.Series([1, 2, 3], name="pd")
>>> df = pl.from_pandas(pd_series)
>>> df
shape: (3,)
Series: 'pd' [i64]
[
1
2
3
]
polars.from_records
Python Library Documentation: function from_records in module polars.convert.general
from_records(data: 'Sequence[Any]', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100) -> 'DataFrame'
Construct a DataFrame from a sequence of sequences. This operation clones data.
Note that this is slower than creating from columnar memory.
Parameters
----------
data : Sequence of sequences
Two-dimensional data represented as a sequence of sequences.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the columns param will be overridden.
strict : bool, default True
Throw an error if any `data` value does not exactly match the given or inferred
data type for that column. If set to `False`, values that do not match the data
type are cast to that data type or, if casting is not possible, set to null
instead.
orient : {None, 'col', 'row'}
Whether to interpret two-dimensional data as columns or as rows. If None,
the orientation is inferred by matching the columns and data dimensions. If
this does not yield conclusive results, column orientation is used.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
Returns
-------
DataFrame
Examples
--------
>>> data = [[1, 2, 3], [4, 5, 6]]
>>> df = pl.from_records(data, schema=["a", "b"])
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 4 │
│ 2 ┆ 5 │
│ 3 ┆ 6 │
└─────┴─────┘
polars.from_repr
Python Library Documentation: function from_repr in module polars.convert.general
from_repr(data: 'str') -> 'DataFrame | Series'
Construct a Polars DataFrame or Series from its string representation.
.. versionchanged:: 0.20.17
The `tbl` parameter was renamed to `data`.
Parameters
----------
data
A string containing a polars DataFrame or Series repr; does not need
to be trimmed of whitespace (or leading prompts) as the repr will be
found/extracted automatically.
Notes
-----
This function handles the UTF8_FULL (default) and UTF8_FULL_CONDENSED
DataFrame table formats (with or without rounded corners), as well as
ASCII_FULL and ASCII_FULL_CONDENSED. Truncated columns/rows are omitted,
wrapped headers are accounted for, and dtypes are automatically identified.
Currently compound/nested dtypes such as List and Struct are not supported;
neither are Object dtypes. The DuckDB table/relation repr is also compatible
with this function.
See Also
--------
polars.DataFrame.to_init_repr
polars.Series.to_init_repr
Examples
--------
From DataFrame table repr:
>>> df = pl.from_repr(
... '''
... Out[3]:
... shape: (1, 5)
... ┌───────────┬────────────┬───┬───────┬────────────────────────────────┐
... │ source_ac ┆ source_cha ┆ … ┆ ident ┆ timestamp │
... │ tor_id ┆ nnel_id ┆ ┆ --- ┆ --- │
... │ --- ┆ --- ┆ ┆ str ┆ datetime[μs, Asia/Tokyo] │
... │ i32 ┆ i64 ┆ ┆ ┆ │
... ╞═══════════╪════════════╪═══╪═══════╪════════════════════════════════╡
... │ 123456780 ┆ 9876543210 ┆ … ┆ a:b:c ┆ 2023-03-25 10:56:59.663053 JST │
... │ … ┆ … ┆ … ┆ … ┆ … │
... │ 803065983 ┆ 2055938745 ┆ … ┆ x:y:z ┆ 2023-03-25 12:38:18.050545 JST │
... └───────────┴────────────┴───┴───────┴────────────────────────────────┘
... '''
... )
>>> df
shape: (2, 4)
┌─────────────────┬───────────────────┬───────┬────────────────────────────────┐
│ source_actor_id ┆ source_channel_id ┆ ident ┆ timestamp │
│ --- ┆ --- ┆ --- ┆ --- │
│ i32 ┆ i64 ┆ str ┆ datetime[μs, Asia/Tokyo] │
╞═════════════════╪═══════════════════╪═══════╪════════════════════════════════╡
│ 123456780 ┆ 9876543210 ┆ a:b:c ┆ 2023-03-25 10:56:59.663053 JST │
│ 803065983 ┆ 2055938745 ┆ x:y:z ┆ 2023-03-25 12:38:18.050545 JST │
└─────────────────┴───────────────────┴───────┴────────────────────────────────┘
From Series repr:
>>> s = pl.from_repr(
... '''
... shape: (3,)
... Series: 's' [bool]
... [
... true
... false
... true
... ]
... '''
... )
>>> s.to_list()
[True, False, True]
polars.from_torch
Python Library Documentation: function from_torch in module polars.convert.general
from_torch(tensor: 'torch.Tensor', schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, orient: 'Orientation | None' = None, force: 'bool' = False) -> 'DataFrame'
Construct a DataFrame from a PyTorch Tensor.
Parameters
----------
tensor : :class:`torch.Tensor`
A PyTorch `Tensor` object of one or more dimensions.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the columns param will be overridden.
orient : {None, 'col', 'row'}
Whether to interpret two-dimensional data as columns or as rows. If None,
the orientation is inferred by matching the columns and data dimensions. If
this does not yield conclusive results, column orientation is used.
force : bool
If False, the conversion is performed only if the Tensor is on CPU, does not
require grad, does not have its conjugate bit set, and is of a dtype (and
layout) that NumPy supports; this will typically be zero-copy. If True, it
is equivalent to calling `.detach().cpu().resolve_conj().resolve_neg()`
before passing the Tensor to Polars.
Returns
-------
DataFrame
Examples
--------
>>> import torch
>>> data = torch.tensor(
... [
... [1234.5, 200.0, 3000.5],
... [8000.0, 500.5, 6000.0],
... ]
... )
>>> df = pl.from_torch(
... data,
... schema=["colx", "coly", "colz"],
... schema_overrides={"colz": pl.Float64},
... )
>>> df
shape: (2, 3)
┌────────┬───────┬────────┐
│ colx ┆ coly ┆ colz │
│ --- ┆ --- ┆ --- │
│ f32 ┆ f32 ┆ f64 │
╞════════╪═══════╪════════╡
│ 1234.5 ┆ 200.0 ┆ 3000.5 │
│ 8000.0 ┆ 500.5 ┆ 6000.0 │
└────────┴───────┴────────┘
polars.functions
Python Library Documentation: package polars.functions in polars
NAME
polars.functions
PACKAGE CONTENTS
aggregation (package)
as_datatype
business
col
datatype
eager
escape_regex
lazy
len
lit
random
range (package)
repeat
whenthen
FUNCTIONS
_row_encode(exprs: 'pl.Selector | pl.Expr | Sequence[str | pl.Expr]', *, unordered: 'bool' = False, descending: 'list[bool] | None' = None, nulls_last: 'list[bool] | None' = None) -> 'Expr'
align_frames(*frames: 'FrameType | Iterable[FrameType]', on: 'str | Expr | Sequence[str] | Sequence[Expr] | Sequence[str | Expr]', how: 'JoinStrategy' = 'full', select: 'str | Expr | Sequence[str | Expr] | None' = None, descending: 'bool | Sequence[bool]' = False) -> 'list[FrameType]'
Align a sequence of frames using common values from one or more columns as a key.
Frames that do not contain the given key values have rows injected (with nulls
filling the non-key columns), and each resulting frame is sorted by the key.
The original column order of input frames is not changed unless `select` is
specified (in which case the final column order is determined from that). In the
case where duplicate key values exist, the alignment behaviour is determined by
the given alignment strategy specified in the `how` parameter (by default this
is a full outer join, but if your data is suitable you can get a large speedup
by setting `how="left"` instead).
Note that this function does not result in a joined frame - you receive the same
number of frames back that you passed in, but each is now aligned by key and has
the same number of rows.
Parameters
----------
frames
Sequence of DataFrames or LazyFrames.
on
One or more columns whose unique values will be used to align the frames.
select
Optional post-alignment column select to constrain and/or order
the columns returned from the newly aligned frames.
descending
Sort the alignment column values in descending order; can be a single
boolean or a list of booleans associated with each column in `on`.
how
By default the row alignment values are determined using a full outer join
strategy across all frames; if you know that the first frame contains all
required keys, you can set `how="left"` for a large performance increase.
Examples
--------
>>> from datetime import date
>>> df1 = pl.DataFrame(
... {
... "dt": [date(2022, 9, 1), date(2022, 9, 2), date(2022, 9, 3)],
... "x": [3.5, 4.0, 1.0],
... "y": [10.0, 2.5, 1.5],
... }
... )
>>> df2 = pl.DataFrame(
... {
... "dt": [date(2022, 9, 2), date(2022, 9, 3), date(2022, 9, 1)],
... "x": [8.0, 1.0, 3.5],
... "y": [1.5, 12.0, 5.0],
... }
... )
>>> df3 = pl.DataFrame(
... {
... "dt": [date(2022, 9, 3), date(2022, 9, 2)],
... "x": [2.0, 5.0],
... "y": [2.5, 2.0],
... }
... ) # doctest: +IGNORE_RESULT
>>> pl.Config.set_tbl_formatting("UTF8_FULL") # doctest: +IGNORE_RESULT
#
# df1 df2 df3
# shape: (3, 3) shape: (3, 3) shape: (2, 3)
# ┌────────────┬─────┬──────┐ ┌────────────┬─────┬──────┐ ┌────────────┬─────┬─────┐
# │ dt ┆ x ┆ y │ │ dt ┆ x ┆ y │ │ dt ┆ x ┆ y │
# │ --- ┆ --- ┆ --- │ │ --- ┆ --- ┆ --- │ │ --- ┆ --- ┆ --- │
# │ date ┆ f64 ┆ f64 │ │ date ┆ f64 ┆ f64 │ │ date ┆ f64 ┆ f64 │
# ╞════════════╪═════╪══════╡ ╞════════════╪═════╪══════╡ ╞════════════╪═════╪═════╡
# │ 2022-09-01 ┆ 3.5 ┆ 10.0 │\ ,->│ 2022-09-02 ┆ 8.0 ┆ 1.5 │\ ,->│ 2022-09-03 ┆ 2.0 ┆ 2.5 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ \/ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ \/ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┤
# │ 2022-09-02 ┆ 4.0 ┆ 2.5 │_/\,->│ 2022-09-03 ┆ 1.0 ┆ 12.0 │_/`-->│ 2022-09-02 ┆ 5.0 ┆ 2.0 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ /\ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ └────────────┴─────┴─────┘
# │ 2022-09-03 ┆ 1.0 ┆ 1.5 │_/ `>│ 2022-09-01 ┆ 3.5 ┆ 5.0 │-//-
# └────────────┴─────┴──────┘ └────────────┴─────┴──────┘
...
Align frames by the "dt" column:
>>> af1, af2, af3 = pl.align_frames(
... df1, df2, df3, on="dt"
... ) # doctest: +IGNORE_RESULT
#
# df1 df2 df3
# shape: (3, 3) shape: (3, 3) shape: (3, 3)
# ┌────────────┬─────┬──────┐ ┌────────────┬─────┬──────┐ ┌────────────┬──────┬──────┐
# │ dt ┆ x ┆ y │ │ dt ┆ x ┆ y │ │ dt ┆ x ┆ y │
# │ --- ┆ --- ┆ --- │ │ --- ┆ --- ┆ --- │ │ --- ┆ --- ┆ --- │
# │ date ┆ f64 ┆ f64 │ │ date ┆ f64 ┆ f64 │ │ date ┆ f64 ┆ f64 │
# ╞════════════╪═════╪══════╡ ╞════════════╪═════╪══════╡ ╞════════════╪══════╪══════╡
# │ 2022-09-01 ┆ 3.5 ┆ 10.0 │----->│ 2022-09-01 ┆ 3.5 ┆ 5.0 │----->│ 2022-09-01 ┆ null ┆ null │
# ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┤
# │ 2022-09-02 ┆ 4.0 ┆ 2.5 │----->│ 2022-09-02 ┆ 8.0 ┆ 1.5 │----->│ 2022-09-02 ┆ 5.0 ┆ 2.0 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┤
# │ 2022-09-03 ┆ 1.0 ┆ 1.5 │----->│ 2022-09-03 ┆ 1.0 ┆ 12.0 │----->│ 2022-09-03 ┆ 2.0 ┆ 2.5 │
# └────────────┴─────┴──────┘ └────────────┴─────┴──────┘ └────────────┴──────┴──────┘
...
Align frames by "dt" using "left" alignment, but keep only cols "x" and "y":
>>> af1, af2, af3 = pl.align_frames(
... df1, df2, df3, on="dt", select=["x", "y"], how="left"
... ) # doctest: +IGNORE_RESULT
#
# af1 af2 af3
# shape: (3, 3) shape: (3, 3) shape: (3, 3)
# ┌─────┬──────┐ ┌─────┬──────┐ ┌──────┬──────┐
# │ x ┆ y │ │ x ┆ y │ │ x ┆ y │
# │ --- ┆ --- │ │ --- ┆ --- │ │ --- ┆ --- │
# │ f64 ┆ f64 │ │ f64 ┆ f64 │ │ f64 ┆ f64 │
# ╞═════╪══════╡ ╞═════╪══════╡ ╞══════╪══════╡
# │ 3.5 ┆ 10.0 │ │ 3.5 ┆ 5.0 │ │ null ┆ null │
# ├╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌┼╌╌╌╌╌╌┤
# │ 4.0 ┆ 2.5 │ │ 8.0 ┆ 1.5 │ │ 5.0 ┆ 2.0 │
# ├╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌┼╌╌╌╌╌╌┤ ├╌╌╌╌╌╌┼╌╌╌╌╌╌┤
# │ 1.0 ┆ 1.5 │ │ 1.0 ┆ 12.0 │ │ 2.0 ┆ 2.5 │
# └─────┴──────┘ └─────┴──────┘ └──────┴──────┘
...
Now data is aligned, and you can easily calculate the row-wise dot product:
>>> (af1 * af2 * af3).fill_null(0).select(pl.sum_horizontal("*").alias("dot"))
shape: (3, 1)
┌───────┐
│ dot │
│ --- │
│ f64 │
╞═══════╡
│ 0.0 │
├╌╌╌╌╌╌╌┤
│ 167.5 │
├╌╌╌╌╌╌╌┤
│ 47.0 │
└───────┘
all(*names: 'str', ignore_nulls: 'bool' = True) -> 'Expr'
Either return an expression representing all columns, or evaluate a bitwise AND operation.
If no arguments are passed, this function is syntactic sugar for `col("*")`.
Otherwise, this function is syntactic sugar for `col(names).all()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
ignore_nulls
* If set to `True` (default), null values are ignored. If there
are no non-null values, the output is `True`.
* If set to `False`, `Kleene logic`_ is used to deal with nulls:
if the column contains any null values and no `False` values,
the output is null.
.. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
See Also
--------
all_horizontal
Examples
--------
Selecting all columns.
>>> df = pl.DataFrame(
... {
... "a": [True, False, True],
... "b": [False, False, False],
... }
... )
>>> df.select(pl.all().sum())
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ u32 ┆ u32 │
╞═════╪═════╡
│ 2 ┆ 0 │
└─────┴─────┘
Evaluate bitwise AND for a column.
>>> df.select(pl.all("a"))
shape: (1, 1)
┌───────┐
│ a │
│ --- │
│ bool │
╞═══════╡
│ false │
└───────┘
all_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Compute the logical AND horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Notes
-----
`Kleene logic`_ is used to deal with nulls: if the column contains any null values
and no `False` values, the output is null.
.. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [False, False, True, True, False, None],
... "b": [False, True, True, None, None, None],
... "c": ["u", "v", "w", "x", "y", "z"],
... }
... )
>>> df.with_columns(all=pl.all_horizontal("a", "b"))
shape: (6, 4)
┌───────┬───────┬─────┬───────┐
│ a ┆ b ┆ c ┆ all │
│ --- ┆ --- ┆ --- ┆ --- │
│ bool ┆ bool ┆ str ┆ bool │
╞═══════╪═══════╪═════╪═══════╡
│ false ┆ false ┆ u ┆ false │
│ false ┆ true ┆ v ┆ false │
│ true ┆ true ┆ w ┆ true │
│ true ┆ null ┆ x ┆ null │
│ false ┆ null ┆ y ┆ false │
│ null ┆ null ┆ z ┆ null │
└───────┴───────┴─────┴───────┘
any(*names: 'str', ignore_nulls: 'bool' = True) -> 'Expr | bool | None'
Evaluate a bitwise OR operation.
Syntactic sugar for `col(names).any()`.
See Also
--------
any_horizontal
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
ignore_nulls
* If set to `True` (default), null values are ignored. If there
are no non-null values, the output is `False`.
* If set to `False`, `Kleene logic`_ is used to deal with nulls:
if the column contains any null values and no `True` values,
the output is null.
.. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [True, False, True],
... "b": [False, False, False],
... }
... )
>>> df.select(pl.any("a"))
shape: (1, 1)
┌──────┐
│ a │
│ --- │
│ bool │
╞══════╡
│ true │
└──────┘
any_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Compute the logical OR horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Notes
-----
`Kleene logic`_ is used to deal with nulls: if the column contains any null values
and no `True` values, the output is null.
.. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [False, False, True, True, False, None],
... "b": [False, True, True, None, None, None],
... "c": ["u", "v", "w", "x", "y", "z"],
... }
... )
>>> df.with_columns(any=pl.any_horizontal("a", "b"))
shape: (6, 4)
┌───────┬───────┬─────┬───────┐
│ a ┆ b ┆ c ┆ any │
│ --- ┆ --- ┆ --- ┆ --- │
│ bool ┆ bool ┆ str ┆ bool │
╞═══════╪═══════╪═════╪═══════╡
│ false ┆ false ┆ u ┆ false │
│ false ┆ true ┆ v ┆ true │
│ true ┆ true ┆ w ┆ true │
│ true ┆ null ┆ x ┆ true │
│ false ┆ null ┆ y ┆ null │
│ null ┆ null ┆ z ┆ null │
└───────┴───────┴─────┴───────┘
approx_n_unique(*columns: 'str') -> 'Expr'
Approximate count of unique values.
This function is syntactic sugar for `pl.col(columns).approx_n_unique()`, and
uses the HyperLogLog++ algorithm for cardinality estimation.
Parameters
----------
columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 1],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.approx_n_unique("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 2 │
└─────┘
>>> df.select(pl.approx_n_unique("b", "c"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ u32 ┆ u32 │
╞═════╪═════╡
│ 3 ┆ 2 │
└─────┴─────┘
arange(start: 'int | IntoExprColumn' = 0, end: 'int | IntoExprColumn | None' = None, step: 'int' = 1, *, dtype: 'PolarsIntegerType | DataTypeExpr' = Int64, eager: 'bool' = False) -> 'Expr | Series'
Generate a range of integers.
Alias for :func:`int_range`.
Parameters
----------
start
Lower bound of the range (inclusive).
end
Upper bound of the range (exclusive).
step
Step size of the range.
dtype
Data type of the range. Defaults to `Int64`.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of integer data type `dtype`.
See Also
--------
int_range : Generate a range of integers.
int_ranges : Generate a range of integers for each row of the input columns.
Examples
--------
>>> pl.arange(0, 3, eager=True)
shape: (3,)
Series: 'literal' [i64]
[
0
1
2
]
arctan2(y: 'str | Expr', x: 'str | Expr') -> 'Expr'
Compute two argument arctan in radians.
Returns the angle (in radians) in the plane between the
positive x-axis and the ray from the origin to (x,y).
Parameters
----------
y
Column name or Expression.
x
Column name or Expression.
Examples
--------
>>> c = (2**0.5) / 2
>>> df = pl.DataFrame(
... {
... "y": [c, -c, c, -c],
... "x": [c, c, -c, -c],
... }
... )
>>> df.with_columns(pl.arctan2("y", "x").alias("atan2"))
shape: (4, 3)
┌───────────┬───────────┬───────────┐
│ y ┆ x ┆ atan2 │
│ --- ┆ --- ┆ --- │
│ f64 ┆ f64 ┆ f64 │
╞═══════════╪═══════════╪═══════════╡
│ 0.707107 ┆ 0.707107 ┆ 0.785398 │
│ -0.707107 ┆ 0.707107 ┆ -0.785398 │
│ 0.707107 ┆ -0.707107 ┆ 2.356194 │
│ -0.707107 ┆ -0.707107 ┆ -2.356194 │
└───────────┴───────────┴───────────┘
arctan2d(y: 'str | Expr', x: 'str | Expr') -> 'Expr'
Compute two argument arctan in degrees.
.. deprecated:: 1.0.0
Use `arctan2` followed by :meth:`Expr.degrees` instead.
Returns the angle (in degrees) in the plane between the positive x-axis
and the ray from the origin to (x,y).
Parameters
----------
y
Column name or Expression.
x
Column name or Expression.
Examples
--------
>>> c = (2**0.5) / 2
>>> df = pl.DataFrame(
... {
... "y": [c, -c, c, -c],
... "x": [c, c, -c, -c],
... }
... )
>>> df.select( # doctest: +SKIP
... pl.arctan2d("y", "x").alias("atan2d"),
... pl.arctan2("y", "x").alias("atan2"),
... )
shape: (4, 2)
┌────────┬───────────┐
│ atan2d ┆ atan2 │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞════════╪═══════════╡
│ 45.0 ┆ 0.785398 │
│ -45.0 ┆ -0.785398 │
│ 135.0 ┆ 2.356194 │
│ -135.0 ┆ -2.356194 │
└────────┴───────────┘
arg_sort_by(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False, multithreaded: 'bool' = True, maintain_order: 'bool' = False) -> 'Expr'
Return the row indices that would sort the column(s).
Parameters
----------
exprs
Column(s) to arg sort by. Accepts expression input. Strings are parsed as column
names.
*more_exprs
Additional columns to arg sort by, specified as positional arguments.
descending
Sort in descending order. When sorting by multiple columns, can be specified
per column by passing a sequence of booleans.
nulls_last
Place null values last.
multithreaded
Sort using multiple threads.
maintain_order
Whether the order should be maintained if elements are equal.
See Also
--------
Expr.gather: Take values by index.
Expr.rank : Get the rank of each row.
Examples
--------
Pass a single column name to compute the arg sort by that column.
>>> df = pl.DataFrame(
... {
... "a": [0, 1, 1, 0],
... "b": [3, 2, 3, 2],
... "c": [1, 2, 3, 4],
... }
... )
>>> df.select(pl.arg_sort_by("a"))
shape: (4, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 0 │
│ 3 │
│ 1 │
│ 2 │
└─────┘
Compute the arg sort by multiple columns by either passing a list of columns, or by
specifying each column as a positional argument.
>>> df.select(pl.arg_sort_by(["a", "b"], descending=True))
shape: (4, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 2 │
│ 1 │
│ 0 │
│ 3 │
└─────┘
Use gather to apply the arg sort to other columns.
>>> df.select(pl.col("c").gather(pl.arg_sort_by("a")))
shape: (4, 1)
┌─────┐
│ c │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 4 │
│ 2 │
│ 3 │
└─────┘
arg_where(condition: 'Expr | Series', *, eager: 'bool' = False) -> 'Expr | Series'
Return indices where `condition` evaluates `True`.
Parameters
----------
condition
Boolean expression to evaluate
eager
Evaluate immediately and return a `Series`; this requires that the given
condition is itself a `Series`. If set to `False` (default), return
an expression instead.
See Also
--------
Series.arg_true : Return indices where Series is True
Examples
--------
>>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5]})
>>> df.select(
... [
... pl.arg_where(pl.col("a") % 2 == 0),
... ]
... ).to_series()
shape: (2,)
Series: 'a' [u32]
[
1
3
]
business_day_count(start: 'date | IntoExprColumn', end: 'date | IntoExprColumn', *, week_mask: 'Iterable[bool]' = (True, True, True, True, True, False, False), holidays: 'Iterable[date] | Expr | Series' = ()) -> 'Expr'
Count the number of business days between `start` and `end` (not including `end`).
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
.. versionchanged:: 1.27.0
Parameters after `start` and `end` should now be passed as keyword arguments.
Parameters
----------
start
Start dates.
end
End dates.
week_mask
Which days of the week to count. The default is Monday to Friday.
If you wanted to count only Monday to Thursday, you would pass
`(True, True, True, True, False, False, False)`.
holidays
Holidays to exclude from the count. The Python package
`python-holidays <https://github.com/vacanza/python-holidays>`_
may come in handy here. You can install it with ``pip install holidays``,
and then, to get all Dutch holidays for years 2020-2024:
.. code-block:: python
import holidays
my_holidays = holidays.country_holidays("NL", years=range(2020, 2025))
and pass `holidays=my_holidays` when you call `business_day_count`.
Returns
-------
Expr
Examples
--------
>>> from datetime import date
>>> df = pl.DataFrame(
... {
... "start": [date(2020, 1, 1), date(2020, 1, 2)],
... "end": [date(2020, 1, 2), date(2020, 1, 10)],
... }
... )
>>> df.with_columns(
... business_day_count=pl.business_day_count("start", "end"),
... )
shape: (2, 3)
┌────────────┬────────────┬────────────────────┐
│ start ┆ end ┆ business_day_count │
│ --- ┆ --- ┆ --- │
│ date ┆ date ┆ i32 │
╞════════════╪════════════╪════════════════════╡
│ 2020-01-01 ┆ 2020-01-02 ┆ 1 │
│ 2020-01-02 ┆ 2020-01-10 ┆ 6 │
└────────────┴────────────┴────────────────────┘
Note how the business day count is 6 (as opposed a regular day count of 8)
due to the weekend (2020-01-04 - 2020-01-05) not being counted.
You can pass a custom weekend - for example, if you only take Sunday off:
>>> week_mask = (True, True, True, True, True, True, False)
>>> df.with_columns(
... business_day_count=pl.business_day_count(
... "start", "end", week_mask=week_mask
... ),
... )
shape: (2, 3)
┌────────────┬────────────┬────────────────────┐
│ start ┆ end ┆ business_day_count │
│ --- ┆ --- ┆ --- │
│ date ┆ date ┆ i32 │
╞════════════╪════════════╪════════════════════╡
│ 2020-01-01 ┆ 2020-01-02 ┆ 1 │
│ 2020-01-02 ┆ 2020-01-10 ┆ 7 │
└────────────┴────────────┴────────────────────┘
You can also pass a list of holidays to exclude from the count:
>>> from datetime import date
>>> holidays = [date(2020, 1, 1), date(2020, 1, 2)]
>>> df.with_columns(
... business_day_count=pl.business_day_count("start", "end", holidays=holidays)
... )
shape: (2, 3)
┌────────────┬────────────┬────────────────────┐
│ start ┆ end ┆ business_day_count │
│ --- ┆ --- ┆ --- │
│ date ┆ date ┆ i32 │
╞════════════╪════════════╪════════════════════╡
│ 2020-01-01 ┆ 2020-01-02 ┆ 0 │
│ 2020-01-02 ┆ 2020-01-10 ┆ 5 │
└────────────┴────────────┴────────────────────┘
coalesce(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr', eager: 'bool' = False) -> 'Expr | Series'
Folds the columns from left to right, keeping the first non-null value.
Parameters
----------
exprs
Columns to coalesce. Accepts expression input. Strings are parsed as column
names, other non-expression inputs are parsed as literals.
*more_exprs
Additional columns to coalesce, specified as positional arguments.
eager
Evaluate immediately and return a `Series`; this requires that at least one
of the given arguments is a `Series`. If set to `False` (default), return
an expression instead.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, None, None, None],
... "b": [1, 2, None, None],
... "c": [5, None, 3, None],
... }
... )
>>> df.with_columns(pl.coalesce("a", "b", "c", 10).alias("d"))
shape: (4, 4)
┌──────┬──────┬──────┬─────┐
│ a ┆ b ┆ c ┆ d │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞══════╪══════╪══════╪═════╡
│ 1 ┆ 1 ┆ 5 ┆ 1 │
│ null ┆ 2 ┆ null ┆ 2 │
│ null ┆ null ┆ 3 ┆ 3 │
│ null ┆ null ┆ null ┆ 10 │
└──────┴──────┴──────┴─────┘
>>> df.with_columns(pl.coalesce(pl.col(["a", "b", "c"]), 10.0).alias("d"))
shape: (4, 4)
┌──────┬──────┬──────┬──────┐
│ a ┆ b ┆ c ┆ d │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ f64 │
╞══════╪══════╪══════╪══════╡
│ 1 ┆ 1 ┆ 5 ┆ 1.0 │
│ null ┆ 2 ┆ null ┆ 2.0 │
│ null ┆ null ┆ 3 ┆ 3.0 │
│ null ┆ null ┆ null ┆ 10.0 │
└──────┴──────┴──────┴──────┘
>>> s1 = pl.Series("a", [None, 2, None])
>>> s2 = pl.Series("b", [1, None, 3])
>>> pl.coalesce(s1, s2, eager=True)
shape: (3,)
Series: 'a' [i64]
[
1
2
3
]
collect_all(lazy_frames: 'Iterable[LazyFrame]', *, type_coercion: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, no_optimization: 'bool' = False, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, engine: 'EngineType' = 'auto', lazy: 'bool' = False) -> 'list[DataFrame] | LazyFrame'
Collect multiple LazyFrames at the same time.
This can run all the computation graphs in parallel or combined.
Common Subplan Elimination is applied on the combined plan, meaning
that diverging queries will run only once.
Parameters
----------
lazy_frames
A list of LazyFrames to collect.
type_coercion
Do type coercion optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
predicate_pushdown
Do predicate pushdown optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
projection_pushdown
Do projection pushdown optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
simplify_expression
Run simplify expressions optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
no_optimization
Turn off optimizations.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
slice_pushdown
Slice pushdown optimization.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
comm_subplan_elim
Will try to cache branching subplans that occur on self-joins or unions.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
comm_subexpr_elim
Common subexpressions will be cached and reused.
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
cluster_with_columns
Combine sequential independent calls to with_columns
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
collapse_joins
Collapse a join and filters into a faster join
.. deprecated:: 1.30.0
Use the `optimizations` parameters.
optimizations
The optimization passes done during query optimization.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
engine
Select the engine used to process the query (default ``"auto"``):
* ``"auto"``: use the engine set by
:meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
back to ``"in-memory"`` if unset (this default may change in
a future release).
* ``"in-memory"``: use the in-memory engine, this is the default engine.
* ``"streaming"``: use the streaming engine, which processes
queries in batches, reducing memory pressure and often
outperforming the in-memory engine. This will soon become
the default engine of Polars.
* ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
``cudf-polars``). Pass a :class:`~.GPUEngine` object for
fine-grained control (e.g. device selection on multi-GPU
systems).
If the selected engine cannot run the query, Polars falls back to
the in-memory engine.
.. note::
The GPU engine does not support async, or running in the
background. If either are enabled, then GPU execution is switched off.
lazy:
Return as LazyFrame that can be collected later.
This is only correct if all inputs sink to disk.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Returns
-------
list of DataFrames
The collected DataFrames, returned in the same order as the input LazyFrames.
collect_all_async(lazy_frames: 'Iterable[LazyFrame]', *, gevent: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'Awaitable[list[DataFrame]] | _GeventDataFrameResult[list[DataFrame]]'
Collect multiple LazyFrames at the same time asynchronously in thread pool.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Collects into a list of DataFrame (like :func:`polars.collect_all`),
but instead of returning them directly, they are scheduled to be collected
inside thread pool, while this method returns almost instantly.
May be useful if you use gevent or asyncio and want to release control to other
greenlets/tasks while LazyFrames are being collected.
Parameters
----------
lazy_frames
A list of LazyFrames to collect.
gevent
Return wrapper to `gevent.event.AsyncResult` instead of Awaitable
optimizations
The optimization passes done during query optimization.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
engine
Select the engine used to process the query (default ``"auto"``):
* ``"auto"``: use the engine set by
:meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
back to ``"in-memory"`` if unset (this default may change in
a future release).
* ``"in-memory"``: use the in-memory engine, this is the default engine.
* ``"streaming"``: use the streaming engine, which processes
queries in batches, reducing memory pressure and often
outperforming the in-memory engine. This will soon become
the default engine of Polars.
* ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
``cudf-polars``). Pass a :class:`~.GPUEngine` object for
fine-grained control (e.g. device selection on multi-GPU
systems).
If the selected engine cannot run the query, Polars falls back to
the in-memory engine.
.. note::
The GPU engine does not support async, or running in the
background. If either are enabled, then GPU execution is switched off.
See Also
--------
polars.collect_all : Collect multiple LazyFrames at the same time.
LazyFrame.collect_async : To collect single frame.
Notes
-----
In case of error `set_exception` is used on
`asyncio.Future`/`gevent.event.AsyncResult` and will be reraised by them.
Returns
-------
If `gevent=False` (default) then returns awaitable.
If `gevent=True` then returns wrapper that has
`.get(block=True, timeout=None)` method.
concat(items: 'Iterable[PolarsType]', *, how: 'ConcatMethod' = 'vertical', rechunk: 'bool' = False, parallel: 'bool' = True, strict: 'bool | None' = None) -> 'PolarsType'
Combine multiple DataFrames, LazyFrames, or Series into a single object.
Parameters
----------
items
DataFrames, LazyFrames, or Series to concatenate.
how : {'vertical', 'vertical_relaxed', 'diagonal', 'diagonal_relaxed', 'horizontal', 'horizontal_extend', 'align', 'align_full', 'align_inner', 'align_left', 'align_right'}
Note that `Series` only support the `vertical` strategy.
* vertical: Applies multiple `vstack` operations.
* vertical_relaxed: Same as `vertical`, but additionally coerces columns to
their common supertype *if* they are mismatched (eg: Int32 → Int64).
* diagonal: Finds a union between the column schemas and fills missing column
values with `null`.
* diagonal_relaxed: Same as `diagonal`, but additionally coerces columns to
their common supertype *if* they are mismatched (eg: Int32 → Int64).
* horizontal: Stacks Series from DataFrames horizontally. All input frames
must have the same height; raises a `ShapeError` otherwise.
* horizontal_extend: Same as `horizontal`, but pads shorter frames with `null`
if the lengths don't match.
* align, align_full, align_left, align_right: Combines frames horizontally,
auto-determining the common key columns and aligning rows using the same
logic as `align_frames` (note that "align" is an alias for "align_full").
The "align" strategy determines the type of join used to align the frames,
equivalent to the "how" parameter on `align_frames`. Note that the common
join columns are automatically coalesced, but other column collisions
will raise an error (if you need more control over this you should use
a suitable `join` method directly).
rechunk
Make sure that the result data is in contiguous memory.
parallel
Only relevant for LazyFrames. This determines if the concatenated
lazy computations may be executed in parallel.
strict
When how=`horizontal`, require all DataFrames to be the same height, raising an error if not.
.. deprecated:: 1.42.1
Use `how='horizontal'` (equal heights) or
`how='horizontal_extend'` (pad with null) instead.
Examples
--------
>>> df1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df2 = pl.DataFrame({"a": [2], "b": [4]})
>>> pl.concat([df1, df2]) # default is 'vertical' strategy
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 3 │
│ 2 ┆ 4 │
└─────┴─────┘
>>> df1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df2 = pl.DataFrame({"a": [2.5], "b": [4]})
>>> pl.concat([df1, df2], how="vertical_relaxed") # 'a' coerced into f64
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ f64 ┆ i64 │
╞═════╪═════╡
│ 1.0 ┆ 3 │
│ 2.5 ┆ 4 │
└─────┴─────┘
>>> df_h1 = pl.DataFrame({"l1": [1, 2], "l2": [3, 4]})
>>> df_h2 = pl.DataFrame({"r1": [5, 6], "r2": [7, 8], "r3": [9, 10]})
>>> pl.concat([df_h1, df_h2], how="horizontal_extend")
shape: (2, 5)
┌─────┬─────┬─────┬─────┬─────┐
│ l1 ┆ l2 ┆ r1 ┆ r2 ┆ r3 │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 5 ┆ 7 ┆ 9 │
│ 2 ┆ 4 ┆ 6 ┆ 8 ┆ 10 │
└─────┴─────┴─────┴─────┴─────┘
The "diagonal" strategy allows for some frames to have missing columns,
the values for which are filled with `null`:
>>> df_d1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df_d2 = pl.DataFrame({"a": [2], "c": [4]})
>>> pl.concat([df_d1, df_d2], how="diagonal")
shape: (2, 3)
┌─────┬──────┬──────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╡
│ 1 ┆ 3 ┆ null │
│ 2 ┆ null ┆ 4 │
└─────┴──────┴──────┘
The "align" strategies require at least one common column to align on:
>>> df_a1 = pl.DataFrame({"id": [1, 2], "x": [3, 4]})
>>> df_a2 = pl.DataFrame({"id": [2, 3], "y": [5, 6]})
>>> df_a3 = pl.DataFrame({"id": [1, 3], "z": [7, 8]})
>>> pl.concat([df_a1, df_a2, df_a3], how="align") # equivalent to "align_full"
shape: (3, 4)
┌─────┬──────┬──────┬──────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╪══════╡
│ 1 ┆ 3 ┆ null ┆ 7 │
│ 2 ┆ 4 ┆ 5 ┆ null │
│ 3 ┆ null ┆ 6 ┆ 8 │
└─────┴──────┴──────┴──────┘
>>> pl.concat([df_a1, df_a2, df_a3], how="align_left")
shape: (2, 4)
┌─────┬─────┬──────┬──────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪══════╪══════╡
│ 1 ┆ 3 ┆ null ┆ 7 │
│ 2 ┆ 4 ┆ 5 ┆ null │
└─────┴─────┴──────┴──────┘
>>> pl.concat([df_a1, df_a2, df_a3], how="align_right")
shape: (2, 4)
┌─────┬──────┬──────┬─────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╪═════╡
│ 1 ┆ null ┆ null ┆ 7 │
│ 3 ┆ null ┆ 6 ┆ 8 │
└─────┴──────┴──────┴─────┘
>>> pl.concat([df_a1, df_a2, df_a3], how="align_inner")
shape: (0, 4)
┌─────┬─────┬─────┬─────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
└─────┴─────┴─────┴─────┘
concat_arr(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr') -> 'Expr'
Horizontally concatenate columns into a single array column.
Non-array columns are reshaped to a unit-width array. All columns must have
a dtype of either `pl.Array(<DataType>, width)` or `pl.<DataType>`.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
exprs
Columns to concatenate into a single array column. Accepts expression input.
Strings are parsed as column names, other non-expression inputs are parsed as
literals.
*more_exprs
Additional columns to concatenate into a single array column, specified as
positional arguments.
Examples
--------
Concatenate 2 array columns:
>>> (
... pl.select(
... a=pl.Series([[1], [3], None], dtype=pl.Array(pl.Int64, 1)),
... b=pl.Series([[3], [None], [5]], dtype=pl.Array(pl.Int64, 1)),
... ).with_columns(
... pl.concat_arr("a", "b").alias("concat_arr(a, b)"),
... pl.concat_arr("a", pl.first("b")).alias("concat_arr(a, first(b))"),
... )
... )
shape: (3, 4)
┌───────────────┬───────────────┬──────────────────┬─────────────────────────┐
│ a ┆ b ┆ concat_arr(a, b) ┆ concat_arr(a, first(b)) │
│ --- ┆ --- ┆ --- ┆ --- │
│ array[i64, 1] ┆ array[i64, 1] ┆ array[i64, 2] ┆ array[i64, 2] │
╞═══════════════╪═══════════════╪══════════════════╪═════════════════════════╡
│ [1] ┆ [3] ┆ [1, 3] ┆ [1, 3] │
│ [3] ┆ [null] ┆ [3, null] ┆ [3, 3] │
│ null ┆ [5] ┆ null ┆ null │
└───────────────┴───────────────┴──────────────────┴─────────────────────────┘
Concatenate non-array columns:
>>> (
... pl.select(
... c=pl.Series([None, 5, 6], dtype=pl.Int64),
... )
... .with_columns(d=pl.col("c").reverse())
... .with_columns(
... pl.concat_arr("c", "d").alias("concat_arr(c, d)"),
... )
... )
shape: (3, 3)
┌──────┬──────┬──────────────────┐
│ c ┆ d ┆ concat_arr(c, d) │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ array[i64, 2] │
╞══════╪══════╪══════════════════╡
│ null ┆ 6 ┆ [null, 6] │
│ 5 ┆ 5 ┆ [5, 5] │
│ 6 ┆ null ┆ [6, null] │
└──────┴──────┴──────────────────┘
Concatenate mixed array and non-array columns:
>>> (
... pl.select(
... a=pl.Series([[1], [3], None], dtype=pl.Array(pl.Int64, 1)),
... b=pl.Series([[3], [None], [5]], dtype=pl.Array(pl.Int64, 1)),
... c=pl.Series([None, 5, 6], dtype=pl.Int64),
... ).with_columns(
... pl.concat_arr("a", "b", "c").alias("concat_arr(a, b, c)"),
... )
... )
shape: (3, 4)
┌───────────────┬───────────────┬──────┬─────────────────────┐
│ a ┆ b ┆ c ┆ concat_arr(a, b, c) │
│ --- ┆ --- ┆ --- ┆ --- │
│ array[i64, 1] ┆ array[i64, 1] ┆ i64 ┆ array[i64, 3] │
╞═══════════════╪═══════════════╪══════╪═════════════════════╡
│ [1] ┆ [3] ┆ null ┆ [1, 3, null] │
│ [3] ┆ [null] ┆ 5 ┆ [3, null, 5] │
│ null ┆ [5] ┆ 6 ┆ null │
└───────────────┴───────────────┴──────┴─────────────────────┘
Unit-length columns are broadcasted:
>>> (
... pl.select(
... a=pl.Series([1, 3, None]),
... ).with_columns(
... pl.concat_arr("a", pl.lit(0, dtype=pl.Int64)).alias("concat_arr(a, 0)"),
... pl.concat_arr("a", pl.sum("a")).alias("concat_arr(a, sum(a))"),
... pl.concat_arr("a", pl.max("a")).alias("concat_arr(a, max(a))"),
... )
... )
shape: (3, 4)
┌──────┬──────────────────┬───────────────────────┬───────────────────────┐
│ a ┆ concat_arr(a, 0) ┆ concat_arr(a, sum(a)) ┆ concat_arr(a, max(a)) │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ array[i64, 2] ┆ array[i64, 2] ┆ array[i64, 2] │
╞══════╪══════════════════╪═══════════════════════╪═══════════════════════╡
│ 1 ┆ [1, 0] ┆ [1, 4] ┆ [1, 3] │
│ 3 ┆ [3, 0] ┆ [3, 4] ┆ [3, 3] │
│ null ┆ [null, 0] ┆ [null, 4] ┆ [null, 3] │
└──────┴──────────────────┴───────────────────────┴───────────────────────┘
concat_list(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr') -> 'Expr'
Horizontally concatenate columns into a single list column.
Operates in linear time.
Parameters
----------
exprs
Columns to concatenate into a single list column. Accepts expression input.
Strings are parsed as column names, other non-expression inputs are parsed as
literals.
*more_exprs
Additional columns to concatenate into a single list column, specified as
positional arguments.
Examples
--------
Concatenate two existing list columns. Null values are propagated.
>>> df = pl.DataFrame({"a": [[1, 2], [3], [4, 5]], "b": [[4], [], None]})
>>> df.with_columns(concat_list=pl.concat_list("a", "b"))
shape: (3, 3)
┌───────────┬───────────┬─────────────┐
│ a ┆ b ┆ concat_list │
│ --- ┆ --- ┆ --- │
│ list[i64] ┆ list[i64] ┆ list[i64] │
╞═══════════╪═══════════╪═════════════╡
│ [1, 2] ┆ [4] ┆ [1, 2, 4] │
│ [3] ┆ [] ┆ [3] │
│ [4, 5] ┆ null ┆ null │
└───────────┴───────────┴─────────────┘
Non-list columns are cast to a list before concatenation. The output data type
is the supertype of the concatenated columns.
>>> df.select("a", concat_list=pl.concat_list("a", pl.lit("x")))
shape: (3, 2)
┌───────────┬─────────────────┐
│ a ┆ concat_list │
│ --- ┆ --- │
│ list[i64] ┆ list[str] │
╞═══════════╪═════════════════╡
│ [1, 2] ┆ ["1", "2", "x"] │
│ [3] ┆ ["3", "x"] │
│ [4, 5] ┆ ["4", "5", "x"] │
└───────────┴─────────────────┘
Create lagged columns and collect them into a list. This mimics a rolling window.
>>> df = pl.DataFrame({"A": [1.0, 2.0, 9.0, 2.0, 13.0]})
>>> df = df.select([pl.col("A").shift(i).alias(f"A_lag_{i}") for i in range(3)])
>>> df.select(
... pl.concat_list([f"A_lag_{i}" for i in range(3)][::-1]).alias("A_rolling")
... )
shape: (5, 1)
┌───────────────────┐
│ A_rolling │
│ --- │
│ list[f64] │
╞═══════════════════╡
│ [null, null, 1.0] │
│ [null, 1.0, 2.0] │
│ [1.0, 2.0, 9.0] │
│ [2.0, 9.0, 2.0] │
│ [9.0, 2.0, 13.0] │
└───────────────────┘
concat_str(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr', separator: 'str' = '', ignore_nulls: 'bool' = False) -> 'Expr'
Horizontally concatenate columns into a single string column.
Operates in linear time.
Parameters
----------
exprs
Columns to concatenate into a single string column. Accepts expression input.
Strings are parsed as column names, other non-expression inputs are parsed as
literals. Non-`String` columns are cast to `String`.
*more_exprs
Additional columns to concatenate into a single string column, specified as
positional arguments.
separator
String that will be used to separate the values of each column.
ignore_nulls
Ignore null values (default is ``False``).
If set to ``False``, null values will be propagated.
if the row contains any null values, the output is null.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": ["dogs", "cats", None],
... "c": ["play", "swim", "walk"],
... }
... )
>>> df.with_columns(
... pl.concat_str(
... [
... pl.col("a") * 2,
... pl.col("b"),
... pl.col("c"),
... ],
... separator=" ",
... ).alias("full_sentence"),
... )
shape: (3, 4)
┌─────┬──────┬──────┬───────────────┐
│ a ┆ b ┆ c ┆ full_sentence │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str ┆ str │
╞═════╪══════╪══════╪═══════════════╡
│ 1 ┆ dogs ┆ play ┆ 2 dogs play │
│ 2 ┆ cats ┆ swim ┆ 4 cats swim │
│ 3 ┆ null ┆ walk ┆ null │
└─────┴──────┴──────┴───────────────┘
corr(a: 'IntoExpr', b: 'IntoExpr', *, method: 'CorrelationMethod' = 'pearson', ddof: 'int | None' = None, propagate_nans: 'bool' = False, eager: 'bool' = False) -> 'Expr | Series'
Compute the Pearson's or Spearman rank correlation between two columns.
Parameters
----------
a
Column name or Expression.
b
Column name or Expression.
ddof
Has no effect, do not use.
.. deprecated:: 1.17.0
method : {'pearson', 'spearman'}
Correlation method.
propagate_nans
If `True` any `NaN` encountered will lead to `NaN` in the output.
Defaults to `False` where `NaN` are regarded as larger than any finite number
and thus lead to the highest rank.
eager
Evaluate immediately and return a `Series`; this requires that at least one
of the given arguments is a `Series`. If set to `False` (default), return
an expression instead.
Examples
--------
Pearson's correlation:
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.corr("a", "b"))
shape: (1, 1)
┌──────────┐
│ a │
│ --- │
│ f64 │
╞══════════╡
│ 0.544705 │
└──────────┘
Spearman rank correlation:
>>> df.select(pl.corr("a", "b", method="spearman"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ f64 │
╞═════╡
│ 0.5 │
└─────┘
Eager evaluation:
>>> s1 = pl.Series("a", [1, 8, 3])
>>> s2 = pl.Series("b", [4, 5, 2])
>>> pl.corr(s1, s2, eager=True)
shape: (1,)
Series: 'a' [f64]
[
0.544705
]
>>> pl.corr(s1, s2, method="spearman", eager=True)
shape: (1,)
Series: 'a' [f64]
[
0.5
]
count(*columns: 'str') -> 'Expr'
Return the number of non-null values in the column.
This function is syntactic sugar for `col(columns).count()`.
Calling this function without any arguments returns the number of rows in the
context. **This way of using the function is deprecated.** Please use :func:`len`
instead.
Parameters
----------
*columns
One or more column names.
Returns
-------
Expr
Expression of data type :class:`UInt32`.
See Also
--------
Expr.count
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, None],
... "b": [3, None, None],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.count("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 2 │
└─────┘
Return the number of non-null values in multiple columns.
>>> df.select(pl.count("b", "c"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ u32 ┆ u32 │
╞═════╪═════╡
│ 1 ┆ 3 │
└─────┴─────┘
Return the number of rows in a context. **This way of using the function is
deprecated.** Please use :func:`len` instead.
>>> df.select(pl.count()) # doctest: +SKIP
shape: (1, 1)
┌───────┐
│ count │
│ --- │
│ u32 │
╞═══════╡
│ 3 │
└───────┘
cov(a: 'IntoExpr', b: 'IntoExpr', *, ddof: 'int' = 1, eager: 'bool' = False) -> 'Expr | Series'
Compute the covariance between two columns/ expressions.
Parameters
----------
a
Column name or Expression.
b
Column name or Expression.
ddof
"Delta Degrees of Freedom": the divisor used in the calculation is N - ddof,
where N represents the number of elements.
By default ddof is 1.
eager
Evaluate immediately and return a `Series`; this requires that at least one
of the given arguments is a `Series`. If set to `False` (default), return
an expression instead.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... },
... )
>>> df.select(
... x=pl.cov("a", "b"),
... y=pl.cov("a", "b", ddof=2),
... )
shape: (1, 2)
┌─────┬─────┐
│ x ┆ y │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞═════╪═════╡
│ 3.0 ┆ 6.0 │
└─────┴─────┘
Eager evaluation:
>>> s1 = pl.Series("a", [1, 8, 3])
>>> s2 = pl.Series("b", [4, 5, 2])
>>> pl.cov(s1, s2, eager=True)
shape: (1,)
Series: 'a' [f64]
[
3.0
]
cum_count(*columns: 'str', reverse: 'bool' = False) -> 'Expr'
Return the cumulative count of the non-null values in the column.
This function is syntactic sugar for `col(columns).cum_count()`.
Parameters
----------
*columns
Name(s) of the columns to use.
reverse
Reverse the operation.
Examples
--------
>>> df = pl.DataFrame({"a": [1, 2, None], "b": [3, None, None]})
>>> df.with_columns(
... ca=pl.cum_count("a"),
... cb=pl.cum_count("b"),
... )
shape: (3, 4)
┌──────┬──────┬─────┬─────┐
│ a ┆ b ┆ ca ┆ cb │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ u32 ┆ u32 │
╞══════╪══════╪═════╪═════╡
│ 1 ┆ 3 ┆ 1 ┆ 1 │
│ 2 ┆ null ┆ 2 ┆ 1 │
│ null ┆ null ┆ 2 ┆ 1 │
└──────┴──────┴─────┴─────┘
cum_fold(acc: 'IntoExpr', function: 'Callable[[Series, Series], Series]', exprs: 'Sequence[Expr | str] | Expr', *, returns_scalar: 'bool' = False, return_dtype: 'pl.DataTypeExpr | PolarsDataType | None' = None, include_init: 'bool' = False) -> 'Expr'
Cumulatively fold horizontally across columns with a left fold.
Every cumulative result is added as a separate field in a Struct column.
Parameters
----------
acc
Accumulator expression. This is the value that will be initialized when the fold
starts. For a sum this could for instance be lit(0).
function
Function to apply over the accumulator and the value.
Fn(acc, value) -> new_value
exprs
Expressions to aggregate over. May also be a wildcard expression.
returns_scalar
Whether or not `function` applied returns a scalar. This must be set correctly
by the user.
return_dtype
Output datatype.
If not set, the dtype will be inferred based on the dtype of the accumulator.
include_init
Include the initial accumulator state as struct field.
Notes
-----
If you simply want the first encountered expression as accumulator,
consider using :func:`cum_reduce`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [3, 4, 5],
... "c": [5, 6, 7],
... }
... )
>>> df.with_columns(
... pl.cum_fold(acc=pl.lit(1), function=lambda acc, x: acc + x, exprs=pl.all())
... )
shape: (3, 4)
┌─────┬─────┬─────┬───────────┐
│ a ┆ b ┆ c ┆ cum_fold │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ struct[3] │
╞═════╪═════╪═════╪═══════════╡
│ 1 ┆ 3 ┆ 5 ┆ {2,5,10} │
│ 2 ┆ 4 ┆ 6 ┆ {3,7,13} │
│ 3 ┆ 5 ┆ 7 ┆ {4,9,16} │
└─────┴─────┴─────┴───────────┘
cum_reduce(function: 'Callable[[Series, Series], Series]', exprs: 'Sequence[Expr | str] | Expr', *, returns_scalar: 'bool' = False, return_dtype: 'pl.DataTypeExpr | PolarsDataType | None' = None) -> 'Expr'
Cumulatively reduce horizontally across columns with a left fold.
Every cumulative result is added as a separate field in a Struct column.
Parameters
----------
function
Function to apply over the accumulator and the value.
Fn(acc, value) -> new_value
exprs
Expressions to aggregate over. May also be a wildcard expression.
returns_scalar
Whether or not `function` applied returns a scalar. This must be set correctly
by the user.
return_dtype
Output datatype.
If not set, the dtype will be inferred based on the dtype of the input
expressions.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [3, 4, 5],
... "c": [5, 6, 7],
... }
... )
>>> df.with_columns(pl.cum_reduce(function=lambda acc, x: acc + x, exprs=pl.all()))
shape: (3, 4)
┌─────┬─────┬─────┬────────────┐
│ a ┆ b ┆ c ┆ cum_reduce │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ struct[3] │
╞═════╪═════╪═════╪════════════╡
│ 1 ┆ 3 ┆ 5 ┆ {1,4,9} │
│ 2 ┆ 4 ┆ 6 ┆ {2,6,12} │
│ 3 ┆ 5 ┆ 7 ┆ {3,8,15} │
└─────┴─────┴─────┴────────────┘
cum_sum(*names: 'str') -> 'Expr'
Cumulatively sum all values.
Syntactic sugar for `col(names).cum_sum()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
See Also
--------
cumsum_horizontal
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [4, 5, 6],
... }
... )
>>> df.select(pl.cum_sum("a"))
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 3 │
│ 6 │
└─────┘
cum_sum_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Cumulatively sum all values horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(pl.cum_sum_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬───────────┐
│ a ┆ b ┆ c ┆ cum_sum │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ struct[2] │
╞═════╪══════╪═════╪═══════════╡
│ 1 ┆ 4 ┆ x ┆ {1,5} │
│ 8 ┆ 5 ┆ y ┆ {8,13} │
│ 3 ┆ null ┆ z ┆ {3,null} │
└─────┴──────┴─────┴───────────┘
date = date_(year: 'Expr | str | int', month: 'Expr | str | int', day: 'Expr | str | int') -> 'Expr'
Create a Polars literal expression of type Date.
Parameters
----------
year
column or literal.
month
column or literal, ranging from 1-12.
day
column or literal, ranging from 1-31.
Returns
-------
Expr
Expression of data type :class:`Date`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "month": [1, 2, 3],
... "day": [4, 5, 6],
... }
... )
>>> df.with_columns(pl.date(2024, pl.col("month"), pl.col("day")))
shape: (3, 3)
┌───────┬─────┬────────────┐
│ month ┆ day ┆ date │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ date │
╞═══════╪═════╪════════════╡
│ 1 ┆ 4 ┆ 2024-01-04 │
│ 2 ┆ 5 ┆ 2024-02-05 │
│ 3 ┆ 6 ┆ 2024-03-06 │
└───────┴─────┴────────────┘
We can also use `pl.date` for filtering:
>>> from datetime import date
>>> df = pl.DataFrame(
... {
... "start": [date(2024, 1, 1), date(2024, 1, 1), date(2024, 1, 1)],
... "end": [date(2024, 5, 1), date(2024, 7, 1), date(2024, 9, 1)],
... }
... )
>>> df.filter(pl.col("end") > pl.date(2024, 6, 1))
shape: (2, 2)
┌────────────┬────────────┐
│ start ┆ end │
│ --- ┆ --- │
│ date ┆ date │
╞════════════╪════════════╡
│ 2024-01-01 ┆ 2024-07-01 │
│ 2024-01-01 ┆ 2024-09-01 │
└────────────┴────────────┘
date_range(start: 'date | datetime | IntoExprColumn', end: 'date | datetime | IntoExprColumn', interval: 'str | timedelta' = '1d', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Series | Expr'
Generate a date range.
Parameters
----------
start
Lower bound of the date range.
end
Upper bound of the date range.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
Must consist of full days.
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type :class:`Date`.
See Also
--------
date_ranges
datetime_range
Notes
-----
`interval` is created according to the following string language:
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"1w2d" # 1 week, 2 days
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
Examples
--------
Using Polars duration string to specify the interval:
>>> from datetime import date
>>> pl.date_range(date(2022, 1, 1), date(2022, 3, 1), "1mo", eager=True).alias(
... "date"
... )
shape: (3,)
Series: 'date' [date]
[
2022-01-01
2022-02-01
2022-03-01
]
Using `timedelta` object to specify the interval:
>>> from datetime import timedelta
>>> pl.date_range(
... date(1985, 1, 1),
... date(1985, 1, 10),
... timedelta(days=2),
... eager=True,
... ).alias("date")
shape: (5,)
Series: 'date' [date]
[
1985-01-01
1985-01-03
1985-01-05
1985-01-07
1985-01-09
]
Omit `eager=True` if you want to use `date_range` as an expression:
>>> df = pl.DataFrame(
... {
... "date": [
... date(2024, 1, 1),
... date(2024, 1, 2),
... date(2024, 1, 1),
... date(2024, 1, 3),
... ],
... "key": ["one", "one", "two", "two"],
... }
... )
>>> result = (
... df.group_by("key")
... .agg(pl.date_range(pl.col("date").min(), pl.col("date").max()))
... .sort("key")
... )
>>> with pl.Config(fmt_str_lengths=50):
... print(result)
shape: (2, 2)
┌─────┬──────────────────────────────────────┐
│ key ┆ date │
│ --- ┆ --- │
│ str ┆ list[date] │
╞═════╪══════════════════════════════════════╡
│ one ┆ [2024-01-01, 2024-01-02] │
│ two ┆ [2024-01-01, 2024-01-02, 2024-01-03] │
└─────┴──────────────────────────────────────┘
date_ranges(start: 'date | datetime | IntoExprColumn', end: 'date | datetime | IntoExprColumn', interval: 'str | timedelta' = '1d', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Series | Expr'
Create a column of date ranges.
Parameters
----------
start
Lower bound of the date range.
end
Upper bound of the date range.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
Must consist of full days.
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type `List(Date)`.
See Also
--------
date_range
datetime_ranges
Notes
-----
`interval` is created according to the following string language:
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"1w2d" # 1 week, 2 days
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
Examples
--------
>>> from datetime import date
>>> df = pl.DataFrame(
... {
... "start": [date(2022, 1, 1), date(2022, 1, 2)],
... "end": date(2022, 1, 3),
... }
... )
>>> with pl.Config(fmt_str_lengths=50):
... df.with_columns(date_range=pl.date_ranges("start", "end"))
shape: (2, 3)
┌────────────┬────────────┬──────────────────────────────────────┐
│ start ┆ end ┆ date_range │
│ --- ┆ --- ┆ --- │
│ date ┆ date ┆ list[date] │
╞════════════╪════════════╪══════════════════════════════════════╡
│ 2022-01-01 ┆ 2022-01-03 ┆ [2022-01-01, 2022-01-02, 2022-01-03] │
│ 2022-01-02 ┆ 2022-01-03 ┆ [2022-01-02, 2022-01-03] │
└────────────┴────────────┴──────────────────────────────────────┘
datetime = datetime_(year: 'int | IntoExpr', month: 'int | IntoExpr', day: 'int | IntoExpr', hour: 'int | IntoExpr | None' = None, minute: 'int | IntoExpr | None' = None, second: 'int | IntoExpr | None' = None, microsecond: 'int | IntoExpr | None' = None, *, time_unit: 'TimeUnit' = 'us', time_zone: 'str | None' = None, ambiguous: 'Ambiguous | Expr' = 'raise') -> 'Expr'
Create a Polars literal expression of type Datetime.
Parameters
----------
year
Column or literal.
month
Column or literal, ranging from 1-12.
day
Column or literal, ranging from 1-31.
hour
Column or literal, ranging from 0-23.
minute
Column or literal, ranging from 0-59.
second
Column or literal, ranging from 0-59.
microsecond
Column or literal, ranging from 0-999999.
time_unit : {'us', 'ms', 'ns'}
Time unit of the resulting expression.
time_zone
Time zone of the resulting expression.
ambiguous
Determine how to deal with ambiguous datetimes:
- `'raise'` (default): raise
- `'earliest'`: use the earliest datetime
- `'latest'`: use the latest datetime
- `'null'`: set to null
Returns
-------
Expr
Expression of data type :class:`Datetime`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "month": [1, 2, 3],
... "day": [4, 5, 6],
... "hour": [12, 13, 14],
... "minute": [15, 30, 45],
... }
... )
>>> df.with_columns(
... pl.datetime(
... 2024,
... pl.col("month"),
... pl.col("day"),
... pl.col("hour"),
... pl.col("minute"),
... time_zone="Australia/Sydney",
... )
... )
shape: (3, 5)
┌───────┬─────┬──────┬────────┬────────────────────────────────┐
│ month ┆ day ┆ hour ┆ minute ┆ datetime │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 ┆ datetime[μs, Australia/Sydney] │
╞═══════╪═════╪══════╪════════╪════════════════════════════════╡
│ 1 ┆ 4 ┆ 12 ┆ 15 ┆ 2024-01-04 12:15:00 AEDT │
│ 2 ┆ 5 ┆ 13 ┆ 30 ┆ 2024-02-05 13:30:00 AEDT │
│ 3 ┆ 6 ┆ 14 ┆ 45 ┆ 2024-03-06 14:45:00 AEDT │
└───────┴─────┴──────┴────────┴────────────────────────────────┘
We can also use `pl.datetime` for filtering:
>>> from datetime import datetime
>>> df = pl.DataFrame(
... {
... "start": [
... datetime(2024, 1, 1, 0, 0, 0),
... datetime(2024, 1, 1, 0, 0, 0),
... datetime(2024, 1, 1, 0, 0, 0),
... ],
... "end": [
... datetime(2024, 5, 1, 20, 15, 10),
... datetime(2024, 7, 1, 21, 25, 20),
... datetime(2024, 9, 1, 22, 35, 30),
... ],
... }
... )
>>> df.filter(pl.col("end") > pl.datetime(2024, 6, 1))
shape: (2, 2)
┌─────────────────────┬─────────────────────┐
│ start ┆ end │
│ --- ┆ --- │
│ datetime[μs] ┆ datetime[μs] │
╞═════════════════════╪═════════════════════╡
│ 2024-01-01 00:00:00 ┆ 2024-07-01 21:25:20 │
│ 2024-01-01 00:00:00 ┆ 2024-09-01 22:35:30 │
└─────────────────────┴─────────────────────┘
datetime_range(start: 'datetime | date | IntoExprColumn', end: 'datetime | date | IntoExprColumn', interval: 'str | timedelta' = '1d', *, closed: 'ClosedInterval' = 'both', time_unit: 'TimeUnit | None' = None, time_zone: 'str | None' = None, eager: 'bool' = False) -> 'Series | Expr'
Generate a datetime range.
Parameters
----------
start
Lower bound of the datetime range.
end
Upper bound of the datetime range.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
time_unit : {None, 'ns', 'us', 'ms'}
Time unit of the resulting `Datetime` data type.
time_zone
Time zone of the resulting `Datetime` data type.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type :class:`Datetime`.
See Also
--------
datetime_ranges
date_range
Notes
-----
`interval` is created according to the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
Examples
--------
Using Polars duration string to specify the interval:
>>> from datetime import datetime
>>> pl.datetime_range(
... datetime(2022, 1, 1), datetime(2022, 3, 1), "1mo", eager=True
... ).alias("datetime")
shape: (3,)
Series: 'datetime' [datetime[μs]]
[
2022-01-01 00:00:00
2022-02-01 00:00:00
2022-03-01 00:00:00
]
Using `timedelta` object to specify the interval:
>>> from datetime import date, timedelta
>>> pl.datetime_range(
... date(1985, 1, 1),
... date(1985, 1, 10),
... timedelta(days=1, hours=12),
... time_unit="ms",
... eager=True,
... ).alias("datetime")
shape: (7,)
Series: 'datetime' [datetime[ms]]
[
1985-01-01 00:00:00
1985-01-02 12:00:00
1985-01-04 00:00:00
1985-01-05 12:00:00
1985-01-07 00:00:00
1985-01-08 12:00:00
1985-01-10 00:00:00
]
Specifying a time zone:
>>> pl.datetime_range(
... datetime(2022, 1, 1),
... datetime(2022, 3, 1),
... "1mo",
... time_zone="America/New_York",
... eager=True,
... ).alias("datetime")
shape: (3,)
Series: 'datetime' [datetime[μs, America/New_York]]
[
2022-01-01 00:00:00 EST
2022-02-01 00:00:00 EST
2022-03-01 00:00:00 EST
]
Omit `eager=True` if you want to use `datetime_range` as an expression:
>>> df = pl.DataFrame(
... {
... "date": [
... date(2024, 1, 1),
... date(2024, 1, 2),
... date(2024, 1, 1),
... date(2024, 1, 3),
... ],
... "key": ["one", "one", "two", "two"],
... }
... )
>>> result = (
... df.group_by("key")
... .agg(pl.datetime_range(pl.col("date").min(), pl.col("date").max()))
... .sort("key")
... )
>>> with pl.Config(fmt_str_lengths=70):
... print(result)
shape: (2, 2)
┌─────┬─────────────────────────────────────────────────────────────────┐
│ key ┆ date │
│ --- ┆ --- │
│ str ┆ list[datetime[μs]] │
╞═════╪═════════════════════════════════════════════════════════════════╡
│ one ┆ [2024-01-01 00:00:00, 2024-01-02 00:00:00] │
│ two ┆ [2024-01-01 00:00:00, 2024-01-02 00:00:00, 2024-01-03 00:00:00] │
└─────┴─────────────────────────────────────────────────────────────────┘
datetime_ranges(start: 'datetime | date | IntoExprColumn', end: 'datetime | date | IntoExprColumn', interval: 'str | timedelta' = '1d', *, closed: 'ClosedInterval' = 'both', time_unit: 'TimeUnit | None' = None, time_zone: 'str | None' = None, eager: 'bool' = False) -> 'Series | Expr'
Create a column of datetime ranges.
Parameters
----------
start
Lower bound of the datetime range.
end
Upper bound of the datetime range.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
time_unit : {None, 'ns', 'us', 'ms'}
Time unit of the resulting `Datetime` data type.
time_zone
Time zone of the resulting `Datetime` data type.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Notes
-----
`interval` is created according to the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
Returns
-------
Expr or Series
Column of data type `List(Datetime)`.
See Also
--------
datetime_range
date_ranges
Examples
--------
>>> from datetime import datetime
>>> df = pl.DataFrame(
... {
... "start": [datetime(2022, 1, 1), datetime(2022, 1, 2)],
... "end": datetime(2022, 1, 3),
... }
... )
>>> with pl.Config(fmt_str_lengths=100):
... df.select(datetime_range=pl.datetime_ranges("start", "end"))
shape: (2, 1)
┌─────────────────────────────────────────────────────────────────┐
│ datetime_range │
│ --- │
│ list[datetime[μs]] │
╞═════════════════════════════════════════════════════════════════╡
│ [2022-01-01 00:00:00, 2022-01-02 00:00:00, 2022-01-03 00:00:00] │
│ [2022-01-02 00:00:00, 2022-01-03 00:00:00] │
└─────────────────────────────────────────────────────────────────┘
dtype_of(col_or_expr: 'str | Expr') -> 'pl.DataTypeExpr'
Get a lazily evaluated :class:`DataType` of a column or expression.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Examples
--------
>>> def inspect(expr: pl.Expr) -> pl.Expr:
... def print_and_return(s: pl.Series) -> pl.Series:
... print(s)
... return s
...
... return expr.map_batches(
... print_and_return,
... # Clarify that the expression returns the same datatype as the input
... # datatype.
... return_dtype=pl.dtype_of(expr),
... )
>>> df = pl.DataFrame(
... {
... "UserID": [1, 2, 3, 4, 5],
... "Name": ["Alice", "Bob", "Charlie", "Diana", "Ethan"],
... }
... )
>>> df.select(inspect(pl.col("Name")))
shape: (5,)
Series: 'Name' [str]
[
"Alice"
"Bob"
"Charlie"
"Diana"
"Ethan"
]
shape: (5, 1)
┌─────────┐
│ Name │
│ --- │
│ str │
╞═════════╡
│ Alice │
│ Bob │
│ Charlie │
│ Diana │
│ Ethan │
└─────────┘
duration(*, weeks: 'Expr | str | int | float | None' = None, days: 'Expr | str | int | float | None' = None, hours: 'Expr | str | int | float | None' = None, minutes: 'Expr | str | int | float | None' = None, seconds: 'Expr | str | int | float | None' = None, milliseconds: 'Expr | str | int | float | None' = None, microseconds: 'Expr | str | int | float | None' = None, nanoseconds: 'Expr | str | int | float | None' = None, time_unit: 'TimeUnit | None' = None) -> 'Expr'
Create polars `Duration` from distinct time components.
Parameters
----------
weeks
Number of weeks.
days
Number of days.
hours
Number of hours.
minutes
Number of minutes.
seconds
Number of seconds.
milliseconds
Number of milliseconds.
microseconds
Number of microseconds.
nanoseconds
Number of nanoseconds.
time_unit : {None, 'us', 'ms', 'ns'}
Time unit of the resulting expression. If set to `None` (default), the time
unit will be inferred from the other inputs: `'ns'` if `nanoseconds` was
specified, `'us'` otherwise.
Returns
-------
Expr
Expression of data type :class:`Duration`.
Notes
-----
A `duration` represents a fixed amount of time. For example,
`pl.duration(days=1)` means "exactly 24 hours". By contrast,
`Expr.dt.offset_by('1d')` means "1 calendar day", which could sometimes be
23 hours or 25 hours depending on Daylight Savings Time.
For non-fixed durations such as "calendar month" or "calendar day",
please use :meth:`polars.Expr.dt.offset_by` instead.
Examples
--------
>>> from datetime import datetime
>>> df = pl.DataFrame(
... {
... "dt": [datetime(2022, 1, 1), datetime(2022, 1, 2)],
... "add": [1, 2],
... }
... )
>>> df
shape: (2, 2)
┌─────────────────────┬─────┐
│ dt ┆ add │
│ --- ┆ --- │
│ datetime[μs] ┆ i64 │
╞═════════════════════╪═════╡
│ 2022-01-01 00:00:00 ┆ 1 │
│ 2022-01-02 00:00:00 ┆ 2 │
└─────────────────────┴─────┘
>>> with pl.Config(tbl_width_chars=120):
... df.select(
... (pl.col("dt") + pl.duration(weeks="add")).alias("add_weeks"),
... (pl.col("dt") + pl.duration(days="add")).alias("add_days"),
... (pl.col("dt") + pl.duration(seconds="add")).alias("add_seconds"),
... (pl.col("dt") + pl.duration(milliseconds="add")).alias("add_millis"),
... (pl.col("dt") + pl.duration(hours="add")).alias("add_hours"),
... )
shape: (2, 5)
┌─────────────────────┬─────────────────────┬─────────────────────┬─────────────────────────┬─────────────────────┐
│ add_weeks ┆ add_days ┆ add_seconds ┆ add_millis ┆ add_hours │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] │
╞═════════════════════╪═════════════════════╪═════════════════════╪═════════════════════════╪═════════════════════╡
│ 2022-01-08 00:00:00 ┆ 2022-01-02 00:00:00 ┆ 2022-01-01 00:00:01 ┆ 2022-01-01 00:00:00.001 ┆ 2022-01-01 01:00:00 │
│ 2022-01-16 00:00:00 ┆ 2022-01-04 00:00:00 ┆ 2022-01-02 00:00:02 ┆ 2022-01-02 00:00:00.002 ┆ 2022-01-02 02:00:00 │
└─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────────┴─────────────────────┘
If you need to add non-fixed durations, you should use :meth:`polars.Expr.dt.offset_by` instead:
>>> with pl.Config(tbl_width_chars=120):
... df.select(
... add_calendar_days=pl.col("dt").dt.offset_by(
... pl.format("{}d", pl.col("add"))
... ),
... add_calendar_months=pl.col("dt").dt.offset_by(
... pl.format("{}mo", pl.col("add"))
... ),
... add_calendar_years=pl.col("dt").dt.offset_by(
... pl.format("{}y", pl.col("add"))
... ),
... )
shape: (2, 3)
┌─────────────────────┬─────────────────────┬─────────────────────┐
│ add_calendar_days ┆ add_calendar_months ┆ add_calendar_years │
│ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] │
╞═════════════════════╪═════════════════════╪═════════════════════╡
│ 2022-01-02 00:00:00 ┆ 2022-02-01 00:00:00 ┆ 2023-01-01 00:00:00 │
│ 2022-01-04 00:00:00 ┆ 2022-03-02 00:00:00 ┆ 2024-01-02 00:00:00 │
└─────────────────────┴─────────────────────┴─────────────────────┘
element() -> 'Expr'
Alias for an element being evaluated in an `eval` or `filter` expression.
Examples
--------
A horizontal rank computation by taking the elements of a list
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... }
... )
>>> df.with_columns(
... pl.concat_list(["a", "b"]).list.eval(pl.element().rank()).alias("rank")
... )
shape: (3, 3)
┌─────┬─────┬────────────┐
│ a ┆ b ┆ rank │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[f64] │
╞═════╪═════╪════════════╡
│ 1 ┆ 4 ┆ [1.0, 2.0] │
│ 8 ┆ 5 ┆ [2.0, 1.0] │
│ 3 ┆ 2 ┆ [2.0, 1.0] │
└─────┴─────┴────────────┘
A mathematical operation on array elements
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... }
... )
>>> df.with_columns(
... pl.concat_list(["a", "b"]).list.eval(pl.element() * 2).alias("a_b_doubled")
... )
shape: (3, 3)
┌─────┬─────┬─────────────┐
│ a ┆ b ┆ a_b_doubled │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[i64] │
╞═════╪═════╪═════════════╡
│ 1 ┆ 4 ┆ [2, 8] │
│ 8 ┆ 5 ┆ [16, 10] │
│ 3 ┆ 2 ┆ [6, 4] │
└─────┴─────┴─────────────┘
A filter operation on list elements
>>> import polars as pl
>>> df = pl.DataFrame({"a": [1, 8, 3], "b": [4, 5, 2]})
>>> df.with_columns(
... evens=pl.concat_list("a", "b").list.filter(pl.element() % 2 == 0)
... )
shape: (3, 3)
┌─────┬─────┬───────────┐
│ a ┆ b ┆ evens │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[i64] │
╞═════╪═════╪═══════════╡
│ 1 ┆ 4 ┆ [4] │
│ 8 ┆ 5 ┆ [8] │
│ 3 ┆ 2 ┆ [2] │
└─────┴─────┴───────────┘
escape_regex(s: 'str') -> 'str'
Escapes string regex meta characters.
Parameters
----------
s
The string whose meta characters will be escaped.
exclude(columns: 'str | PolarsDataType | Collection[str] | Collection[PolarsDataType]', *more_columns: 'str | PolarsDataType') -> 'Expr'
Represent all columns except for the given columns.
Syntactic sugar for `pl.all().exclude(columns)`.
Parameters
----------
columns
The name or datatype of the column(s) to exclude. Accepts regular expression
input. Regular expressions should start with `^` and end with `$`.
*more_columns
Additional names or datatypes of columns to exclude, specified as positional
arguments.
Examples
--------
Exclude by column name(s):
>>> df = pl.DataFrame(
... {
... "aa": [1, 2, 3],
... "ba": ["a", "b", None],
... "cc": [None, 2.5, 1.5],
... }
... )
>>> df.select(pl.exclude("ba"))
shape: (3, 2)
┌─────┬──────┐
│ aa ┆ cc │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪══════╡
│ 1 ┆ null │
│ 2 ┆ 2.5 │
│ 3 ┆ 1.5 │
└─────┴──────┘
Exclude by regex, e.g. removing all columns whose names end with the letter "a":
>>> df.select(pl.exclude("^.*a$"))
shape: (3, 1)
┌──────┐
│ cc │
│ --- │
│ f64 │
╞══════╡
│ null │
│ 2.5 │
│ 1.5 │
└──────┘
Exclude by dtype(s), e.g. removing all columns of type Int64 or Float64:
>>> df.select(pl.exclude([pl.Int64, pl.Float64]))
shape: (3, 1)
┌──────┐
│ ba │
│ --- │
│ str │
╞══════╡
│ a │
│ b │
│ null │
└──────┘
explain_all(lazy_frames: 'Iterable[LazyFrame]', *, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'str'
Explain multiple LazyFrames as if passed to `collect_all`.
Common Subplan Elimination is applied on the combined plan, meaning
that diverging queries will run only once.
Parameters
----------
lazy_frames
A list of LazyFrames to collect.
optimizations
The optimization passes done during query optimization.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Returns
-------
Explained plan.
field(name: 'str | list[str]') -> 'Expr'
Select a field in the current `struct.with_fields` scope.
Parameters
----------
name
Name of the field(s) to select.
Examples
--------
>>> df = pl.DataFrame({"a": [{"x": 5, "y": 2}, {"x": 3, "y": 4}]})
>>> df.select(pl.col("a").struct.with_fields(pl.field("x") ** 2))
shape: (2, 1)
┌───────────┐
│ a │
│ --- │
│ struct[2] │
╞═══════════╡
│ {25,2} │
│ {9,4} │
└───────────┘
first(*columns: 'str') -> 'Expr'
Get the first column or value.
This function has different behavior depending on the presence of `columns`
values. If none given (the default), returns an expression that takes the first
column of the context; otherwise, takes the first value of the given column(s).
Parameters
----------
*columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "baz"],
... }
... )
Return the first column:
>>> df.select(pl.first())
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 8 │
│ 3 │
└─────┘
Return the first value for the given column(s):
>>> df.select(pl.first("b"))
shape: (1, 1)
┌─────┐
│ b │
│ --- │
│ i64 │
╞═════╡
│ 4 │
└─────┘
>>> df.select(pl.first("a", "c"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ c │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1 ┆ foo │
└─────┴─────┘
fold(acc: 'IntoExpr', function: 'Callable[[Series, Series], Series]', exprs: 'Sequence[Expr | str] | Expr', *, returns_scalar: 'bool' = False, return_dtype: 'pl.DataTypeExpr | PolarsDataType | None' = None) -> 'Expr'
Accumulate over multiple columns horizontally/ row wise with a left fold.
Parameters
----------
acc
Accumulator Expression. This is the value that will be initialized when the fold
starts. For a sum this could for instance be lit(0).
function
Function to apply over the accumulator and the value.
Fn(acc, value) -> new_value
exprs
Expressions to aggregate over. May also be a wildcard expression.
returns_scalar
Whether or not `function` applied returns a scalar. This must be set correctly
by the user.
return_dtype
Output datatype.
If not set, the dtype will be inferred based on the dtype
of the accumulator.
Notes
-----
If you simply want the first encountered expression as accumulator,
consider using `reduce`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [3, 4, 5],
... "c": [5, 6, 7],
... }
... )
>>> df
shape: (3, 3)
┌─────┬─────┬─────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 5 │
│ 2 ┆ 4 ┆ 6 │
│ 3 ┆ 5 ┆ 7 │
└─────┴─────┴─────┘
Horizontally sum over all columns and add 1.
>>> df.select(
... pl.fold(
... acc=pl.lit(1), function=lambda acc, x: acc + x, exprs=pl.col("*")
... ).alias("sum"),
... )
shape: (3, 1)
┌─────┐
│ sum │
│ --- │
│ i32 │
╞═════╡
│ 10 │
│ 13 │
│ 16 │
└─────┘
You can also apply a condition/predicate on all columns:
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [0, 1, 2],
... }
... )
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 0 │
│ 2 ┆ 1 │
│ 3 ┆ 2 │
└─────┴─────┘
>>> df.filter(
... pl.fold(
... acc=pl.lit(True),
... function=lambda acc, x: acc & x,
... exprs=pl.col("*") > 1,
... )
... )
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 3 ┆ 2 │
└─────┴─────┘
format(f_string: 'str', *args: 'IntoExpr') -> 'Expr'
Format expressions as a string.
Parameters
----------
f_string
A string with placeholders of the form `{}`, `{index}` or `{name}`.
A placeholder can be empty, in which case it consumes the next argument
to pl.format. It can also be an index in which case it addresses the
nth argument to pl.format. Finally a placeholder can also be an ASCII
identifier, in which case it directly refers to a column name.
If you wish to use the characters `{}` literally you must escape them
by doubling, e.g. `'{{"json": 42}}'`.
args
Expression(s) that fill the placeholders
Notes
-----
If any input expression evaluates to null for a row, the output of
``pl.format`` is null for that row.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": ["a", "b", "c"],
... "b": [1, 2, 3],
... }
... )
>>> df.select(
... [
... pl.format("foo_{}_bar_{}", pl.col("a"), "b").alias("fmt"),
... ]
... )
shape: (3, 1)
┌─────────────┐
│ fmt │
│ --- │
│ str │
╞═════════════╡
│ foo_a_bar_1 │
│ foo_b_bar_2 │
│ foo_c_bar_3 │
└─────────────┘
>>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, None, 6]})
>>> df.select(pl.format("{}_{}", "a", "b").alias("fmt"))
shape: (3, 1)
┌──────┐
│ fmt │
│ --- │
│ str │
╞══════╡
│ 1_4 │
│ null │
│ 3_6 │
└──────┘
from_epoch(column: 'str | Expr | Series | Sequence[int | float]', time_unit: 'EpochTimeUnit' = 's') -> 'Expr | Series'
Utility function that parses an epoch timestamp (or Unix time) to Polars Date(time).
Depending on the `time_unit` provided, this function will return a different dtype:
- time_unit="d" returns pl.Date
- time_unit="ns" returns pl.Datetime["ns"]
- otherwise returns pl.Datetime["us"] (Polars' default time unit)
Parameters
----------
column
Series or Expression to parse to Datetime.
time_unit
The unit of time of the timesteps since epoch time.
Examples
--------
Convert from integer seconds:
>>> df = pl.LazyFrame({"ts": [1666683077, 1666683099]})
>>> df.select(pl.from_epoch(pl.col("ts"), time_unit="s")).collect()
shape: (2, 1)
┌─────────────────────┐
│ ts │
│ --- │
│ datetime[μs] │
╞═════════════════════╡
│ 2022-10-25 07:31:17 │
│ 2022-10-25 07:31:39 │
└─────────────────────┘
Convert from fractional seconds:
>>> df = pl.LazyFrame({"ts": [-609066.723456, 1066445333.8888, 3405071999.987654]})
>>> df.select(pl.from_epoch(pl.col("ts"), time_unit="s")).collect()
shape: (3, 1)
┌────────────────────────────┐
│ ts │
│ --- │
│ datetime[μs] │
╞════════════════════════════╡
│ 1969-12-24 22:48:53.276544 │
│ 2003-10-18 02:48:53.888800 │
│ 2077-11-25 13:19:59.987654 │
└────────────────────────────┘
The function can also be used in an eager context by passing a Series
or sequence of int/float values.
>>> s = pl.Series([12345, 12346])
>>> pl.from_epoch(s, time_unit="d")
shape: (2,)
Series: '' [date]
[
2003-10-20
2003-10-21
]
groups(column: 'str') -> 'Expr'
Syntactic sugar for `pl.col("foo").agg_groups()`.
.. deprecated:: 1.35
Use `df.with_row_index().group_by(...).agg(pl.col('index'))` instead.
This method will be removed in Polars 2.0.
head(column: 'str', n: 'int' = 10) -> 'Expr'
Get the first `n` rows.
This function is syntactic sugar for `pl.col(column).head(n)`.
Parameters
----------
column
Column name.
n
Number of rows to return.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.head("a"))
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 8 │
│ 3 │
└─────┘
>>> df.select(pl.head("a", 2))
shape: (2, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 8 │
└─────┘
implode(*columns: 'str', maintain_order: 'bool' = True) -> 'Expr'
Aggregate all column values into a list.
This function is syntactic sugar for `pl.col(name).implode()`.
Parameters
----------
*columns
One or more column names.
maintain_order
Whether to preserve the order of elements in the list. Setting this
to `False` can improve performance, especially within `group_by`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [9, 8, 7],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.implode("a"))
shape: (1, 1)
┌───────────┐
│ a │
│ --- │
│ list[i64] │
╞═══════════╡
│ [1, 2, 3] │
└───────────┘
>>> df.select(pl.implode("b", "c"))
shape: (1, 2)
┌───────────┬───────────────────────┐
│ b ┆ c │
│ --- ┆ --- │
│ list[i64] ┆ list[str] │
╞═══════════╪═══════════════════════╡
│ [9, 8, 7] ┆ ["foo", "bar", "foo"] │
└───────────┴───────────────────────┘
int_range(start: 'int | IntoExprColumn' = 0, end: 'int | IntoExprColumn | None' = None, step: 'int' = 1, *, dtype: 'PolarsIntegerType | DataTypeExpr' = Int64, eager: 'bool' = False) -> 'Expr | Series'
Generate a range of integers.
Parameters
----------
start
Start of the range (inclusive). Defaults to 0.
end
End of the range (exclusive). If set to `None` (default),
the value of `start` is used and `start` is set to `0`.
step
Step size of the range.
dtype
Data type of the range.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of integer data type `dtype`.
See Also
--------
int_ranges : Generate a range of integers for each row of the input columns.
Examples
--------
>>> pl.int_range(0, 3, eager=True)
shape: (3,)
Series: 'literal' [i64]
[
0
1
2
]
`end` can be omitted for a shorter syntax.
>>> pl.int_range(3, eager=True)
shape: (3,)
Series: 'literal' [i64]
[
0
1
2
]
Generate an index column by using `int_range` in conjunction with :func:`len`.
>>> df = pl.DataFrame({"a": [1, 3, 5], "b": [2, 4, 6]})
>>> df.select(
... pl.int_range(pl.len(), dtype=pl.UInt32).alias("index"),
... pl.all(),
... )
shape: (3, 3)
┌───────┬─────┬─────┐
│ index ┆ a ┆ b │
│ --- ┆ --- ┆ --- │
│ u32 ┆ i64 ┆ i64 │
╞═══════╪═════╪═════╡
│ 0 ┆ 1 ┆ 2 │
│ 1 ┆ 3 ┆ 4 │
│ 2 ┆ 5 ┆ 6 │
└───────┴─────┴─────┘
int_ranges(start: 'int | IntoExprColumn' = 0, end: 'int | IntoExprColumn | None' = None, step: 'int | IntoExprColumn' = 1, *, dtype: 'PolarsIntegerType | DataTypeExpr' = Int64, eager: 'bool' = False) -> 'Expr | Series'
Generate a range of integers for each row of the input columns.
Parameters
----------
start
Start of the range (inclusive). Defaults to 0.
end
End of the range (exclusive). If set to `None` (default),
the value of `start` is used and `start` is set to `0`.
step
Step size of the range.
dtype
Integer data type of the ranges. Defaults to `Int64`.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type `List(dtype)`.
See Also
--------
int_range : Generate a single range of integers.
Examples
--------
>>> df = pl.DataFrame({"start": [1, -1], "end": [3, 2]})
>>> df.with_columns(int_range=pl.int_ranges("start", "end"))
shape: (2, 3)
┌───────┬─────┬────────────┐
│ start ┆ end ┆ int_range │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[i64] │
╞═══════╪═════╪════════════╡
│ 1 ┆ 3 ┆ [1, 2] │
│ -1 ┆ 2 ┆ [-1, 0, 1] │
└───────┴─────┴────────────┘
`end` can be omitted for a shorter syntax.
>>> df.select("end", int_range=pl.int_ranges("end"))
shape: (2, 2)
┌─────┬───────────┐
│ end ┆ int_range │
│ --- ┆ --- │
│ i64 ┆ list[i64] │
╞═════╪═══════════╡
│ 3 ┆ [0, 1, 2] │
│ 2 ┆ [0, 1] │
└─────┴───────────┘
last(*columns: 'str') -> 'Expr'
Get the last column or value.
This function has different behavior depending on the presence of `columns`
values. If none given (the default), returns an expression that takes the last
column of the context; otherwise, takes the last value of the given column(s).
Parameters
----------
*columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "baz"],
... }
... )
Return the last column:
>>> df.select(pl.last())
shape: (3, 1)
┌─────┐
│ c │
│ --- │
│ str │
╞═════╡
│ foo │
│ bar │
│ baz │
└─────┘
Return the last value for the given column(s):
>>> df.select(pl.last("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 3 │
└─────┘
>>> df.select(pl.last("b", "c"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 2 ┆ baz │
└─────┴─────┘
len() -> 'Expr'
Return the number of rows in the context.
This is similar to `COUNT(*)` in SQL.
Returns
-------
Expr
Expression of data type :class:`UInt32`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, None],
... "b": [3, None, None],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.len())
shape: (1, 1)
┌─────┐
│ len │
│ --- │
│ u32 │
╞═════╡
│ 3 │
└─────┘
Generate an index column by using `len` in conjunction with :func:`int_range`.
>>> df.select(
... pl.int_range(pl.len(), dtype=pl.UInt32).alias("index"),
... pl.all(),
... )
shape: (3, 4)
┌───────┬──────┬──────┬─────┐
│ index ┆ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- ┆ --- │
│ u32 ┆ i64 ┆ i64 ┆ str │
╞═══════╪══════╪══════╪═════╡
│ 0 ┆ 1 ┆ 3 ┆ foo │
│ 1 ┆ 2 ┆ null ┆ bar │
│ 2 ┆ null ┆ null ┆ foo │
└───────┴──────┴──────┴─────┘
linear_space(start: 'NumericLiteral | TemporalLiteral | IntoExpr', end: 'NumericLiteral | TemporalLiteral | IntoExpr', num_samples: 'int | IntoExpr', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Expr | Series'
Create sequence of evenly-spaced points.
Parameters
----------
start
Lower bound of the range.
end
Upper bound of the range.
num_samples
Number of samples in the output sequence.
closed : {'both', 'left', 'right', 'none'}
Define which sides of the interval are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
.. warning::
This functionality is experimental. It may be changed at any point without it
being considered a breaking change.
Notes
-----
`linear_space` works with numeric and temporal dtypes. When the `start` and `end`
parameters are `Date` dtypes, the output sequence consists of equally-spaced
`Datetime` elements with millisecond precision.
Returns
-------
Expr or Series
Column of data type `:class:Time`.
Examples
--------
>>> pl.linear_space(start=0, end=1, num_samples=3, eager=True)
shape: (3,)
Series: 'literal' [f64]
[
0.0
0.5
1.0
]
>>> pl.linear_space(start=0, end=1, num_samples=3, closed="left", eager=True)
shape: (3,)
Series: 'literal' [f64]
[
0.0
0.333333
0.666667
]
>>> pl.linear_space(start=0, end=1, num_samples=3, closed="right", eager=True)
shape: (3,)
Series: 'literal' [f64]
[
0.333333
0.666667
1.0
]
>>> pl.linear_space(start=0, end=1, num_samples=3, closed="none", eager=True)
shape: (3,)
Series: 'literal' [f64]
[
0.25
0.5
0.75
]
>>> from datetime import time
>>> pl.linear_space(
... start=time(hour=1), end=time(hour=12), num_samples=3, eager=True
... )
shape: (3,)
Series: 'literal' [time]
[
01:00:00
06:30:00
12:00:00
]
`Date` endpoints generate a sequence of `Datetime` values:
>>> from datetime import date
>>> pl.linear_space(
... start=date(2025, 1, 1),
... end=date(2025, 2, 1),
... num_samples=3,
... closed="right",
... eager=True,
... )
shape: (3,)
Series: 'literal' [datetime[μs]]
[
2025-01-11 08:00:00
2025-01-21 16:00:00
2025-02-01 00:00:00
]
When `eager=False` (default), an expression is produced. You can generate a sequence
using the length of the dataframe:
>>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5]})
>>> df.with_columns(pl.linear_space(0, 1, pl.len()).alias("ls"))
shape: (5, 2)
┌─────┬──────┐
│ a ┆ ls │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪══════╡
│ 1 ┆ 0.0 │
│ 2 ┆ 0.25 │
│ 3 ┆ 0.5 │
│ 4 ┆ 0.75 │
│ 5 ┆ 1.0 │
└─────┴──────┘
linear_spaces(start: 'NumericLiteral | TemporalLiteral | IntoExprColumn', end: 'NumericLiteral | TemporalLiteral | IntoExprColumn', num_samples: 'int | IntoExprColumn', *, closed: 'ClosedInterval' = 'both', as_array: 'bool' = False, eager: 'bool' = False) -> 'Expr | Series'
Generate a sequence of evenly-spaced values for each row between `start` and `end`.
The number of values in each sequence is determined by `num_samples`.
Parameters
----------
start
Lower bound of the range.
end
Upper bound of the range.
num_samples
Number of samples in the output sequence.
closed : {'both', 'left', 'right', 'none'}
Define which sides of the interval are closed (inclusive).
as_array
Return result as a fixed-length `Array`. `num_samples` must be a constant.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
.. warning::
This functionality is experimental. It may be changed at any point without it
being considered a breaking change.
Returns
-------
Expr or Series
Column of data type `List(dtype)`.
See Also
--------
linear_space : Generate a single sequence of linearly-spaced values.
Examples
--------
>>> df = pl.DataFrame({"start": [1, -1], "end": [3, 2], "num_samples": [4, 5]})
>>> df.with_columns(ls=pl.linear_spaces("start", "end", "num_samples"))
shape: (2, 4)
┌───────┬─────┬─────────────┬────────────────────────┐
│ start ┆ end ┆ num_samples ┆ ls │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ list[f64] │
╞═══════╪═════╪═════════════╪════════════════════════╡
│ 1 ┆ 3 ┆ 4 ┆ [1.0, 1.666667, … 3.0] │
│ -1 ┆ 2 ┆ 5 ┆ [-1.0, -0.25, … 2.0] │
└───────┴─────┴─────────────┴────────────────────────┘
>>> df.with_columns(ls=pl.linear_spaces("start", "end", 3, as_array=True))
shape: (2, 4)
┌───────┬─────┬─────────────┬──────────────────┐
│ start ┆ end ┆ num_samples ┆ ls │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ array[f64, 3] │
╞═══════╪═════╪═════════════╪══════════════════╡
│ 1 ┆ 3 ┆ 4 ┆ [1.0, 2.0, 3.0] │
│ -1 ┆ 2 ┆ 5 ┆ [-1.0, 0.5, 2.0] │
└───────┴─────┴─────────────┴──────────────────┘
list(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr') -> 'Expr'
Collect columns into a list column, treating each expression's value as one element.
Unlike :func:`concat_list`, list-typed inputs are not extended — each input's value
becomes a single element of the output list. This means ``List(T)`` inputs produce
``List(List(T))`` output.
Parameters
----------
exprs
Columns to collect into a list. Accepts expression input. Strings are parsed
as column names, other non-expression inputs are parsed as literals.
*more_exprs
Additional columns, specified as positional arguments.
Examples
--------
Wrap scalar columns into a list (same as ``concat_list`` for scalars):
>>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
>>> df.with_columns(a_b=pl.list("a", "b"))
shape: (3, 3)
┌─────┬─────┬───────────┐
│ a ┆ b ┆ a_b │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[i64] │
╞═════╪═════╪═══════════╡
│ 1 ┆ 4 ┆ [1, 4] │
│ 2 ┆ 5 ┆ [2, 5] │
│ 3 ┆ 6 ┆ [3, 6] │
└─────┴─────┴───────────┘
Collect list columns into a list-of-lists (unlike ``concat_list``, which extends):
>>> df = pl.DataFrame({"a": [[1, 2], [3], [4, 5]], "b": [[6], [7, 8], [9]]})
>>> df.with_columns(a_b=pl.list("a", "b"))
shape: (3, 3)
┌───────────┬───────────┬─────────────────┐
│ a ┆ b ┆ a_b │
│ --- ┆ --- ┆ --- │
│ list[i64] ┆ list[i64] ┆ list[list[i64]] │
╞═══════════╪═══════════╪═════════════════╡
│ [1, 2] ┆ [6] ┆ [[1, 2], [6]] │
│ [3] ┆ [7, 8] ┆ [[3], [7, 8]] │
│ [4, 5] ┆ [9] ┆ [[4, 5], [9]] │
└───────────┴───────────┴─────────────────┘
lit(value: 'Any', dtype: 'PolarsDataType | DataTypeExpr | None' = None, *, allow_object: 'bool' = False) -> 'Expr'
Return an expression representing a literal value.
Parameters
----------
value
Value that should be used as a `literal`.
dtype
The data type of the resulting expression.
If set to `None` (default), the data type is inferred from the `value` input.
allow_object
If type is unknown use an 'object' type.
By default, we will raise a `ValueException`
if the type is unknown.
Notes
-----
Expected datatypes:
- `pl.lit([])` -> empty List<Null>
- `pl.lit([1, 2, 3])` -> List<i64>
- `pl.lit(pl.Series([]))`-> empty Series Null
- `pl.lit(pl.Series([1, 2, 3]))` -> Series Int64
- `pl.lit(None)` -> Null
Examples
--------
Literal scalar values:
>>> pl.lit(1) # doctest: +IGNORE_RESULT
>>> pl.lit(5.5) # doctest: +IGNORE_RESULT
>>> pl.lit(None) # doctest: +IGNORE_RESULT
>>> pl.lit("foo_bar") # doctest: +IGNORE_RESULT
>>> pl.lit(date(2021, 1, 20)) # doctest: +IGNORE_RESULT
>>> pl.lit(datetime(2023, 3, 31, 10, 30, 45)) # doctest: +IGNORE_RESULT
Literal list/Series data (1D):
>>> pl.lit([1, 2, 3]) # doctest: +SKIP
>>> pl.lit(pl.Series("x", [1, 2, 3])) # doctest: +IGNORE_RESULT
Literal list/Series data (2D):
>>> pl.lit([[1, 2], [3, 4]]) # doctest: +SKIP
>>> pl.lit(pl.Series("y", [[1, 2], [3, 4]])) # doctest: +IGNORE_RESULT
map_batches(exprs: 'Sequence[str | Expr]', function: 'Callable[[Sequence[Series]], Series | Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, is_elementwise: 'bool' = False, returns_scalar: 'bool' = False) -> 'Expr'
Map a custom function over multiple columns/expressions.
Produces a single Series result.
.. warning::
This method is much slower than the native expressions API.
Only use it if you cannot implement your logic otherwise.
Parameters
----------
exprs
Expression(s) representing the input Series to the function.
function
Function to apply over the input.
return_dtype
Datatype of the output Series.
It is recommended to set this whenever possible. If this is `None`, it tries
to infer the datatype by calling the function with dummy data and looking at
the output.
is_elementwise
Set to true if the operations is elementwise for better performance
and optimization.
An elementwise operations has unit or equal length for all inputs
and can be ran sequentially on slices without results being affected.
returns_scalar
If the function returns a scalar, by default it will be wrapped in
a list in the output, since the assumption is that the function
always returns something Series-like. If you want to keep the
result as a scalar, set this argument to True.
Notes
-----
A UDF passed to `map_batches` must be pure, meaning that it cannot modify
or depend on state other than its arguments. We may call the function
with arbitrary input data.
Returns
-------
Expr
Expression with the data type given by `return_dtype`.
Examples
--------
>>> def test_func(a, b, c):
... return a + b + c
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3, 4],
... "b": [4, 5, 6, 7],
... }
... )
>>>
>>> df.with_columns(
... (
... pl.struct(["a", "b"]).map_batches(
... lambda x: test_func(x.struct.field("a"), x.struct.field("b"), 1)
... )
... ).alias("a+b+c")
... )
shape: (4, 3)
┌─────┬─────┬───────┐
│ a ┆ b ┆ a+b+c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═══════╡
│ 1 ┆ 4 ┆ 6 │
│ 2 ┆ 5 ┆ 8 │
│ 3 ┆ 6 ┆ 10 │
│ 4 ┆ 7 ┆ 12 │
└─────┴─────┴───────┘
map_groups(exprs: 'Sequence[str | Expr]', function: 'Callable[[Sequence[Series]], Series | Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, is_elementwise: 'bool' = False, returns_scalar: 'bool' = False) -> 'Expr'
Apply a custom/user-defined function (UDF) in a GroupBy context.
.. warning::
This method is much slower than the native expressions API.
Only use it if you cannot implement your logic otherwise.
Parameters
----------
exprs
Expression(s) representing the input Series to the function.
function
Function to apply over the input; should be of type Callable[[Series], Series].
return_dtype
Datatype of the output Series.
It is recommended to set this whenever possible. If this is `None`, it tries
to infer the datatype by calling the function with dummy data and looking at
the output.
is_elementwise
Set to true if the operations is elementwise for better performance
and optimization.
An elementwise operations has unit or equal length for all inputs
and can be ran sequentially on slices without results being affected.
returns_scalar
If the function returns a single scalar as output.
Notes
-----
A UDF passed to `map_batches` must be pure, meaning that it cannot modify
or depend on state other than its arguments. Polars may call the function
with arbitrary input data.
Returns
-------
Expr
Expression with the data type given by `return_dtype`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "group": [1, 1, 2],
... "a": [1, 3, 3],
... "b": [5, 6, 7],
... }
... )
>>> df
shape: (3, 3)
┌───────┬─────┬─────┐
│ group ┆ a ┆ b │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═══════╪═════╪═════╡
│ 1 ┆ 1 ┆ 5 │
│ 1 ┆ 3 ┆ 6 │
│ 2 ┆ 3 ┆ 7 │
└───────┴─────┴─────┘
>>> (
... df.group_by("group").agg(
... pl.map_groups(
... exprs=["a", "b"],
... function=lambda list_of_series: (
... list_of_series[0] / list_of_series[0].sum() + list_of_series[1]
... ),
... return_dtype=pl.Float64,
... ).alias("my_custom_aggregation")
... )
... ).sort("group")
shape: (2, 2)
┌───────┬───────────────────────┐
│ group ┆ my_custom_aggregation │
│ --- ┆ --- │
│ i64 ┆ list[f64] │
╞═══════╪═══════════════════════╡
│ 1 ┆ [5.25, 6.75] │
│ 2 ┆ [8.0] │
└───────┴───────────────────────┘
The output for group `1` can be understood as follows:
- group `1` contains Series `'a': [1, 3]` and `'b': [5, 6]`
- applying the function to those lists of Series, one gets the output
`[1 / 4 + 5, 3 / 4 + 6]`, i.e. `[5.25, 6.75]`
max(*names: 'str') -> 'Expr'
Get the maximum value.
Syntactic sugar for `col(names).max()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
See Also
--------
max_horizontal
Examples
--------
Get the maximum value of a column.
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.max("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 8 │
└─────┘
Get the maximum value of multiple columns.
>>> df.select(pl.max("^a|b$"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 8 ┆ 5 │
└─────┴─────┘
>>> df.select(pl.max("a", "b"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 8 ┆ 5 │
└─────┴─────┘
max_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Get the maximum value horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(max=pl.max_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬─────┐
│ a ┆ b ┆ c ┆ max │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ i64 │
╞═════╪══════╪═════╪═════╡
│ 1 ┆ 4 ┆ x ┆ 4 │
│ 8 ┆ 5 ┆ y ┆ 8 │
│ 3 ┆ null ┆ z ┆ 3 │
└─────┴──────┴─────┴─────┘
mean(*columns: 'str') -> 'Expr'
Get the mean value.
This function is syntactic sugar for `pl.col(columns).mean()`.
Parameters
----------
*columns
One or more column names.
See Also
--------
mean_horizontal
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.mean("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ f64 │
╞═════╡
│ 4.0 │
└─────┘
>>> df.select(pl.mean("a", "b"))
shape: (1, 2)
┌─────┬──────────┐
│ a ┆ b │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞═════╪══════════╡
│ 4.0 ┆ 3.666667 │
└─────┴──────────┘
mean_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]', ignore_nulls: 'bool' = True) -> 'Expr'
Compute the mean of all values horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
ignore_nulls
Ignore null values (default).
If set to `False`, any null value in the input will lead to a null output.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(mean=pl.mean_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬──────┐
│ a ┆ b ┆ c ┆ mean │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ f64 │
╞═════╪══════╪═════╪══════╡
│ 1 ┆ 4 ┆ x ┆ 2.5 │
│ 8 ┆ 5 ┆ y ┆ 6.5 │
│ 3 ┆ null ┆ z ┆ 3.0 │
└─────┴──────┴─────┴──────┘
median(*columns: 'str') -> 'Expr'
Get the median value.
This function is syntactic sugar for `pl.col(columns).median()`.
Parameters
----------
columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.median("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ f64 │
╞═════╡
│ 3.0 │
└─────┘
>>> df.select(pl.median("a", "b"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞═════╪═════╡
│ 3.0 ┆ 4.0 │
└─────┴─────┘
merge_sorted(items: 'Iterable[PolarsType]', key: 'str | Sequence[str]', *, maintain_order: 'bool' = False) -> 'PolarsType'
Merge multiple sorted DataFrames or LazyFrames by the sorted key.
The output of this operation will also be sorted.
It is the callers responsibility that the frames
are sorted in ascending order by that key(s) otherwise
the output will not make sense.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
items
DataFrames or LazyFrames to merge.
key
Key column(s) that the frames are sorted by. A single column name or a
sequence of column names can be passed. When multiple keys are given the
frames are merged as if sorted by those keys in order.
maintain_order
If ``True``, the output is guaranteed to have left-biased ordering
for equal keys: rows from the left frame appear before rows from
the right frame when their keys are equal.
Examples
--------
>>> df0 = pl.DataFrame(
... {"name": ["steve", "elise", "bob"], "age": [42, 44, 18]}
... ).sort("age")
>>> df1 = pl.DataFrame(
... {"name": ["anna", "megan", "steve", "thomas"], "age": [21, 33, 17, 20]}
... ).sort("age")
>>> df2 = pl.DataFrame({"name": ["ida", "maya"], "age": [37, 27]}).sort("age")
>>> pl.merge_sorted([df0, df1, df2], key="age")
shape: (9, 2)
┌────────┬─────┐
│ name ┆ age │
│ --- ┆ --- │
│ str ┆ i64 │
╞════════╪═════╡
│ steve ┆ 17 │
│ bob ┆ 18 │
│ thomas ┆ 20 │
│ anna ┆ 21 │
│ maya ┆ 27 │
│ megan ┆ 33 │
│ ida ┆ 37 │
│ steve ┆ 42 │
│ elise ┆ 44 │
└────────┴─────┘
Notes
-----
Unless ``maintain_order=True``, no guarantee is given over the output
row order when the key is equal between dataframes.
The key(s) must be sorted in ascending order.
min(*names: 'str') -> 'Expr'
Get the minimum value.
Syntactic sugar for `col(names).min()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
See Also
--------
min_horizontal
Examples
--------
Get the minimum value of a column.
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.min("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
└─────┘
Get the minimum value of multiple columns.
>>> df.select(pl.min("^a|b$"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 2 │
└─────┴─────┘
>>> df.select(pl.min("a", "b"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 2 │
└─────┴─────┘
min_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Get the minimum value horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(min=pl.min_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬─────┐
│ a ┆ b ┆ c ┆ min │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ i64 │
╞═════╪══════╪═════╪═════╡
│ 1 ┆ 4 ┆ x ┆ 1 │
│ 8 ┆ 5 ┆ y ┆ 5 │
│ 3 ┆ null ┆ z ┆ 3 │
└─────┴──────┴─────┴─────┘
n_unique(*columns: 'str') -> 'Expr'
Count unique values.
This function is syntactic sugar for `pl.col(columns).n_unique()`.
`null` is considered to be a unique value for the purposes of this operation.
Parameters
----------
columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 1, None],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.n_unique("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 2 │
└─────┘
>>> df.select(pl.n_unique("b", "c"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ u32 ┆ u32 │
╞═════╪═════╡
│ 3 ┆ 2 │
└─────┴─────┘
nth(*indices: 'int | Sequence[int]', strict: 'bool' = True) -> 'Expr'
Get the nth column(s) of the context.
Parameters
----------
indices
One or more indices representing the columns to retrieve.
strict
By default, all specified indices must be valid; if any index is out of bounds,
an error is raised. If set to `False`, out-of-bounds indices are ignored.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "baz"],
... }
... )
>>> df.select(pl.nth(1))
shape: (3, 1)
┌─────┐
│ b │
│ --- │
│ i64 │
╞═════╡
│ 4 │
│ 5 │
│ 2 │
└─────┘
>>> df.select(pl.nth(2, 0))
shape: (3, 2)
┌─────┬─────┐
│ c ┆ a │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═════╡
│ foo ┆ 1 │
│ bar ┆ 8 │
│ baz ┆ 3 │
└─────┴─────┘
ones(n: 'int | Expr', dtype: 'PolarsDataType' = Float64, *, eager: 'bool' = False) -> 'Expr | Series'
Construct a column of length `n` filled with ones.
This is syntactic sugar for the `repeat` function.
Parameters
----------
n
Length of the resulting column.
dtype
Data type of the resulting column. Defaults to Float64.
eager
Evaluate immediately and return a `Series`. If set to `False`,
return an expression instead.
Notes
-----
If you want to construct a column in lazy mode and do not need a pre-determined
length, use :func:`lit` instead.
See Also
--------
repeat
lit
Examples
--------
>>> pl.ones(3, pl.Int8, eager=True)
shape: (3,)
Series: 'ones' [i8]
[
1
1
1
]
quantile(column: 'str', quantile: 'float | Expr', interpolation: 'QuantileMethod' = 'nearest') -> 'Expr'
Syntactic sugar for `pl.col("foo").quantile(..)`.
Parameters
----------
column
Column name.
quantile
Quantile between 0.0 and 1.0.
interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
Interpolation method.
reduce(function: 'Callable[[Series, Series], Series]', exprs: 'Sequence[Expr | str] | Expr', *, returns_scalar: 'bool' = False, return_dtype: 'pl.DataTypeExpr | PolarsDataType | None' = None) -> 'Expr'
Accumulate over multiple columns horizontally/ row wise with a left fold.
Parameters
----------
function
Function to apply over the accumulator and the value.
Fn(acc, value) -> new_value
exprs
Expressions to aggregate over. May also be a wildcard expression.
returns_scalar
Whether or not `function` applied returns a scalar. This must be set correctly
by the user.
return_dtype
Output datatype.
If not set, the dtype will be inferred based on the dtype of the input
expressions.
Notes
-----
See `fold` for the version with an explicit accumulator.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [0, 1, 2],
... }
... )
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 0 │
│ 2 ┆ 1 │
│ 3 ┆ 2 │
└─────┴─────┘
Horizontally sum over all columns.
>>> df.select(
... pl.reduce(function=lambda acc, x: acc + x, exprs=pl.col("*")).alias("sum")
... )
shape: (3, 1)
┌─────┐
│ sum │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 3 │
│ 5 │
└─────┘
repeat(value: 'IntoExpr | None', n: 'int | Expr', *, dtype: 'PolarsDataType | None' = None, eager: 'bool' = False) -> 'Expr | Series'
Construct a column of length `n` filled with the given value.
Parameters
----------
value
Value to repeat.
n
Length of the resulting column.
dtype
Data type of the resulting column. If set to `None` (default), data type is
inferred from the given value. Defaults to Int32 for integer values, unless
Int64 is required to fit the given value. Defaults to Float64 for float values.
eager
Evaluate immediately and return a `Series`. If set to `False` (default),
return an expression instead.
Notes
-----
If you want to construct a column in lazy mode and do not need a pre-determined
length, use :func:`lit` instead.
See Also
--------
lit
Examples
--------
Construct a column with a repeated value in a lazy context.
>>> pl.select(pl.repeat("z", n=3)).to_series()
shape: (3,)
Series: 'repeat' [str]
[
"z"
"z"
"z"
]
Generate a Series directly by setting `eager=True`.
>>> pl.repeat(3, n=3, dtype=pl.Int8, eager=True)
shape: (3,)
Series: 'repeat' [i8]
[
3
3
3
]
rolling_corr(a: 'str | Expr', b: 'str | Expr', *, window_size: 'int', min_samples: 'int | None' = None, ddof: 'int' = 1) -> 'Expr'
Compute the rolling correlation between two columns/ expressions.
The window at a given row includes the row itself and the
`window_size - 1` elements before it.
.. versionchanged:: 1.21.0
The `min_periods` parameter was renamed `min_samples`.
Parameters
----------
a
Column name or Expression.
b
Column name or Expression.
window_size
The length of the window.
min_samples
The number of values in the window that should be non-null before computing
a result. If None, it will be set equal to window size.
ddof
Has no effect, do not use.
.. deprecated:: 1.40.0
rolling_cov(a: 'str | Expr', b: 'str | Expr', *, window_size: 'int', min_samples: 'int | None' = None, ddof: 'int' = 1) -> 'Expr'
Compute the rolling covariance between two columns/ expressions.
The window at a given row includes the row itself and the
`window_size - 1` elements before it.
.. versionchanged:: 1.21.0
The `min_periods` parameter was renamed `min_samples`.
Parameters
----------
a
Column name or Expression.
b
Column name or Expression.
window_size
The length of the window.
min_samples
The number of values in the window that should be non-null before computing
a result. If None, it will be set equal to window size.
ddof
Delta degrees of freedom. The divisor used in calculations
is `N - ddof`, where `N` represents the number of elements.
row_index(name: 'str' = 'index') -> 'pl.Expr'
Generates a sequence of integers.
The length of the returned sequence will match the context length, and the
datatype will match the one returned by `get_index_dtype()`.
.. versionadded:: 1.32.0
If you would like to generate sequences with custom offsets / length /
step size / datatypes, it is recommended to use `int_range` instead.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
name
Name of the returned column.
Returns
-------
Expr
Column of integers.
See Also
--------
int_range : Generate a range of integers.
Examples
--------
>>> df = pl.DataFrame({"x": ["A", "A", "B", "B", "B"]})
>>> df.with_columns(pl.row_index(), pl.row_index("another_index"))
shape: (5, 3)
┌─────┬───────┬───────────────┐
│ x ┆ index ┆ another_index │
│ --- ┆ --- ┆ --- │
│ str ┆ u32 ┆ u32 │
╞═════╪═══════╪═══════════════╡
│ A ┆ 0 ┆ 0 │
│ A ┆ 1 ┆ 1 │
│ B ┆ 2 ┆ 2 │
│ B ┆ 3 ┆ 3 │
│ B ┆ 4 ┆ 4 │
└─────┴───────┴───────────────┘
>>> df.group_by("x").agg(pl.row_index()).sort("x")
shape: (2, 2)
┌─────┬───────────┐
│ x ┆ index │
│ --- ┆ --- │
│ str ┆ list[u32] │
╞═════╪═══════════╡
│ A ┆ [0, 1] │
│ B ┆ [0, 1, 2] │
└─────┴───────────┘
>>> df.select(pl.row_index())
shape: (5, 1)
┌───────┐
│ index │
│ --- │
│ u32 │
╞═══════╡
│ 0 │
│ 1 │
│ 2 │
│ 3 │
│ 4 │
└───────┘
select(*exprs: 'IntoExpr | Iterable[IntoExpr]', eager: 'bool' = True, **named_exprs: 'IntoExpr') -> 'DataFrame | LazyFrame'
Run polars expressions without a context.
This is syntactic sugar for running `df.select` on an empty DataFrame
(or LazyFrame if eager=False).
Parameters
----------
*exprs
Column(s) to select, specified as positional arguments.
Accepts expression input. Strings are parsed as column names,
other non-expression inputs are parsed as literals.
eager
Evaluate immediately and return a `DataFrame` (default); if set to `False`,
return a `LazyFrame` instead.
**named_exprs
Additional columns to select, specified as keyword arguments.
The columns will be renamed to the keyword used.
Returns
-------
DataFrame or LazyFrame
Examples
--------
>>> foo = pl.Series("foo", [1, 2, 3])
>>> bar = pl.Series("bar", [3, 2, 1])
>>> pl.select(min=pl.min_horizontal(foo, bar))
shape: (3, 1)
┌─────┐
│ min │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
│ 1 │
└─────┘
>>> pl.select(pl.int_range(0, 100_000, 2).alias("n"), eager=False).filter(
... pl.col("n") % 22_500 == 0
... ).collect()
shape: (5, 1)
┌───────┐
│ n │
│ --- │
│ i64 │
╞═══════╡
│ 0 │
│ 22500 │
│ 45000 │
│ 67500 │
│ 90000 │
└───────┘
self_dtype() -> 'pl.DataTypeExpr'
Get the dtype of `self` in `map_elements` and `map_batches`.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
set_random_seed(seed: 'int') -> 'None'
Set the global random seed for Polars.
This random seed is used to determine things such as shuffle ordering.
Parameters
----------
seed
A non-negative integer < 2\ :sup:`64` used to seed the internal global
random number generator.
sql_expr(sql: 'str | Sequence[str]') -> 'Expr | list[Expr]'
Parse one or more SQL expressions to Polars expression(s).
Parameters
----------
sql
One or more SQL expressions.
Examples
--------
Parse a single SQL expression:
>>> df = pl.DataFrame({"a": [2, 1]})
>>> expr = pl.sql_expr("MAX(a)")
>>> df.select(expr)
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 2 │
└─────┘
Parse multiple SQL expressions:
>>> df.with_columns(
... *pl.sql_expr(["POWER(a,a) AS a_a", "CAST(a AS TEXT) AS a_txt"]),
... )
shape: (2, 3)
┌─────┬─────┬───────┐
│ a ┆ a_a ┆ a_txt │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str │
╞═════╪═════╪═══════╡
│ 2 ┆ 4 ┆ 2 │
│ 1 ┆ 1 ┆ 1 │
└─────┴─────┴───────┘
std(column: 'str', ddof: 'int' = 1) -> 'Expr'
Get the standard deviation.
This function is syntactic sugar for `pl.col(column).std(ddof)`.
Parameters
----------
column
Column name.
ddof
“Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
where N represents the number of elements.
By default ddof is 1.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.std("a"))
shape: (1, 1)
┌──────────┐
│ a │
│ --- │
│ f64 │
╞══════════╡
│ 3.605551 │
└──────────┘
>>> df["a"].std()
3.605551275463989
struct(*exprs: 'IntoExpr | Iterable[IntoExpr]', schema: 'SchemaDict | None' = None, eager: 'bool' = False, **named_exprs: 'IntoExpr') -> 'Expr | Series'
Collect columns into a struct column.
Parameters
----------
*exprs
Column(s) to collect into a struct column, specified as positional arguments.
Accepts expression input. Strings are parsed as column names,
other non-expression inputs are parsed as literals.
schema
Optional schema that explicitly defines the struct field dtypes. If no columns
or expressions are provided, schema keys are used to define columns.
eager
Evaluate immediately and return a `Series`. If set to `False` (default),
return an expression instead.
**named_exprs
Additional columns to collect into the struct column, specified as keyword
arguments. The columns will be renamed to the keyword used.
Examples
--------
Collect all columns of a dataframe into a struct by passing `pl.all()`.
>>> df = pl.DataFrame(
... {
... "int": [1, 2],
... "str": ["a", "b"],
... "bool": [True, None],
... "list": [[1, 2], [3]],
... }
... )
>>> df.select(pl.struct(pl.all()).alias("my_struct"))
shape: (2, 1)
┌─────────────────────┐
│ my_struct │
│ --- │
│ struct[4] │
╞═════════════════════╡
│ {1,"a",true,[1, 2]} │
│ {2,"b",null,[3]} │
└─────────────────────┘
Collect selected columns into a struct by either passing a list of columns, or by
specifying each column as a positional argument.
>>> df.select(pl.struct("int", False).alias("my_struct"))
shape: (2, 1)
┌───────────┐
│ my_struct │
│ --- │
│ struct[2] │
╞═══════════╡
│ {1,false} │
│ {2,false} │
└───────────┘
Use keyword arguments to easily name each struct field.
>>> df.select(pl.struct(p="int", q="bool").alias("my_struct")).schema
Schema({'my_struct': Struct({'p': Int64, 'q': Boolean})})
struct_with_fields(mapping: 'Mapping[str, PolarsDataType | pl.DataTypeExpr]') -> 'pl.DataTypeExpr'
Create a new datatype expression that represents a Struct datatype.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
sum(*names: 'str') -> 'Expr'
Sum all values.
Syntactic sugar for `col(name).sum()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
Notes
-----
If there are no non-null values, then the output is `0`.
If you would prefer empty sums to return `None`, you can
use `pl.when(pl.col(name).count()>0).then(pl.sum(name))` instead
of `pl.sum(name)`.
See Also
--------
sum_horizontal
Examples
--------
Sum a column.
>>> df = pl.DataFrame(
... {
... "a": [1, 2],
... "b": [3, 4],
... "c": [5, 6],
... }
... )
>>> df.select(pl.sum("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 3 │
└─────┘
Sum multiple columns.
>>> df.select(pl.sum("a", "c"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ c │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 3 ┆ 11 │
└─────┴─────┘
>>> df.select(pl.sum("^.*[bc]$"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 7 ┆ 11 │
└─────┴─────┘
sum_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]', ignore_nulls: 'bool' = True) -> 'Expr'
Sum all values horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
ignore_nulls
Ignore null values (default).
If set to `False`, any null value in the input will lead to a null output.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(sum=pl.sum_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬─────┐
│ a ┆ b ┆ c ┆ sum │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ i64 │
╞═════╪══════╪═════╪═════╡
│ 1 ┆ 4 ┆ x ┆ 5 │
│ 8 ┆ 5 ┆ y ┆ 13 │
│ 3 ┆ null ┆ z ┆ 3 │
└─────┴──────┴─────┴─────┘
tail(column: 'str', n: 'int' = 10) -> 'Expr'
Get the last `n` rows.
This function is syntactic sugar for `pl.col(column).tail(n)`.
Parameters
----------
column
Column name.
n
Number of rows to return.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.tail("a"))
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 8 │
│ 3 │
└─────┘
>>> df.select(pl.tail("a", 2))
shape: (2, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 8 │
│ 3 │
└─────┘
time = time_(hour: 'Expr | str | int | None' = None, minute: 'Expr | str | int | None' = None, second: 'Expr | str | int | None' = None, microsecond: 'Expr | str | int | None' = None) -> 'Expr'
Create a Polars literal expression of type Time.
Parameters
----------
hour
column or literal, ranging from 0-23.
minute
column or literal, ranging from 0-59.
second
column or literal, ranging from 0-59.
microsecond
column or literal, ranging from 0-999999.
Returns
-------
Expr
Expression of data type :class:`Date`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "hour": [12, 13, 14],
... "minute": [15, 30, 45],
... }
... )
>>> df.with_columns(pl.time(pl.col("hour"), pl.col("minute")))
shape: (3, 3)
┌──────┬────────┬──────────┐
│ hour ┆ minute ┆ time │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ time │
╞══════╪════════╪══════════╡
│ 12 ┆ 15 ┆ 12:15:00 │
│ 13 ┆ 30 ┆ 13:30:00 │
│ 14 ┆ 45 ┆ 14:45:00 │
└──────┴────────┴──────────┘
time_range(start: 'time | IntoExprColumn | None' = None, end: 'time | IntoExprColumn | None' = None, interval: 'str | timedelta' = '1h', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Series | Expr'
Generate a time range.
Parameters
----------
start
Lower bound of the time range.
If omitted, defaults to `time(0,0,0,0)`.
end
Upper bound of the time range.
If omitted, defaults to `time(23,59,59,999999)`.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type `:class:Time`.
Notes
-----
`interval` is created according to the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
See Also
--------
time_ranges : Create a column of time ranges.
Examples
--------
>>> from datetime import time, timedelta
>>> pl.time_range(
... start=time(14, 0),
... interval=timedelta(hours=3, minutes=15),
... eager=True,
... ).alias("time")
shape: (4,)
Series: 'time' [time]
[
14:00:00
17:15:00
20:30:00
23:45:00
]
time_ranges(start: 'time | IntoExprColumn | None' = None, end: 'time | IntoExprColumn | None' = None, interval: 'str | timedelta' = '1h', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Series | Expr'
Create a column of time ranges.
Parameters
----------
start
Lower bound of the time range.
If omitted, defaults to `time(0, 0, 0, 0)`.
end
Upper bound of the time range.
If omitted, defaults to `time(23, 59, 59, 999999)`.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type `List(Time)`.
Notes
-----
`interval` is created according to the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
See Also
--------
time_range : Generate a single time range.
Examples
--------
>>> from datetime import time
>>> df = pl.DataFrame(
... {
... "start": [time(9, 0), time(10, 0)],
... "end": time(11, 0),
... }
... )
>>> df.with_columns(time_range=pl.time_ranges("start", "end"))
shape: (2, 3)
┌──────────┬──────────┬────────────────────────────────┐
│ start ┆ end ┆ time_range │
│ --- ┆ --- ┆ --- │
│ time ┆ time ┆ list[time] │
╞══════════╪══════════╪════════════════════════════════╡
│ 09:00:00 ┆ 11:00:00 ┆ [09:00:00, 10:00:00, 11:00:00] │
│ 10:00:00 ┆ 11:00:00 ┆ [10:00:00, 11:00:00] │
└──────────┴──────────┴────────────────────────────────┘
union(items: 'Iterable[PolarsType]', *, how: 'ConcatMethod' = 'vertical', strict: 'bool | None' = None) -> 'PolarsType'
Combine multiple DataFrames, LazyFrames, or Series into a single object.
.. warning::
This function does not guarantee any specific ordering of rows in the result.
If you need predictable row ordering, use `pl.concat()` instead.
Parameters
----------
items
DataFrames, LazyFrames, or Series to concatenate.
how : {'vertical', 'vertical_relaxed', 'diagonal', 'diagonal_relaxed', 'horizontal', 'horizontal_extend', 'align', 'align_full', 'align_inner', 'align_left', 'align_right'}
Note that `Series` only support the `vertical` strategy.
* vertical: Applies multiple `vstack` operations.
* vertical_relaxed: Same as `vertical`, but additionally coerces columns to
their common supertype *if* they are mismatched (eg: Int32 → Int64).
* diagonal: Finds a union between the column schemas and fills missing column
values with `null`.
* diagonal_relaxed: Same as `diagonal`, but additionally coerces columns to
their common supertype *if* they are mismatched (eg: Int32 → Int64).
* horizontal: Stacks Series from DataFrames horizontally. All input frames
must have the same height; raises a `ShapeError` otherwise.
* horizontal_extend: Same as `horizontal`, but pads shorter frames with `null`
if the lengths don't match.
* align, align_full, align_left, align_right: Combines frames horizontally,
auto-determining the common key columns and aligning rows using the same
logic as `align_frames` (note that "align" is an alias for "align_full").
The "align" strategy determines the type of join used to align the frames,
equivalent to the "how" parameter on `align_frames`. Note that the common
join columns are automatically coalesced, but other column collisions
will raise an error (if you need more control over this you should use
a suitable `join` method directly).
strict
When how=`horizontal`, require all DataFrames to be the same height, raising an error if not.
.. deprecated:: 1.42.1
Use `how='horizontal'` (equal heights) or
`how='horizontal_extend'` (pad with null) instead.
Examples
--------
>>> df1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df2 = pl.DataFrame({"a": [2], "b": [4]})
>>> pl.union([df1, df2]) # default is 'vertical' strategy
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 3 │
│ 2 ┆ 4 │
└─────┴─────┘
>>> df1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df2 = pl.DataFrame({"a": [2.5], "b": [4]})
>>> pl.union([df1, df2], how="vertical_relaxed") # 'a' coerced into f64
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ f64 ┆ i64 │
╞═════╪═════╡
│ 1.0 ┆ 3 │
│ 2.5 ┆ 4 │
└─────┴─────┘
>>> df_h1 = pl.DataFrame({"l1": [1, 2], "l2": [3, 4]})
>>> df_h2 = pl.DataFrame({"r1": [5, 6], "r2": [7, 8], "r3": [9, 10]})
>>> pl.union([df_h1, df_h2], how="horizontal_extend")
shape: (2, 5)
┌─────┬─────┬─────┬─────┬─────┐
│ l1 ┆ l2 ┆ r1 ┆ r2 ┆ r3 │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 5 ┆ 7 ┆ 9 │
│ 2 ┆ 4 ┆ 6 ┆ 8 ┆ 10 │
└─────┴─────┴─────┴─────┴─────┘
The "diagonal" strategy allows for some frames to have missing columns,
the values for which are filled with `null`:
>>> df_d1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df_d2 = pl.DataFrame({"a": [2], "c": [4]})
>>> pl.union([df_d1, df_d2], how="diagonal")
shape: (2, 3)
┌─────┬──────┬──────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╡
│ 1 ┆ 3 ┆ null │
│ 2 ┆ null ┆ 4 │
└─────┴──────┴──────┘
The "align" strategies require at least one common column to align on:
>>> df_a1 = pl.DataFrame({"id": [1, 2], "x": [3, 4]})
>>> df_a2 = pl.DataFrame({"id": [2, 3], "y": [5, 6]})
>>> df_a3 = pl.DataFrame({"id": [1, 3], "z": [7, 8]})
>>> pl.union([df_a1, df_a2, df_a3], how="align") # equivalent to "align_full"
shape: (3, 4)
┌─────┬──────┬──────┬──────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╪══════╡
│ 1 ┆ 3 ┆ null ┆ 7 │
│ 2 ┆ 4 ┆ 5 ┆ null │
│ 3 ┆ null ┆ 6 ┆ 8 │
└─────┴──────┴──────┴──────┘
>>> pl.union([df_a1, df_a2, df_a3], how="align_left")
shape: (2, 4)
┌─────┬─────┬──────┬──────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪══════╪══════╡
│ 1 ┆ 3 ┆ null ┆ 7 │
│ 2 ┆ 4 ┆ 5 ┆ null │
└─────┴─────┴──────┴──────┘
>>> pl.union([df_a1, df_a2, df_a3], how="align_right")
shape: (2, 4)
┌─────┬──────┬──────┬─────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╪═════╡
│ 1 ┆ null ┆ null ┆ 7 │
│ 3 ┆ null ┆ 6 ┆ 8 │
└─────┴──────┴──────┴─────┘
>>> pl.union([df_a1, df_a2, df_a3], how="align_inner")
shape: (0, 4)
┌─────┬─────┬─────┬─────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
└─────┴─────┴─────┴─────┘
var(column: 'str', ddof: 'int' = 1) -> 'Expr'
Get the variance.
This function is syntactic sugar for `pl.col(column).var(ddof)`.
Parameters
----------
column
Column name.
ddof
“Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
where N represents the number of elements.
By default ddof is 1.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... },
... )
>>> df.select(pl.var("a"))
shape: (1, 1)
┌──────┐
│ a │
│ --- │
│ f64 │
╞══════╡
│ 13.0 │
└──────┘
>>> df["a"].var()
13.0
when(*predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool', **constraints: 'Any') -> 'pl.When'
Start a `when-then-otherwise` expression.
Always initiated by a `pl.when().then()`., and optionally followed by chaining one
or more `.when().then()` statements.
An optional `.otherwise()` can be appended at the end. If not declared, a default
of `.otherwise(None)` is used.
Similar to :func:`coalesce`, the value from the first condition that
evaluates to True will be picked.
If all conditions are False, the `otherwise` value is picked.
Parameters
----------
predicates
Condition(s) that must be met in order to apply the subsequent statement.
Accepts one or more boolean expressions, which are implicitly combined with
`&`.
constraints
Apply conditions as `col_name = value` keyword arguments that are treated as
equality matches, such as `x = 123`. As with the predicates parameter, multiple
conditions are implicitly combined using `&`.
Warnings
--------
Polars computes all expressions passed to `when-then-otherwise` in parallel and
filters afterwards. This means each expression must be valid on its own, regardless
of the conditions in the `when-then-otherwise` chain.
Notes
-----
* String inputs e.g. `when("string")`, `then("string")` or `otherwise("string")`
are parsed as column names. :func:`lit` can be used to create string values.
* The expression output name is taken from the first `then` statement. It is
not affected by `predicates`, nor by `constraints`.
Examples
--------
Below we add a column with the value 1, where column "foo" > 2 and the value
1 + column "bar" where it isn't.
>>> df = pl.DataFrame({"foo": [1, 3, 4], "bar": [3, 4, 0]})
>>> df.with_columns(
... pl.when(pl.col.foo > 2).then(1).otherwise(1 + pl.col.bar).alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 4 │
│ 3 ┆ 4 ┆ 1 │
│ 4 ┆ 0 ┆ 1 │
└─────┴─────┴─────┘
Note that `when-then` always executes all expressions.
The results are folded left to right, picking the `then` value from the first `when`
condition that is True.
If no `when` condition is True the `otherwise` value is picked.
>>> df.with_columns(
... when = pl.col.foo > 2,
... then = 1,
... otherwise = 1 + pl.col.bar
... ).with_columns(
... pl.when("when").then("then").otherwise("otherwise").alias("val")
... )
shape: (3, 6)
┌─────┬─────┬───────┬──────┬───────────┬─────┐
│ foo ┆ bar ┆ when ┆ then ┆ otherwise ┆ val │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool ┆ i32 ┆ i64 ┆ i64 │
╞═════╪═════╪═══════╪══════╪═══════════╪═════╡
│ 1 ┆ 3 ┆ false ┆ 1 ┆ 4 ┆ 4 │
│ 3 ┆ 4 ┆ true ┆ 1 ┆ 5 ┆ 1 │
│ 4 ┆ 0 ┆ true ┆ 1 ┆ 1 ┆ 1 │
└─────┴─────┴───────┴──────┴───────────┴─────┘
Note that in regular Polars usage, a single string is parsed as a column name.
>>> df.with_columns(
... when = pl.col.foo > 2,
... then = "foo",
... otherwise = "bar"
... )
shape: (3, 5)
┌─────┬─────┬───────┬──────┬───────────┐
│ foo ┆ bar ┆ when ┆ then ┆ otherwise │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool ┆ i64 ┆ i64 │
╞═════╪═════╪═══════╪══════╪═══════════╡
│ 1 ┆ 3 ┆ false ┆ 1 ┆ 3 │
│ 3 ┆ 4 ┆ true ┆ 3 ┆ 4 │
│ 4 ┆ 0 ┆ true ┆ 4 ┆ 0 │
└─────┴─────┴───────┴──────┴───────────┘
For consistency, `when-then` behaves in the same way.
>>> df.with_columns(
... pl.when(pl.col.foo > 2).then("foo").otherwise("bar").alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 3 │
│ 3 ┆ 4 ┆ 3 │
│ 4 ┆ 0 ┆ 4 │
└─────┴─────┴─────┘
:func:`lit` can be used to create string values.
>>> df.with_columns(
... pl.when(pl.col.foo > 2)
... .then(pl.lit("foo"))
... .otherwise(pl.lit("bar"))
... .alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ bar │
│ 3 ┆ 4 ┆ foo │
│ 4 ┆ 0 ┆ foo │
└─────┴─────┴─────┘
Multiple `when-then` statements can be chained.
>>> df.with_columns(
... pl.when(pl.col.foo > 2)
... .then(1)
... .when(pl.col.bar > 2)
... .then(4)
... .otherwise(-1)
... .alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i32 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 4 │
│ 3 ┆ 4 ┆ 1 │
│ 4 ┆ 0 ┆ 1 │
└─────┴─────┴─────┘
In the case of `foo=3` and `bar=4`, both conditions are True but the first value
(i.e. 1) is picked.
>>> df.with_columns(
... when1 = pl.col.foo > 2,
... then1 = 1,
... when2 = pl.col.bar > 2,
... then2 = 4,
... otherwise = -1
... )
shape: (3, 7)
┌─────┬─────┬───────┬───────┬───────┬───────┬───────────┐
│ foo ┆ bar ┆ when1 ┆ then1 ┆ when2 ┆ then2 ┆ otherwise │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool ┆ i32 ┆ bool ┆ i32 ┆ i32 │
╞═════╪═════╪═══════╪═══════╪═══════╪═══════╪═══════════╡
│ 1 ┆ 3 ┆ false ┆ 1 ┆ true ┆ 4 ┆ -1 │
│ 3 ┆ 4 ┆ true ┆ 1 ┆ true ┆ 4 ┆ -1 │
│ 4 ┆ 0 ┆ true ┆ 1 ┆ false ┆ 4 ┆ -1 │
└─────┴─────┴───────┴───────┴───────┴───────┴───────────┘
The `otherwise` statement is optional and defaults to `.otherwise(None)`
if not given.
This idiom is commonly used to null out values.
>>> df.with_columns(pl.when(pl.col.foo == 3).then("bar"))
shape: (3, 2)
┌─────┬──────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪══════╡
│ 1 ┆ null │
│ 3 ┆ 4 │
│ 4 ┆ null │
└─────┴──────┘
`when` accepts keyword arguments as shorthand for equality conditions.
>>> df.with_columns(pl.when(foo=3).then("bar"))
shape: (3, 2)
┌─────┬──────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪══════╡
│ 1 ┆ null │
│ 3 ┆ 4 │
│ 4 ┆ null │
└─────┴──────┘
Multiple predicates passed to `when` are combined with `&`
>>> df.with_columns(
... pl.when(pl.col.foo > 2, pl.col.bar < 3) # when((pred1) & (pred2))
... .then(pl.lit("Yes"))
... .otherwise(pl.lit("No"))
... .alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ No │
│ 3 ┆ 4 ┆ No │
│ 4 ┆ 0 ┆ Yes │
└─────┴─────┴─────┘
It could also be thought of as an implicit :func:`all_horizontal` being present.
>>> df.with_columns(
... when = pl.all_horizontal(pl.col.foo > 2, pl.col.bar < 3)
... )
shape: (3, 3)
┌─────┬─────┬───────┐
│ foo ┆ bar ┆ when │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool │
╞═════╪═════╪═══════╡
│ 1 ┆ 3 ┆ false │
│ 3 ┆ 4 ┆ false │
│ 4 ┆ 0 ┆ true │
└─────┴─────┴───────┘
Structs can be used as a way to return multiple values.
Here we swap the "foo" and "bar" values when "foo" is greater than 2.
>>> df.with_columns(
... pl.when(pl.col.foo > 2)
... .then(pl.struct(foo="bar", bar="foo"))
... .otherwise(pl.struct("foo", "bar"))
... .struct.unnest()
... )
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 3 │
│ 4 ┆ 3 │
│ 0 ┆ 4 │
└─────┴─────┘
The struct fields are given the same name as the target columns, which are then
unnested.
>>> df.with_columns(
... when = pl.col.foo > 2,
... then = pl.struct(foo="bar", bar="foo"),
... otherwise = pl.struct("foo", "bar")
... )
shape: (3, 5)
┌─────┬─────┬───────┬───────────┬───────────┐
│ foo ┆ bar ┆ when ┆ then ┆ otherwise │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool ┆ struct[2] ┆ struct[2] │
╞═════╪═════╪═══════╪═══════════╪═══════════╡
│ 1 ┆ 3 ┆ false ┆ {3,1} ┆ {1,3} │
│ 3 ┆ 4 ┆ true ┆ {4,3} ┆ {3,4} │
│ 4 ┆ 0 ┆ true ┆ {0,4} ┆ {4,0} │
└─────┴─────┴───────┴───────────┴───────────┘
The output name of a `when-then` expression comes from the first `then` branch.
Here we try to set all columns to 0 if any column contains a value less than 2.
>>> df.with_columns( # doctest: +SKIP
... pl.when(pl.any_horizontal(pl.all() < 2))
... .then(0)
... .otherwise(pl.all())
... )
# ComputeError: the name 'literal' passed to `LazyFrame.with_columns` is duplicate
:meth:`.name.keep` could be used to give preference to the column expression.
>>> df.with_columns(
... pl.when(pl.any_horizontal(pl.all() < 2))
... .then(0)
... .otherwise(pl.all())
... .name.keep()
... )
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 0 ┆ 0 │
│ 3 ┆ 4 │
│ 0 ┆ 0 │
└─────┴─────┘
The logic could also be changed to move the column expression inside `then`.
>>> df.with_columns(
... pl.when(pl.any_horizontal(pl.all() < 2).not_())
... .then(pl.all())
... .otherwise(0)
... )
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 0 ┆ 0 │
│ 3 ┆ 4 │
│ 0 ┆ 0 │
└─────┴─────┘
zeros(n: 'int | Expr', dtype: 'PolarsDataType' = Float64, *, eager: 'bool' = False) -> 'Expr | Series'
Construct a column of length `n` filled with zeros.
This is syntactic sugar for the `repeat` function.
Parameters
----------
n
Length of the resulting column.
dtype
Data type of the resulting column. Defaults to Float64.
eager
Evaluate immediately and return a `Series`. If set to `False`,
return an expression instead.
Notes
-----
If you want to construct a column in lazy mode and do not need a pre-determined
length, use :func:`lit` instead.
See Also
--------
repeat
lit
Examples
--------
>>> pl.zeros(3, pl.Int8, eager=True)
shape: (3,)
Series: 'zeros' [i8]
[
0
0
0
]
DATA
__all__ = ['all', 'any', 'cum_sum', 'max', 'min', 'sum', 'all_horizont...
col = <polars.functions.col.Col object>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\functions\__init__.py
polars.get_extension_type
Python Library Documentation: function get_extension_type in module polars.datatypes.extension
get_extension_type(ext_name: 'str') -> 'type[dt.BaseExtension] | str | None'
Get the extension type class for the given extension name.
If an extension is registered to be passed through as storage, this returns
the string "storage".
.. warning::
This functionality is currently considered **unstable**. It may be
changed at any point without it being considered a breaking change.
polars.get_index_type
Python Library Documentation: function get_index_type in module polars.meta.index_type
get_index_type() -> 'PolarsIntegerType'
Return the data type used for Polars indexing.
Returns
-------
PolarsIntegerType
:class:`UInt32` in regular Polars, :class:`UInt64` in bigidx Polars.
Examples
--------
>>> pl.get_index_type()
UInt32
polars.groups
Python Library Documentation: function groups in module polars.functions.lazy
groups(column: 'str') -> 'Expr'
Syntactic sugar for `pl.col("foo").agg_groups()`.
.. deprecated:: 1.35
Use `df.with_row_index().group_by(...).agg(pl.col('index'))` instead.
This method will be removed in Polars 2.0.
polars.head
Python Library Documentation: function head in module polars.functions.lazy
head(column: 'str', n: 'int' = 10) -> 'Expr'
Get the first `n` rows.
This function is syntactic sugar for `pl.col(column).head(n)`.
Parameters
----------
column
Column name.
n
Number of rows to return.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.head("a"))
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 8 │
│ 3 │
└─────┘
>>> df.select(pl.head("a", 2))
shape: (2, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 8 │
└─────┘
polars.implode
Python Library Documentation: function implode in module polars.functions.lazy
implode(*columns: 'str', maintain_order: 'bool' = True) -> 'Expr'
Aggregate all column values into a list.
This function is syntactic sugar for `pl.col(name).implode()`.
Parameters
----------
*columns
One or more column names.
maintain_order
Whether to preserve the order of elements in the list. Setting this
to `False` can improve performance, especially within `group_by`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [9, 8, 7],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.implode("a"))
shape: (1, 1)
┌───────────┐
│ a │
│ --- │
│ list[i64] │
╞═══════════╡
│ [1, 2, 3] │
└───────────┘
>>> df.select(pl.implode("b", "c"))
shape: (1, 2)
┌───────────┬───────────────────────┐
│ b ┆ c │
│ --- ┆ --- │
│ list[i64] ┆ list[str] │
╞═══════════╪═══════════════════════╡
│ [9, 8, 7] ┆ ["foo", "bar", "foo"] │
└───────────┴───────────────────────┘
polars.int_range
Python Library Documentation: function int_range in module polars.functions.range.int_range
int_range(start: 'int | IntoExprColumn' = 0, end: 'int | IntoExprColumn | None' = None, step: 'int' = 1, *, dtype: 'PolarsIntegerType | DataTypeExpr' = Int64, eager: 'bool' = False) -> 'Expr | Series'
Generate a range of integers.
Parameters
----------
start
Start of the range (inclusive). Defaults to 0.
end
End of the range (exclusive). If set to `None` (default),
the value of `start` is used and `start` is set to `0`.
step
Step size of the range.
dtype
Data type of the range.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of integer data type `dtype`.
See Also
--------
int_ranges : Generate a range of integers for each row of the input columns.
Examples
--------
>>> pl.int_range(0, 3, eager=True)
shape: (3,)
Series: 'literal' [i64]
[
0
1
2
]
`end` can be omitted for a shorter syntax.
>>> pl.int_range(3, eager=True)
shape: (3,)
Series: 'literal' [i64]
[
0
1
2
]
Generate an index column by using `int_range` in conjunction with :func:`len`.
>>> df = pl.DataFrame({"a": [1, 3, 5], "b": [2, 4, 6]})
>>> df.select(
... pl.int_range(pl.len(), dtype=pl.UInt32).alias("index"),
... pl.all(),
... )
shape: (3, 3)
┌───────┬─────┬─────┐
│ index ┆ a ┆ b │
│ --- ┆ --- ┆ --- │
│ u32 ┆ i64 ┆ i64 │
╞═══════╪═════╪═════╡
│ 0 ┆ 1 ┆ 2 │
│ 1 ┆ 3 ┆ 4 │
│ 2 ┆ 5 ┆ 6 │
└───────┴─────┴─────┘
polars.int_ranges
Python Library Documentation: function int_ranges in module polars.functions.range.int_range
int_ranges(start: 'int | IntoExprColumn' = 0, end: 'int | IntoExprColumn | None' = None, step: 'int | IntoExprColumn' = 1, *, dtype: 'PolarsIntegerType | DataTypeExpr' = Int64, eager: 'bool' = False) -> 'Expr | Series'
Generate a range of integers for each row of the input columns.
Parameters
----------
start
Start of the range (inclusive). Defaults to 0.
end
End of the range (exclusive). If set to `None` (default),
the value of `start` is used and `start` is set to `0`.
step
Step size of the range.
dtype
Integer data type of the ranges. Defaults to `Int64`.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type `List(dtype)`.
See Also
--------
int_range : Generate a single range of integers.
Examples
--------
>>> df = pl.DataFrame({"start": [1, -1], "end": [3, 2]})
>>> df.with_columns(int_range=pl.int_ranges("start", "end"))
shape: (2, 3)
┌───────┬─────┬────────────┐
│ start ┆ end ┆ int_range │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[i64] │
╞═══════╪═════╪════════════╡
│ 1 ┆ 3 ┆ [1, 2] │
│ -1 ┆ 2 ┆ [-1, 0, 1] │
└───────┴─────┴────────────┘
`end` can be omitted for a shorter syntax.
>>> df.select("end", int_range=pl.int_ranges("end"))
shape: (2, 2)
┌─────┬───────────┐
│ end ┆ int_range │
│ --- ┆ --- │
│ i64 ┆ list[i64] │
╞═════╪═══════════╡
│ 3 ┆ [0, 1, 2] │
│ 2 ┆ [0, 1] │
└─────┴───────────┘
polars.interchange
Python Library Documentation: package polars.interchange in polars
NAME
polars.interchange - Module containing the implementation of the Python dataframe interchange protocol.
DESCRIPTION
Details on the protocol:
https://data-apis.org/dataframe-protocol/latest/index.html
PACKAGE CONTENTS
buffer
column
dataframe
from_dataframe
protocol
utils
CLASSES
builtins.object
polars.interchange.protocol.CompatLevel
class CompatLevel(builtins.object)
| CompatLevel() -> 'None'
|
| Data structure compatibility level.
|
| Methods defined here:
|
| __init__(self) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| newest() -> 'CompatLevel'
| Get the highest supported compatibility level.
|
| .. warning::
| Highest compatibility level is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| oldest() -> 'CompatLevel'
| Get the most compatible level.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_version': 'int'}
DATA
__all__ = ['CompatLevel']
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\interchange\__init__.py
polars.io
Python Library Documentation: package polars.io in polars
NAME
polars.io - Functions for reading data.
PACKAGE CONTENTS
_expand_paths
_utils
arrow_c_stream
avro
clipboard
cloud (package)
csv (package)
database (package)
delta (package)
iceberg (package)
ipc (package)
json (package)
lines
ndjson
parquet (package)
partition
plugins
pyarrow_dataset (package)
scan_options (package)
spreadsheet (package)
CLASSES
builtins.object
polars.io.partition.FileProviderArgs
polars.io.partition.PartitionBy
polars.io.scan_options.cast_options.ScanCastOptions
class FileProviderArgs(builtins.object)
| FileProviderArgs(*, index_in_partition: 'int', partition_keys: 'DataFrame') -> None
|
| Holds information on the file being sinked to.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, *, index_in_partition: 'int', partition_keys: 'DataFrame') -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'index_in_partition': 'int', 'partition_keys': 'Dat...
|
| __dataclass_fields__ = {'index_in_partition': Field(name='index_in_par...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ()
class PartitionBy(builtins.object)
| PartitionBy(base_path: 'str | Path', *, file_path_provider: 'Callable[[FileProviderArgs], str | Path | IO[bytes] | IO[str]] | _InternalPlPathProviderConfig | None' = None, key: 'str | Expr | Sequence[str | Expr] | Mapping[str, Expr] | None' = None, include_key: 'bool | None' = None, max_rows_per_file: 'int | None' = None, approximate_bytes_per_file: "int | Literal['auto'] | None" = 'auto') -> 'None'
|
| Configuration for writing to multiple output files.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| base_path
| Base path to write to.
| file_path_provider
| Callable for custom file output paths.
| key
| Expressions to partition by.
| include_key
| Include the partition key expression outputs in the output files.
| max_rows_per_file
| Maximum number of rows to write for each file. Note that files may have
| less than this amount of rows.
| approximate_bytes_per_file
| Approximate number of bytes to write to each file. This is measured as
| the estimated size of the DataFrame in memory.
|
| Examples
| --------
| Split to multiple files partitioned by year:
|
| >>> pl.LazyFrame({"year": [2026, 2027, 1970], "month": [0, 0, 0]}).sink_parquet(
| ... pl.PartitionBy("data/", key="year")
| ... ) # doctest: +SKIP
|
| Split to multiple files based on size:
|
| >>> pl.LazyFrame({"year": [2026, 2027, 1970], "month": [0, 0, 0]}).sink_parquet(
| ... pl.PartitionBy(
| ... "data/", max_rows_per_file=1000, approximate_bytes_per_file=100_000_000
| ... )
| ... ) # doctest: +SKIP
|
| Split to multiple files partitioned by year, with limits on individual file sizes:
|
| >>> pl.LazyFrame({"year": [2026, 2027, 1970], "month": [0, 0, 0]}).sink_parquet(
| ... pl.PartitionBy(
| ... "data/",
| ... key="year",
| ... max_rows_per_file=1000,
| ... approximate_bytes_per_file=100_000_000,
| ... )
| ... ) # doctest: +SKIP
|
| Methods defined here:
|
| __init__(self, base_path: 'str | Path', *, file_path_provider: 'Callable[[FileProviderArgs], str | Path | IO[bytes] | IO[str]] | _InternalPlPathProviderConfig | None' = None, key: 'str | Expr | Sequence[str | Expr] | Mapping[str, Expr] | None' = None, include_key: 'bool | None' = None, max_rows_per_file: 'int | None' = None, approximate_bytes_per_file: "int | Literal['auto'] | None" = 'auto') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class ScanCastOptions(builtins.object)
| ScanCastOptions(*, integer_cast: 'IntegerCastOption | Collection[IntegerCastOption]' = 'forbid', float_cast: 'FloatCastOption | Collection[FloatCastOption]' = 'forbid', datetime_cast: 'DatetimeCastOption | Collection[DatetimeCastOption]' = 'forbid', missing_struct_fields: "Literal['insert', 'raise']" = 'raise', extra_struct_fields: "Literal['ignore', 'raise']" = 'raise', categorical_to_string: "Literal['allow', 'forbid']" = 'forbid', _internal_call: 'bool' = False) -> 'None'
|
| Cast options applied when scanning files.
|
| Methods defined here:
|
| __init__(self, *, integer_cast: 'IntegerCastOption | Collection[IntegerCastOption]' = 'forbid', float_cast: 'FloatCastOption | Collection[FloatCastOption]' = 'forbid', datetime_cast: 'DatetimeCastOption | Collection[DatetimeCastOption]' = 'forbid', missing_struct_fields: "Literal['insert', 'raise']" = 'raise', extra_struct_fields: "Literal['ignore', 'raise']" = 'raise', categorical_to_string: "Literal['allow', 'forbid']" = 'forbid', _internal_call: 'bool' = False) -> 'None'
| Common configuration for scanning files.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| integer_cast
| Configuration for casting from integer types:
|
| * `upcast`: Allow lossless casting to wider integer types.
| * `allow-float`: Allow casting integers to float types.
| * `forbid`: Raises an error if dtypes do not match.
|
| float_cast
| Configuration for casting from float types:
|
| * `upcast`: Allow casting to higher precision float types.
| * `downcast`: Allow casting to lower precision float types.
| * `forbid`: Raises an error if dtypes do not match.
|
| datetime_cast
| Configuration for casting from datetime types:
|
| * `nanosecond-downcast`: Allow nanosecond precision datetime to be
| downcasted to any lower precision. This has a similar effect to
| PyArrow's `coerce_int96_timestamp_unit`.
| * `microsecond-downcast`: Allow microsecond precision datetime to be
| downcasted to millisecond precision.
| * `microsecond-upcast`: Allow microsecond precision datetime to be
| upcasted to nanosecond precision.
| * `millisecond-upcast`: Allow millisecond precision datetime to be
| upcasted to microsecond or nanosecond precision.
| * `downcast`: Allow downcasting to any lower precision (convenience
| aggregate of `nanosecond-downcast` and `microsecond-downcast`).
| * `upcast`: Allow upcasting to any higher precision (convenience
| aggregate of `millisecond-upcast` and `microsecond-upcast`).
| * `convert-timezone`: Allow casting to a different timezone.
| * `forbid`: Raises an error if dtypes do not match.
|
| missing_struct_fields
| Configuration for behavior when struct fields defined in the schema
| are missing from the data:
|
| * `insert`: Inserts the missing fields.
| * `raise`: Raises an error.
|
| extra_struct_fields
| Configuration for behavior when extra struct fields outside of the
| defined schema are encountered in the data:
|
| * `ignore`: Silently ignores.
| * `raise`: Raises an error.
|
| categorical_to_string
| Configuration for behavior when reading in a column whose expected
| type is string, but type in the file is categorical.
|
| * `allow`: Categorical is casted to string.
| * `forbid`: Raises an error.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
defer = _defer(function: 'Callable[[], DataFrame]', *, schema: 'SchemaDict | Callable[[], SchemaDict]', validate_schema: 'bool' = True) -> 'LazyFrame'
Deferred execution.
Takes a function that produces a `DataFrame` but defers execution until the
`LazyFrame` is collected.
Parameters
----------
function
Function that takes no arguments and produces a `DataFrame`.
schema
Schema of the `DataFrame` the deferred function will return.
The caller must ensure this schema is correct.
validate_schema
Whether the engine should validate if the batches generated match
the given schema. It's an implementation error if this isn't
the case and can lead to bugs that are hard to solve.
Examples
--------
Delay DataFrame execution until query is executed.
>>> import numpy as np
>>> np.random.seed(0)
>>> lf = pl.defer(
... lambda: pl.DataFrame({"a": np.random.randn(3)}), schema={"a": pl.Float64}
... )
>>> lf.collect()
shape: (3, 1)
┌──────────┐
│ a │
│ --- │
│ f64 │
╞══════════╡
│ 1.764052 │
│ 0.400157 │
│ 0.978738 │
└──────────┘
Run an eager source in Polars Cloud
>>> (
... pl.defer(
... lambda: pl.read_database("select * from tbl"),
... schema={"a": pl.Float64, "b": pl.Boolean},
... )
... .filter("b")
... .sum("a")
... .remote()
... .collect()
... ) # doctest: +SKIP
read_avro(source: 'str | Path | IO[bytes] | bytes', *, columns: 'list[int] | list[str] | None' = None, n_rows: 'int | None' = None) -> 'DataFrame'
Read into a DataFrame from Apache Avro format.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
columns
Columns to select. Accepts a list of column indices (starting at zero) or a list
of column names.
n_rows
Stop reading from Apache Avro file after reading `n_rows`.
Returns
-------
DataFrame
read_clipboard(separator: 'str' = '\t', **kwargs: 'Any') -> 'DataFrame'
Read text from clipboard and pass to `read_csv`.
Useful for reading data copied from Excel or other similar spreadsheet software.
Parameters
----------
separator
Single byte character to use as separator parsing csv from clipboard.
kwargs
Additional arguments passed to `read_csv`.
See Also
--------
read_csv : Read a csv file into a DataFrame.
DataFrame.write_clipboard : Write a DataFrame to the clipboard.
read_csv(source: 'str | Path | IO[str] | IO[bytes] | bytes', *, has_header: 'bool' = True, columns: 'Sequence[int] | Sequence[str] | None' = None, new_columns: 'Sequence[str] | None' = None, separator: 'str' = ',', comment_prefix: 'str | None' = None, quote_char: 'str | None' = '"', skip_rows: 'int' = 0, skip_lines: 'int' = 0, schema: 'SchemaDict | None' = None, schema_overrides: 'Mapping[str, PolarsDataType] | Sequence[PolarsDataType] | None' = None, null_values: 'str | Sequence[str] | dict[str, str] | None' = None, empty_string_is_null: 'bool' = True, ignore_errors: 'bool' = False, try_parse_dates: 'bool' = False, n_threads: 'int | None' = None, infer_schema: 'bool' = True, infer_schema_length: 'int | None' = 100, batch_size: 'int' = 8192, n_rows: 'int | None' = None, encoding: 'CsvEncoding | str' = 'utf8', low_memory: 'bool' = False, rechunk: 'bool' = False, use_pyarrow: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, skip_rows_after_header: 'int' = 0, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, sample_size: 'int' = 1024, eol_char: 'str' = '\n', raise_if_empty: 'bool' = True, truncate_ragged_lines: 'bool' = False, decimal_comma: 'bool' = False, glob: 'bool' = True) -> 'DataFrame'
Read a CSV file into a DataFrame.
Polars expects CSV data to strictly conform to RFC 4180, unless documented
otherwise. Malformed data, though common, may lead to undefined behavior.
.. versionchanged:: 0.20.31
The `dtypes` parameter was renamed `schema_overrides`.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). If `fsspec` is installed, it might be used
to open remote files. Compressed files are supported when reading from a path.
For file-like objects, the stream position may not be updated accordingly after
reading.
has_header
Indicate if the first row of the dataset is a header or not. If set to False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
columns
Columns to select. Accepts a list of column indices (starting
at zero) or a list of column names.
new_columns
Rename columns right after parsing the CSV file. If the given
list is shorter than the width of the DataFrame the remaining
columns will have their original name.
separator
Single byte character to use as separator in the file.
comment_prefix
A string used to indicate the start of a comment line. Comment lines are skipped
during parsing. Common examples of comment prefixes are `#` and `//`.
quote_char
Single byte character used for csv quoting, default = `"`.
Set to None to turn off special handling and escaping of quotes.
skip_rows
Start reading after ``skip_rows`` rows. The header will be parsed at this
offset. Note that we respect CSV escaping/comments when skipping rows.
If you want to skip by newline char only, use `skip_lines`.
skip_lines
Start reading after `skip_lines` lines. The header will be parsed at this
offset. Note that CSV escaping will not be respected when skipping lines.
If you want to skip valid CSV rows, use ``skip_rows``.
schema
Provide the schema. This means that polars doesn't do schema inference.
This argument expects the complete schema, whereas `schema_overrides` can be
used to partially overwrite a schema. Note that the order of the columns in
the provided `schema` must match the order of the columns in the CSV being read.
schema_overrides
Overwrite dtypes for specific or all columns during schema inference.
null_values
Values to interpret as null values. You can provide a:
- `str`: All values equal to this string will be null.
- `List[str]`: All values equal to any string in this list will be null.
- `Dict[str, str]`: A dictionary that maps column name to a
null value string.
empty_string_is_null
By default a missing string value is considered to be null. If
`empty_string_is_null` is set to False, missing string values are considered to
decoded as empty strings.
ignore_errors
Try to keep reading lines if some lines yield errors.
Before using this option, try to increase the number of lines used for schema
inference with e.g `infer_schema_length=10000` or override automatic dtype
inference for specific columns with the `schema_overrides` option or use
`infer_schema=False` to read all columns as `pl.String` to check which
values might cause an issue.
try_parse_dates
Try to automatically parse dates. Most ISO8601-like formats can
be inferred, as well as a handful of others. If this does not succeed,
the column remains of data type `pl.String`.
If `use_pyarrow=True`, dates will always be parsed.
n_threads
Number of threads to use in csv parsing.
Defaults to the number of physical cpu's of your system.
infer_schema
When `True`, the schema is inferred from the data using the first
`infer_schema_length` rows.
When `False`, the schema is not inferred and will be `pl.String` if not
specified in `schema` or `schema_overrides`.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data will be scanned into memory
**(this is slow)**.
Alternatively set `infer_schema=False` to read all columns as
`pl.String`.
batch_size
Number of lines to read into the buffer at once.
Modify this to change performance.
n_rows
Stop reading from CSV file after reading `n_rows`.
During multi-threaded parsing, an upper bound of `n_rows`
rows cannot be guaranteed.
encoding : {'utf8', 'utf8-lossy', 'windows-1252', 'windows-1252-lossy', ...}
Lossy means that invalid utf8 values are replaced with `�`
characters. When using other encodings than `utf8` or
`utf8-lossy`, the input is first decoded in memory with
python. Defaults to `utf8`.
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
use_pyarrow
Try to use pyarrow's native CSV parser. This will always
parse dates, even if `try_parse_dates=False`.
This is not always possible. The set of arguments given to
this function determines if it is possible to use pyarrow's
native parser. Note that pyarrow and polars may have a
different strategy regarding type inference.
storage_options
Extra options that make sense for `fsspec.open()` or a
particular storage connection.
e.g. host, port, username, password, etc.
skip_rows_after_header
Skip this number of rows when the header is parsed.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
sample_size
Set the sample size. This is used to sample statistics to estimate the
allocation needed.
.. deprecated:: 1.10.0
This parameter is now a no-op.
eol_char
Single byte end of line character (default: ``\n``). When encountering a file
with windows line endings (``\r\n``), one can go with the default ``\n``. The
extra ``\r`` will be removed when processed.
raise_if_empty
When there is no data in the source, `NoDataError` is raised. If this parameter
is set to False, an empty DataFrame (with no columns) is returned instead.
truncate_ragged_lines
Truncate lines that are longer than the schema.
decimal_comma
Parse floats using a comma as the decimal separator instead of a period.
glob
Expand path given via globbing rules.
Returns
-------
DataFrame
See Also
--------
scan_csv : Lazily read from a CSV file or multiple files via glob patterns.
Warnings
--------
Calling `read_csv().lazy()` is an antipattern as this forces Polars to materialize
a full csv file and therefore cannot push any optimizations into the reader.
Therefore always prefer `scan_csv` if you want to work with `LazyFrame` s.
Notes
-----
If the schema is inferred incorrectly (e.g. as `pl.Int64` instead of `pl.Float64`),
try to increase the number of lines used to infer the schema with
`infer_schema_length` or override the inferred dtype for those columns with
`schema_overrides`.
Examples
--------
>>> pl.read_csv("data.csv", separator="|") # doctest: +SKIP
Demonstrate use against a BytesIO object, parsing string dates.
>>> from io import BytesIO
>>> data = BytesIO(
... b"ID,Name,Birthday\n"
... b"1,Alice,1995-07-12\n"
... b"2,Bob,1990-09-20\n"
... b"3,Charlie,2002-03-08\n"
... )
>>> pl.read_csv(data, try_parse_dates=True)
shape: (3, 3)
┌─────┬─────────┬────────────┐
│ ID ┆ Name ┆ Birthday │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ date │
╞═════╪═════════╪════════════╡
│ 1 ┆ Alice ┆ 1995-07-12 │
│ 2 ┆ Bob ┆ 1990-09-20 │
│ 3 ┆ Charlie ┆ 2002-03-08 │
└─────┴─────────┴────────────┘
read_csv_batched(source: 'str | Path', *, has_header: 'bool' = True, columns: 'Sequence[int] | Sequence[str] | None' = None, new_columns: 'Sequence[str] | None' = None, separator: 'str' = ',', comment_prefix: 'str | None' = None, quote_char: 'str | None' = '"', skip_rows: 'int' = 0, skip_lines: 'int' = 0, schema_overrides: 'Mapping[str, PolarsDataType] | Sequence[PolarsDataType] | None' = None, null_values: 'str | Sequence[str] | dict[str, str] | None' = None, empty_string_is_null: 'bool' = True, ignore_errors: 'bool' = False, try_parse_dates: 'bool' = False, n_threads: 'int | None' = None, infer_schema_length: 'int | None' = 100, batch_size: 'int' = 50000, n_rows: 'int | None' = None, encoding: 'CsvEncoding | str' = 'utf8', low_memory: 'bool' = False, rechunk: 'bool' = False, skip_rows_after_header: 'int' = 0, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, sample_size: 'int' = 1024, eol_char: 'str' = '\n', raise_if_empty: 'bool' = True, truncate_ragged_lines: 'bool' = False, decimal_comma: 'bool' = False) -> 'BatchedCsvReader'
Read a CSV file in batches.
Upon creation of the `BatchedCsvReader`, Polars will gather statistics and
determine the file chunks. After that, work will only be done if `next_batches`
is called, which will return a list of `n` frames of the given batch size.
.. versionchanged:: 0.20.31
The `dtypes` parameter was renamed `schema_overrides`.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
.. deprecated:: 1.37.0
Use `scan_csv().collect_batches()` instead.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). If `fsspec` is installed, it might be used
to open remote files. For file-like objects, the stream position may not be
updated accordingly after reading.
has_header
Indicate if the first row of the dataset is a header or not. If set to False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
columns
Columns to select. Accepts a list of column indices (starting
at zero) or a list of column names.
new_columns
Rename columns right after parsing the CSV file. If the given
list is shorter than the width of the DataFrame the remaining
columns will have their original name.
separator
Single byte character to use as separator in the file.
comment_prefix
A string used to indicate the start of a comment line. Comment lines are skipped
during parsing. Common examples of comment prefixes are `#` and `//`.
quote_char
Single byte character used for csv quoting, default = `"`.
Set to None to turn off special handling and escaping of quotes.
skip_rows
Start reading after ``skip_rows`` rows. The header will be parsed at this
offset. Note that we respect CSV escaping/comments when skipping rows.
If you want to skip by newline char only, use `skip_lines`.
skip_lines
Start reading after `skip_lines` lines. The header will be parsed at this
offset. Note that CSV escaping will not be respected when skipping lines.
If you want to skip valid CSV rows, use ``skip_rows``.
schema_overrides
Overwrite dtypes during inference.
null_values
Values to interpret as null values. You can provide a:
- `str`: All values equal to this string will be null.
- `List[str]`: All values equal to any string in this list will be null.
- `Dict[str, str]`: A dictionary that maps column name to a
null value string.
empty_string_is_null
By default a missing string value is considered to be null. If
`empty_string_is_null` is set to False, missing string values are considered to
decoded as empty strings.
ignore_errors
Try to keep reading lines if some lines yield errors.
First try `infer_schema_length=0` to read all columns as
`pl.String` to check which values might cause an issue.
try_parse_dates
Try to automatically parse dates. Most ISO8601-like formats can
be inferred, as well as a handful of others. If this does not succeed,
the column remains of data type `pl.String`.
n_threads
Number of threads to use in csv parsing.
Defaults to the number of physical cpu's of your system.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `0`, all columns will be read as `pl.String`.
If set to `None`, the full data may be scanned *(this is slow)*.
batch_size
Number of lines to read into the buffer at once.
Modify this to change performance.
n_rows
Stop reading from CSV file after reading `n_rows`.
During multi-threaded parsing, an upper bound of `n_rows`
rows cannot be guaranteed.
encoding : {'utf8', 'utf8-lossy', ...}
Lossy means that invalid utf8 values are replaced with `�`
characters. When using other encodings than `utf8` or
`utf8-lossy`, the input is first decoded in memory with
python. Defaults to `utf8`.
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
skip_rows_after_header
Skip this number of rows when the header is parsed.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
sample_size
Set the sample size. This is used to sample statistics to estimate the
allocation needed.
.. deprecated:: 1.10.0
Is a no-op.
eol_char
Single byte end of line character (default: ``\n``). When encountering a file
with windows line endings (``\r\n``), one can go with the default ``\n``. The
extra ``\r`` will be removed when processed.
raise_if_empty
When there is no data in the source,`NoDataError` is raised. If this parameter
is set to False, `None` will be returned from `next_batches(n)` instead.
truncate_ragged_lines
Truncate lines that are longer than the schema.
decimal_comma
Parse floats using a comma as the decimal separator instead of a period.
Returns
-------
BatchedCsvReader
See Also
--------
scan_csv : Lazily read from a CSV file or multiple files via glob patterns.
Examples
--------
>>> reader = pl.read_csv_batched(
... "./pdsh/tables_scale_100/lineitem.tbl",
... separator="|",
... try_parse_dates=True,
... ) # doctest: +SKIP
>>> batches = reader.next_batches(5) # doctest: +SKIP
>>> for df in batches: # doctest: +SKIP
... print(df)
Read big CSV file in batches and write a CSV file for each "group" of interest.
>>> seen_groups = set()
>>> reader = pl.read_csv_batched("big_file.csv") # doctest: +SKIP
>>> batches = reader.next_batches(100) # doctest: +SKIP
>>> while batches: # doctest: +SKIP
... df_current_batches = pl.concat(batches)
... partition_dfs = df_current_batches.partition_by("group", as_dict=True)
...
... for group, df in partition_dfs.items():
... if group in seen_groups:
... with open(f"./data/{group}.csv", "a") as fh:
... fh.write(df.write_csv(file=None, include_header=False))
... else:
... df.write_csv(file=f"./data/{group}.csv", include_header=True)
... seen_groups.add(group)
...
... batches = reader.next_batches(100)
read_database(query: 'str | TextClause | Selectable', connection: 'ConnectionOrCursor | str', *, iter_batches: 'bool' = False, batch_size: 'int | None' = None, schema_overrides: 'SchemaDict | None' = None, infer_schema_length: 'int | None' = 100, execute_options: 'dict[str, Any] | None' = None) -> 'DataFrame | Iterator[DataFrame]'
Read the results of a SQL query into a DataFrame, given a connection object.
Parameters
----------
query
SQL query to execute (if using a SQLAlchemy connection object this can
be a suitable "Selectable", otherwise it is expected to be a string).
connection
An instantiated connection (or cursor/client object) that the query can be
executed against. Can also pass a valid ODBC connection string (identified as
such if it contains the string "Driver={...}"), in which case the `arrow-odbc`
package will be used to establish the connection and return Arrow-native data
to Polars. Async driver connections are also supported, though this is currently
considered unstable. If using SQLAlchemy, you can configure the connection's
`execution_options` before passing to `read_database` to refine its behaviour
(see the `iter_batches` parameter for an example where this can be useful).
.. warning::
Use of asynchronous connections is currently considered **unstable**, and
unexpected issues may arise; if this happens, please report them.
iter_batches
Return an iterator of DataFrames, where each DataFrame represents a batch of
data returned by the query; this can be useful for processing large resultsets
in a more memory-efficient manner. If supported by the backend, this value is
passed to the underlying query execution method (note that lower values will
typically result in poor performance as they will cause many round-trips to
the database). If the backend does not support changing the batch size then
a single DataFrame is yielded from the iterator.
.. note::
If using SQLALchemy, you may also want to pass `stream_results=True` to the
connection's `execution_options` method when setting this parameter, which
will establish a server-side cursor; without this option some drivers (such
as "psycopg2") will still materialise the entire result set client-side
before batching the result locally.
batch_size
Indicate the size of each batch when `iter_batches` is True (note that you can
still set this when `iter_batches` is False, in which case the resulting
DataFrame is constructed internally using batched return before being returned
to you. Note that some backends (such as Snowflake) may support batch operation
but not allow for an explicit size to be set; in this case you will still
receive batches but their size is determined by the backend (in which case any
value set here will be ignored).
schema_overrides
A dictionary mapping column names to dtypes, used to override the schema
inferred from the query cursor or given by the incoming Arrow data (depending
on driver/backend). This can be useful if the given types can be more precisely
defined (for example, if you know that a given column can be declared as `u32`
instead of `i64`).
infer_schema_length
The maximum number of rows to scan for schema inference. If set to `None`, the
full data may be scanned *(this can be slow)*. This parameter only applies if
the data is read as a sequence of rows and the `schema_overrides` parameter
is not set for the given column; Arrow-aware drivers also ignore this value.
execute_options
These options will be passed through into the underlying query execution method
as kwargs. In the case of connections made using an ODBC string (which use
`arrow-odbc`) these options are passed to the `read_arrow_batches_from_odbc`
method.
Notes
-----
* This function supports a wide range of native database drivers (ranging from local
databases such as SQLite to large cloud databases such as Snowflake), as well as
generic libraries such as ADBC, SQLAlchemy and various flavours of ODBC. If the
backend supports returning Arrow data directly then this facility will be used to
efficiently instantiate the DataFrame; otherwise, the DataFrame is initialised
from row-wise data.
* Support for Arrow Flight SQL data is available via the `adbc-driver-flightsql`
package; see https://arrow.apache.org/adbc/current/driver/flight_sql.html for
more details about using this driver (notable databases implementing Flight SQL
include Dremio and InfluxDB).
* The `read_database_uri` function can be noticeably faster than `read_database`
if you are using a SQLAlchemy or DBAPI2 connection, as `connectorx` and `adbc`
optimise translation of the result set into Arrow format. Note that you can
determine a connection's URI from a SQLAlchemy engine object by calling
`conn.engine.url.render_as_string(hide_password=False)`.
* If Polars has to create a cursor from your connection in order to execute the
query then that cursor will be automatically closed when the query completes;
however, Polars will *never* close any other open connection or cursor.
* Polars is able to support more than just relational databases and SQL queries
through this function. For example, you can load local graph database results
from a `KùzuDB` connection in conjunction with a Cypher query, or use SurrealQL
with SurrealDB.
See Also
--------
read_database_uri : Create a DataFrame from a SQL query using a URI string.
Examples
--------
Instantiate a DataFrame from a SQL query against a user-supplied connection:
>>> df = pl.read_database(
... query="SELECT * FROM test_data",
... connection=user_conn,
... schema_overrides={"normalised_score": pl.UInt8},
... ) # doctest: +SKIP
Use a parameterised SQLAlchemy query, passing named values via `execute_options`:
>>> df = pl.read_database(
... query="SELECT * FROM test_data WHERE metric > :value",
... connection=alchemy_conn,
... execute_options={"parameters": {"value": 0}},
... ) # doctest: +SKIP
Use 'qmark' style parameterisation; values are still passed via `execute_options`,
but in this case the "parameters" value is a sequence of literals, not a dict:
>>> df = pl.read_database(
... query="SELECT * FROM test_data WHERE metric > ?",
... connection=alchemy_conn,
... execute_options={"parameters": [0]},
... ) # doctest: +SKIP
Batch the results of a large SQLAlchemy query into DataFrames, each containing
100,000 rows; explicitly establish a server-side cursor using the connection's
"execution_options" method to avoid loading the entire result locally before
batching (this is not required for all drivers, so check your driver's
documentation for more details):
>>> for df in pl.read_database(
... query="SELECT * FROM test_data",
... connection=alchemy_conn.execution_options(stream_results=True),
... iter_batches=True,
... batch_size=100_000,
... ):
... do_something(df) # doctest: +SKIP
Instantiate a DataFrame using an ODBC connection string (requires the `arrow-odbc`
package) setting upper limits on the buffer size of variadic text/binary columns:
>>> df = pl.read_database(
... query="SELECT * FROM test_data",
... connection="Driver={PostgreSQL};Server=localhost;Port=5432;Database=test;Uid=usr;Pwd=",
... execute_options={"max_text_size": 512, "max_binary_size": 1024},
... ) # doctest: +SKIP
Load data from an asynchronous SQLAlchemy driver/engine; note that asynchronous
connections and sessions are also supported here:
>>> from sqlalchemy.ext.asyncio import create_async_engine
>>> async_engine = create_async_engine("sqlite+aiosqlite:///test.db")
>>> df = pl.read_database(
... query="SELECT * FROM test_data",
... connection=async_engine,
... ) # doctest: +SKIP
Load data from an `AsyncSurrealDB` client connection object; note that both the "ws"
and "http" protocols are supported, as is the synchronous `SurrealDB` client. The
async loop can be run with standard `asyncio` or with `uvloop`:
>>> import asyncio # (or uvloop)
>>> async def surreal_query_to_frame(query: str, url: str):
... async with AsyncSurrealDB(url) as client:
... await client.use(namespace="test", database="test")
... return pl.read_database(query=query, connection=client)
>>> df = asyncio.run(
... surreal_query_to_frame(
... query="SELECT * FROM test",
... url="http://localhost:8000",
... )
... ) # doctest: +SKIP
read_database_uri(query: 'list[str] | str', uri: 'str', *, partition_on: 'str | None' = None, partition_range: 'tuple[int, int] | None' = None, partition_num: 'int | None' = None, protocol: 'str | None' = None, engine: 'DbReadEngine | None' = None, schema_overrides: 'SchemaDict | None' = None, execute_options: 'dict[str, Any] | None' = None, pre_execution_query: 'str | list[str] | None' = None) -> 'DataFrame'
Read the results of a SQL query into a DataFrame, given a URI.
Parameters
----------
query
Raw SQL query (or queries).
uri
A connectorx or ADBC connection URI string that starts with the backend's
driver name, for example:
* "postgresql://user:pass@server:port/database"
* "snowflake://user:pass@account/database/schema?warehouse=warehouse&role=role"
The caller is responsible for escaping any special characters in the string,
which will be passed "as-is" to the underlying engine (this is most often
required when coming across special characters in the password).
partition_on
The column on which to partition the result (connectorx).
partition_range
The value range of the partition column (connectorx).
partition_num
How many partitions to generate (connectorx).
protocol
Backend-specific transfer protocol directive (connectorx); see connectorx
documentation for more details.
engine : {'connectorx', 'adbc'}
Selects the engine used for reading the database (defaulting to connectorx):
* `'connectorx'`
Supports a range of databases, such as PostgreSQL, Redshift, MySQL, MariaDB,
Clickhouse, Oracle, BigQuery, SQL Server, and so on. For an up-to-date list
please see the connectorx docs:
https://github.com/sfu-db/connector-x#supported-sources--destinations
* `'adbc'`
Currently there is limited support for this engine, with a relatively small
number of drivers available, most of which are still in development. For
an up-to-date list of drivers please see the ADBC docs:
https://arrow.apache.org/adbc/
schema_overrides
A dictionary mapping column names to dtypes, used to override the schema
given in the data returned by the query.
execute_options
These options will be passed to the underlying query execution method as
kwargs. Note that connectorx does not support this parameter and ADBC currently
only supports positional 'qmark' style parameterization.
pre_execution_query
SQL query or list of SQL queries executed before main query (connectorx>=0.4.2).
Can be used to set runtime configurations using SET statements.
Only applicable for Postgres and MySQL source.
Only applicable with the connectorx engine.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Notes
-----
For `connectorx`, ensure that you have `connectorx>=0.3.2`. The documentation
is available `here <https://sfu-db.github.io/connector-x/intro.html>`_.
For `adbc` you will need to have installed the ADBC driver associated with the
backend you are connecting to, eg: `adbc-driver-postgresql`. For versions of
`adbc-driver-manager` < 1.7.0, `pyarrow` is also required.
If your password contains special characters, you will need to escape them.
This will usually require the use of a URL-escaping function, for example:
>>> from urllib.parse import quote, quote_plus
>>> quote_plus("pass word?")
'pass+word%3F'
>>> quote("pass word?")
'pass%20word%3F'
See Also
--------
read_database : Create a DataFrame from a SQL query using a connection object.
Examples
--------
Create a DataFrame from a SQL query using a single thread:
>>> uri = "postgresql://username:password@server:port/database"
>>> query = "SELECT * FROM lineitem"
>>> pl.read_database_uri(query, uri) # doctest: +SKIP
Create a DataFrame in parallel using 10 threads by automatically partitioning
the provided SQL on the partition column:
>>> uri = "postgresql://username:password@server:port/database"
>>> query = "SELECT * FROM lineitem"
>>> pl.read_database_uri(
... query,
... uri,
... partition_on="partition_col",
... partition_num=10,
... engine="connectorx",
... ) # doctest: +SKIP
Create a DataFrame in parallel using 2 threads by explicitly providing two
SQL queries:
>>> uri = "postgresql://username:password@server:port/database"
>>> queries = [
... "SELECT * FROM lineitem WHERE partition_col <= 10",
... "SELECT * FROM lineitem WHERE partition_col > 10",
... ]
>>> pl.read_database_uri(queries, uri, engine="connectorx") # doctest: +SKIP
Read data from Snowflake using the ADBC driver:
>>> df = pl.read_database_uri(
... "SELECT * FROM test_table",
... "snowflake://user:pass@company-org/testdb/public?warehouse=test&role=myrole",
... engine="adbc",
... ) # doctest: +SKIP
Pass a single parameter via `execute_options` into a query using the ADBC driver:
>>> df = pl.read_database_uri(
... "SELECT * FROM employees WHERE hourly_rate > ?",
... "sqlite:///:memory:",
... engine="adbc",
... execute_options={"parameters": (30,)},
... ) # doctest: +SKIP
Or pass multiple parameters:
>>> df = pl.read_database_uri(
... "SELECT * FROM employees WHERE hourly_rate BETWEEN ? AND ?",
... "sqlite:///:memory:",
... engine="adbc",
... execute_options={"parameters": (40, 20)},
... ) # doctest: +SKIP
read_delta(source: 'str | Path | DeltaTable', *, version: 'int | str | datetime | None' = None, columns: 'list[str] | None' = None, rechunk: 'bool | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', delta_table_options: 'dict[str, Any] | None' = None, use_pyarrow: 'bool' = False, pyarrow_options: 'dict[str, Any] | None' = None) -> 'DataFrame'
Reads into a DataFrame from a Delta lake table.
Parameters
----------
source
DeltaTable or a Path or URI to the root of the Delta lake table.
Note: For Local filesystem, absolute and relative paths are supported but
for the supported object storages - GCS, Azure and S3 full URI must be provided.
version
Numerical version or timestamp version of the Delta lake table.
Note: If `version` is not provided, the latest version of delta lake
table is read.
columns
Columns to select. Accepts a list of column names.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
storage_options
Extra options for the storage backends supported by `deltalake`.
For cloud storages, this may include configurations for authentication etc.
More info is available `here
<https://delta-io.github.io/delta-rs/usage/loading-table/>`__.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
delta_table_options
Additional keyword arguments while reading a Delta lake Table.
use_pyarrow
Flag to enable pyarrow dataset reads.
pyarrow_options
Keyword arguments while converting a Delta lake Table to pyarrow table.
Returns
-------
DataFrame
Examples
--------
Reads a Delta table from local filesystem.
Note: Since version is not provided, the latest version of the delta table is read.
>>> table_path = "/path/to/delta-table/"
>>> pl.read_delta(table_path) # doctest: +SKIP
Reads a specific version of the Delta table from local filesystem.
Note: This will fail if the provided version of the delta table does not exist.
>>> pl.read_delta(table_path, version=1) # doctest: +SKIP
Time travel a delta table from local filesystem using a timestamp version.
>>> pl.read_delta(
... table_path, version=datetime(2020, 1, 1, tzinfo=timezone.utc)
... ) # doctest: +SKIP
Reads a Delta table from AWS S3.
See a list of supported storage options for S3 `here
<https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html#variants>`__.
>>> table_path = "s3://bucket/path/to/delta-table/"
>>> storage_options = {
... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
... }
>>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP
Reads a Delta table from Google Cloud storage (GCS).
See a list of supported storage options for GCS `here
<https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html#variants>`__.
>>> table_path = "gs://bucket/path/to/delta-table/"
>>> storage_options = {"SERVICE_ACCOUNT": "SERVICE_ACCOUNT_JSON_ABSOLUTE_PATH"}
>>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP
Reads a Delta table from Azure.
Following type of table paths are supported,
* az://<container>/<path>
* adl://<container>/<path>
* abfs://<container>/<path>
See a list of supported storage options for Azure `here
<https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants>`__.
>>> table_path = "az://container/path/to/delta-table/"
>>> storage_options = {
... "AZURE_STORAGE_ACCOUNT_NAME": "AZURE_STORAGE_ACCOUNT_NAME",
... "AZURE_STORAGE_ACCOUNT_KEY": "AZURE_STORAGE_ACCOUNT_KEY",
... }
>>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP
Reads a Delta table with additional delta specific options. In the below example,
`without_files` option is used which loads the table without file tracking
information.
>>> table_path = "/path/to/delta-table/"
>>> delta_table_options = {"without_files": True}
>>> pl.read_delta(
... table_path, delta_table_options=delta_table_options
... ) # doctest: +SKIP
read_excel(source: 'FileSource | memoryview[int]', *, sheet_id: 'int | Sequence[int] | None' = None, sheet_name: 'str | list[str] | tuple[str, ...] | None' = None, table_name: 'str | None' = None, engine: 'ExcelSpreadsheetEngine' = 'calamine', engine_options: 'dict[str, Any] | None' = None, read_options: 'dict[str, Any] | None' = None, has_header: 'bool' = True, columns: 'Sequence[int] | Sequence[str] | str | None' = None, schema_overrides: 'SchemaDict | None' = None, infer_schema_length: 'int | None' = 100, include_file_paths: 'str | None' = None, drop_empty_rows: 'bool' = True, drop_empty_cols: 'bool' = True, raise_if_empty: 'bool' = True) -> 'pl.DataFrame | dict[str, pl.DataFrame]'
Read Excel spreadsheet data into a DataFrame.
.. versionadded:: 1.20
Support loading data from named table objects with `table_name` parameter.
.. versionadded:: 1.18
Support loading data from a list (or glob pattern) of multiple workbooks.
.. versionchanged:: 1.0
Default engine is now "calamine" (was "xlsx2csv").
.. versionchanged:: 0.20.7
The `read_csv_options` parameter was renamed `read_options`.
.. versionchanged:: 0.20.6
The `xlsx2csv_options` parameter was renamed `engine_options`.
Parameters
----------
source
Path(s) to a file or a file-like object (by "file-like object" we refer to
objects that have a `read()` method, such as a file handler like the builtin
`open` function, or a `BytesIO` instance). For file-like objects, the stream
position may not be updated after reading.
sheet_id
Sheet number(s) to convert (set `0` to load all sheets as DataFrames) and
return a `{sheetname:frame,}` dict. (Defaults to `1` if neither this nor
`sheet_name` are specified). Can also take a sequence of sheet numbers.
sheet_name
Sheet name(s) to convert; cannot be used in conjunction with `sheet_id`. If
more than one is given then a `{sheetname:frame,}` dict is returned.
table_name
Name of a specific table to read; note that table names are unique across
the workbook, so additionally specifying a sheet id or name is optional;
if one of those parameters *is* specified, an error will be raised if
the named table is not found in that particular sheet.
engine : {'calamine', 'openpyxl', 'xlsx2csv'}
Library used to parse the spreadsheet file; defaults to "calamine".
* "calamine": this engine can be used for reading all major types of Excel
Workbook (`.xlsx`, `.xlsb`, `.xls`) and is dramatically faster than the
other options, using the `fastexcel` module to bind the Rust-based Calamine
parser.
* "openpyxl": this engine is significantly slower than both `calamine` and
`xlsx2csv`, but can provide a useful fallback if you are otherwise unable
to read data from your workbook.
* "xlsx2csv": converts the data to an in-memory CSV before using the native
polars `read_csv` method to parse the result.
engine_options
Additional options passed to the underlying engine's primary parsing
constructor (given below), if supported:
* "calamine": n/a (can only provide `read_options`)
* "openpyxl": `load_workbook <https://openpyxl.readthedocs.io/en/stable/api/openpyxl.reader.excel.html#openpyxl.reader.excel.load_workbook>`_
* "xlsx2csv": `Xlsx2csv <https://github.com/dilshod/xlsx2csv/blob/f35734aa453d65102198a77e7b8cd04928e6b3a2/xlsx2csv.py#L157>`_
read_options
Options passed to the underlying engine method that reads the sheet data.
Where supported, this allows for additional control over parsing. The
specific read methods associated with each engine are:
* "calamine": `load_sheet_by_name <https://fastexcel.toucantoco.dev/fastexcel.html#ExcelReader.load_sheet_by_name>`_
(or `load_table <https://fastexcel.toucantoco.dev/fastexcel.html#ExcelReader.load_table>`_
if using the `table_name` parameter).
* "openpyxl": n/a (can only provide `engine_options`)
* "xlsx2csv": see :meth:`read_csv`
has_header
Indicate if the first row of the table data is a header or not. If False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
columns
Columns to read from the sheet; if not specified, all columns are read. Can
be given as a sequence of column names or indices, or a single column name.
schema_overrides
Support type specification or override of one or more columns.
infer_schema_length
The maximum number of rows to scan for schema inference. If set to `None`, the
entire dataset is scanned to determine the dtypes, which can slow parsing for
large workbooks. Note that only the "calamine" and "xlsx2csv" engines support
this parameter.
include_file_paths
Include the path of the source file(s) as a column with this name.
drop_empty_rows
Indicate whether to omit empty rows when reading data into the DataFrame.
drop_empty_cols
Indicate whether to omit empty columns (with no headers) when reading data into
the DataFrame (note that empty column identification may vary depending on the
underlying engine being used).
raise_if_empty
When there is no data in the sheet,`NoDataError` is raised. If this parameter
is set to False, an empty DataFrame (with no columns) is returned instead.
Returns
-------
DataFrame
If reading a single sheet.
dict
If reading multiple sheets, a "{sheetname: DataFrame, ...}" dict is returned.
See Also
--------
read_ods
Notes
-----
* Where possible, prefer the default "calamine" engine for reading Excel Workbooks,
as it is significantly faster than the other options.
* When using the `xlsx2csv` engine the target Excel sheet is first converted
to CSV using `xlsx2csv.Xlsx2csv(source).convert()` and then parsed with Polars'
:func:`read_csv` function. You can pass additional options to `read_options`
to influence this part of the parsing pipeline.
* If you want to read multiple sheets and set *different* options (`read_options`,
`schema_overrides`, etc), you should make separate calls as the options are set
globally, not on a per-sheet basis.
Examples
--------
Read the "data" worksheet from an Excel file into a DataFrame.
>>> pl.read_excel(
... source="test.xlsx",
... sheet_name="data",
... ) # doctest: +SKIP
If the correct dtypes can't be determined, use the `schema_overrides` parameter
to specify them, or increase the inference length with `infer_schema_length`.
>>> pl.read_excel(
... source="test.xlsx",
... schema_overrides={"dt": pl.Date},
... infer_schema_length=None,
... ) # doctest: +SKIP
Using the `xlsx2csv` engine, read table data from sheet 3 in an Excel workbook as a
DataFrame while skipping empty lines in the sheet. As sheet 3 does not have a header
row, you can pass the necessary additional settings for this to the `read_options`
parameter; these will be passed to :func:`read_csv`.
>>> pl.read_excel(
... source="test.xlsx",
... sheet_id=3,
... engine="xlsx2csv",
... engine_options={"skip_empty_lines": True},
... read_options={"has_header": False, "new_columns": ["a", "b", "c"]},
... ) # doctest: +SKIP
read_ipc(source: 'str | Path | IO[bytes] | bytes', *, columns: 'list[int] | list[str] | None' = None, n_rows: 'int | None' = None, use_pyarrow: 'bool' = False, memory_map: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, rechunk: 'bool' = True) -> 'DataFrame'
Read into a DataFrame from Arrow IPC (Feather v2) file.
See "File or Random Access format" on https://arrow.apache.org/docs/python/ipc.html.
Arrow IPC files are also known as Feather (v2) files.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). If `fsspec` is installed, it might be used
to open remote files. For file-like objects, the stream position may not be
updated accordingly after reading.
columns
Columns to select. Accepts a list of column indices (starting at zero) or a list
of column names.
n_rows
Stop reading from IPC file after reading `n_rows`.
Only valid when `use_pyarrow=False`.
use_pyarrow
Use pyarrow or the native Rust reader.
memory_map
Try to memory map the file. This can greatly improve performance on repeated
queries as the OS may cache pages.
Only uncompressed IPC files can be memory mapped.
storage_options
Extra options that make sense for `fsspec.open()` or a particular storage
connection, e.g. host, port, username, password, etc.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
rechunk
Make sure that all data is contiguous.
Returns
-------
DataFrame
See Also
--------
scan_ipc : Lazily read from an IPC file or multiple files via glob patterns.
Warnings
--------
Calling `read_ipc().lazy()` is an antipattern as this forces Polars to materialize
a full csv file and therefore cannot push any optimizations into the reader.
Therefore always prefer `scan_ipc` if you want to work with `LazyFrame` s.
If `memory_map` is set, the bytes on disk are mapped 1:1 to memory.
That means that:
- Arrow data in the file is not validated to be correct and invalid arrow
data is UB! Ensure this file is correct or set `memory_map=False`.
- You cannot write to the same filename.
E.g. `pl.read_ipc("my_file.arrow").write_ipc("my_file.arrow")`
will fail.
read_ipc_schema(source: 'str | Path | IO[bytes] | bytes') -> 'dict[str, DataType]'
Get the schema of an IPC file without reading data.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
Returns
-------
dict
Dictionary mapping column names to datatypes
read_ipc_stream(source: 'str | Path | IO[bytes] | bytes', *, columns: 'list[int] | list[str] | None' = None, n_rows: 'int | None' = None, use_pyarrow: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, rechunk: 'bool' = True) -> 'DataFrame'
Read into a DataFrame from Arrow IPC record batch stream.
See "Streaming format" on https://arrow.apache.org/docs/python/ipc.html.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). If `fsspec` is installed, it might be used
to open remote files. For file-like objects, the stream position may not be
updated accordingly after reading.
columns
Columns to select. Accepts a list of column indices (starting at zero) or a list
of column names.
n_rows
Stop reading from IPC stream after reading `n_rows`.
Only valid when `use_pyarrow=False`.
use_pyarrow
Use pyarrow or the native Rust reader.
storage_options
Extra options that make sense for `fsspec.open()` or a particular storage
connection, e.g. host, port, username, password, etc.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
rechunk
Make sure that all data is contiguous.
Returns
-------
DataFrame
read_json(source: 'str | Path | IOBase | bytes', *, schema: 'SchemaDefinition | None' = None, schema_overrides: 'SchemaDefinition | None' = None, infer_schema_length: 'int | None' = 100) -> 'DataFrame'
Read into a DataFrame from a JSON file.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the schema param will be overridden.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
See Also
--------
read_ndjson
Examples
--------
>>> from io import StringIO
>>> json_str = '[{"foo":1,"bar":6},{"foo":2,"bar":7},{"foo":3,"bar":8}]'
>>> pl.read_json(StringIO(json_str))
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 6 │
│ 2 ┆ 7 │
│ 3 ┆ 8 │
└─────┴─────┘
With the schema defined.
>>> pl.read_json(StringIO(json_str), schema={"foo": pl.Int64, "bar": pl.Float64})
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 1 ┆ 6.0 │
│ 2 ┆ 7.0 │
│ 3 ┆ 8.0 │
└─────┴─────┘
read_lines(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]]', *, name: 'str' = 'line', n_rows: 'int | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, glob: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', include_file_paths: 'str | None' = None) -> 'DataFrame'
Read lines into a string column from a file.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
name
Name to use for the output column.
n_rows
Stop reading from parquet file after reading `n_rows`.
row_index_name
If not None, this will insert a row index column with the given name into the
DataFrame
row_index_offset
Offset to start the row index column (only used if the name is set)
glob
Expand path given via globbing rules.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \
`{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
include_file_paths
Include the path of the source file(s) as a column with this name.
See Also
--------
scan_lines
Examples
--------
>>> pl.read_lines(b"Hello\nworld")
shape: (2, 1)
┌───────┐
│ line │
│ --- │
│ str │
╞═══════╡
│ Hello │
│ world │
└───────┘
read_ndjson(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]]', *, schema: 'SchemaDefinition | None' = None, schema_overrides: 'SchemaDefinition | None' = None, infer_schema_length: 'int | None' = 100, batch_size: 'int | None' = 1024, n_rows: 'int | None' = None, low_memory: 'bool' = False, rechunk: 'bool' = False, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, ignore_errors: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, file_cache_ttl: 'int | None' = None, include_file_paths: 'str | None' = None) -> 'DataFrame'
Read into a DataFrame from a newline delimited JSON file.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the schema param will be overridden.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
batch_size
Number of rows to read in each batch.
n_rows
Stop reading from JSON file after reading `n_rows`.
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Reallocate to contiguous memory when all chunks/ files are parsed.
row_index_name
If not None, this will insert a row index column with give name into the
DataFrame
row_index_offset
Offset to start the row index column (only use if the name is set)
ignore_errors
Return `Null` if parsing fails because of schema mismatches.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \
`{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
file_cache_ttl
Amount of time to keep downloaded cloud files since their last access time,
in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable
(which defaults to 1 hour) if not given.
.. deprecated:: 1.39.0
File cache is no longer supported.
include_file_paths
Include the path of the source file(s) as a column with this name.
See Also
--------
scan_ndjson : Lazily read from an NDJSON file or multiple files via glob patterns.
Warnings
--------
Calling `read_ndjson().lazy()` is an antipattern as this forces Polars to
materialize a full ndjson file and therefore cannot push any optimizations into
the reader. Therefore always prefer `scan_ndjson` if you want to work with
`LazyFrame` s.
Examples
--------
>>> from io import StringIO
>>> json_str = '{"foo":1,"bar":6}\n{"foo":2,"bar":7}\n{"foo":3,"bar":8}\n'
>>> pl.read_ndjson(StringIO(json_str))
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 6 │
│ 2 ┆ 7 │
│ 3 ┆ 8 │
└─────┴─────┘
read_ods(source: 'FileSource', *, sheet_id: 'int | Sequence[int] | None' = None, sheet_name: 'str | list[str] | tuple[str, ...] | None' = None, has_header: 'bool' = True, columns: 'Sequence[int] | Sequence[str] | None' = None, schema_overrides: 'SchemaDict | None' = None, infer_schema_length: 'int | None' = 100, include_file_paths: 'str | None' = None, drop_empty_rows: 'bool' = True, drop_empty_cols: 'bool' = True, raise_if_empty: 'bool' = True) -> 'pl.DataFrame | dict[str, pl.DataFrame]'
Read OpenOffice (ODS) spreadsheet data into a DataFrame.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
sheet_id
Sheet number(s) to convert, starting from 1 (set `0` to load *all* worksheets
as DataFrames) and return a `{sheetname:frame,}` dict. (Defaults to `1` if
neither this nor `sheet_name` are specified). Can also take a sequence of sheet
numbers.
sheet_name
Sheet name(s) to convert; cannot be used in conjunction with `sheet_id`. If
more than one is given then a `{sheetname:frame,}` dict is returned.
has_header
Indicate if the first row of the table data is a header or not. If False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
columns
Columns to read from the sheet; if not specified, all columns are read. Can
be given as a sequence of column names or indices.
schema_overrides
Support type specification or override of one or more columns.
infer_schema_length
The maximum number of rows to scan for schema inference. If set to `None`, the
entire dataset is scanned to determine the dtypes, which can slow parsing for
large workbooks.
include_file_paths
Include the path of the source file(s) as a column with this name.
drop_empty_rows
Indicate whether to omit empty rows when reading data into the DataFrame.
drop_empty_cols
Indicate whether to omit empty columns (with no headers) when reading data into
the DataFrame (note that empty column identification may vary depending on the
underlying engine being used).
raise_if_empty
When there is no data in the sheet,`NoDataError` is raised. If this parameter
is set to False, an empty DataFrame (with no columns) is returned instead.
Returns
-------
DataFrame, or a `{sheetname: DataFrame, ...}` dict if reading multiple sheets.
See Also
--------
read_excel
Examples
--------
Read the "data" worksheet from an OpenOffice spreadsheet file into a DataFrame.
>>> pl.read_ods(
... source="test.ods",
... sheet_name="data",
... ) # doctest: +SKIP
If the correct dtypes can't be determined, use the `schema_overrides` parameter
to specify them, or increase the inference length with `infer_schema_length`.
>>> pl.read_ods(
... source="test.ods",
... sheet_id=3,
... schema_overrides={"dt": pl.Date},
... raise_if_empty=False,
... ) # doctest: +SKIP
read_parquet(source: 'FileSource', *, columns: 'list[int] | list[str] | None' = None, n_rows: 'int | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, parallel: 'ParallelStrategy' = 'auto', use_statistics: 'bool' = True, hive_partitioning: 'bool | None' = None, glob: 'bool' = True, schema: 'SchemaDict | None' = None, hive_schema: 'SchemaDict | None' = None, try_parse_hive_dates: 'bool' = True, rechunk: 'bool' = False, low_memory: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, use_pyarrow: 'bool' = False, pyarrow_options: 'dict[str, Any] | None' = None, memory_map: 'bool' = True, include_file_paths: 'str | None' = None, missing_columns: "Literal['insert', 'raise']" = 'raise', allow_missing_columns: 'bool | None' = None) -> 'DataFrame'
Read into a DataFrame from a parquet file.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
File-like objects are supported (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
columns
Columns to select. Accepts a list of column indices (starting at zero) or a list
of column names.
n_rows
Stop reading from parquet file after reading `n_rows`.
Only valid when `use_pyarrow=False`.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
parallel : {'auto', 'columns', 'row_groups', 'none'}
This determines the direction of parallelism. 'auto' will try to determine the
optimal direction.
use_statistics
Use statistics in the parquet to determine if pages
can be skipped from reading.
hive_partitioning
Infer statistics and schema from Hive partitioned URL and use them
to prune reads. This is unset by default (i.e. `None`), meaning it is
automatically enabled when a single directory is passed, and otherwise
disabled.
glob
Expand path given via globbing rules.
schema
Specify the datatypes of the columns. The datatypes must match the
datatypes in the file(s). If there are extra columns that are not in the
file(s), consider also passing `missing_columns='insert'`.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
hive_schema
The column names and data types of the columns by which the data is partitioned.
If set to `None` (default), the schema of the Hive partitions is inferred.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
try_parse_hive_dates
Whether to try parsing hive values as date/datetime types.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
low_memory
Reduce memory pressure at the expense of performance.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
use_pyarrow
Use PyArrow instead of the Rust-native Parquet reader. The PyArrow reader is
more stable.
pyarrow_options
Keyword arguments for `pyarrow.parquet.read_table
<https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_table.html>`_.
memory_map
Memory map underlying file. This will likely increase performance.
Only used when `use_pyarrow=True`.
include_file_paths
Include the path of the source file(s) as a column with this name.
Only valid when `use_pyarrow=False`.
missing_columns
Configuration for behavior when columns defined in the schema
are missing from the data:
* `insert`: Inserts the missing columns using NULLs as the row values.
* `raise`: Raises an error.
allow_missing_columns
When reading a list of parquet files, if a column existing in the first
file cannot be found in subsequent files, the default behavior is to
raise an error. However, if `allow_missing_columns` is set to
`True`, a full-NULL column is returned instead of erroring for the files
that do not contain the column.
.. deprecated:: 1.30.0
Use the parameter `missing_columns` instead and pass one of
`('insert', 'raise')`.
Returns
-------
DataFrame
See Also
--------
scan_parquet: Lazily read from a parquet file or multiple files via glob patterns.
scan_pyarrow_dataset
Warnings
--------
Calling `read_parquet().lazy()` is an antipattern as this forces Polars to
materialize a full parquet file and therefore cannot push any optimizations
into the reader. Therefore always prefer `scan_parquet` if you want to work
with `LazyFrame` s.
read_parquet_metadata(source: 'str | Path | IO[bytes] | bytes', storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None) -> 'dict[str, str]'
Get file-level custom metadata of a Parquet file without reading data.
.. warning::
This functionality is considered **experimental**. It may be removed or
changed at any point without it being considered a breaking change.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
Returns
-------
dict
Dictionary with the metadata. Empty if no custom metadata is available.
read_parquet_schema(source: 'str | Path | IO[bytes] | bytes') -> 'dict[str, DataType]'
Get the schema of a Parquet file without reading data.
If you would like to read the schema of a cloud file with authentication
configuration, it is recommended use `scan_parquet` - e.g.
`scan_parquet(..., storage_options=...).collect_schema()`.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
Returns
-------
dict
Dictionary mapping column names to datatypes
See Also
--------
scan_parquet
scan_arrow_c_stream(source: 'Any') -> 'LazyFrame'
Scan a source that implements the Arrow PyCapsule Interface.
This creates a lazy scan over an object that exposes a `__arrow_c_stream__`
method (e.g. a `pyarrow.RecordBatchReader`, or a `nanoarrow` stream). The
source is consumed, batch by batch, during query execution.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
source
Any object implementing the Arrow PyCapsule Interface
(i.e. it has a `__arrow_c_stream__` method).
Returns
-------
LazyFrame
Notes
-----
- The source must produce Arrow RecordBatches with a struct-typed schema.
See: https://arrow.apache.org/docs/format/CStreamInterface.html.
- Once the stream is consumed, the same source cannot be scanned again.
Examples
--------
Scan from a PyArrow RecordBatchReader:
>>> import pyarrow as pa
>>> schema = pa.schema([("a", pa.int64()), ("b", pa.string())])
>>> batches = [
... pa.record_batch([[1, 2, 3], ["x", "y", "z"]], schema=schema),
... pa.record_batch([[4, 5], ["a", "b"]], schema=schema),
... ]
>>> reader = pa.RecordBatchReader.from_batches(schema, batches)
>>> pl.scan_arrow_c_stream(reader).collect()
shape: (5, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1 ┆ x │
│ 2 ┆ y │
│ 3 ┆ z │
│ 4 ┆ a │
│ 5 ┆ b │
└─────┴─────┘
scan_csv(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]] | list[bytes]', *, has_header: 'bool' = True, separator: 'str' = ',', comment_prefix: 'str | None' = None, quote_char: 'str | None' = '"', skip_rows: 'int' = 0, skip_lines: 'int' = 0, schema: 'SchemaDict | None' = None, schema_overrides: 'SchemaDict | Sequence[PolarsDataType] | None' = None, null_values: 'str | Sequence[str] | dict[str, str] | None' = None, empty_string_is_null: 'bool' = True, ignore_errors: 'bool' = False, cache: 'bool | None' = None, with_column_names: 'Callable[[list[str]], list[str]] | None' = None, infer_schema: 'bool' = True, infer_schema_length: 'int | None' = 100, n_rows: 'int | None' = None, encoding: 'CsvEncoding' = 'utf8', low_memory: 'bool' = False, rechunk: 'bool' = False, skip_rows_after_header: 'int' = 0, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, try_parse_dates: 'bool' = False, eol_char: 'str' = '\n', new_columns: 'Sequence[str] | None' = None, raise_if_empty: 'bool' = True, truncate_ragged_lines: 'bool' = False, decimal_comma: 'bool' = False, glob: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, file_cache_ttl: 'int | None' = None, include_file_paths: 'str | None' = None, missing_columns: "Literal['insert', 'raise'] | None" = None) -> 'LazyFrame'
Lazily read from a CSV file or multiple files via glob patterns.
This allows the query optimizer to push down predicates and
projections to the scan level, thereby potentially reducing
memory overhead.
.. versionchanged:: 0.20.31
The `dtypes` parameter was renamed `schema_overrides`.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
has_header
Indicate if the first row of the dataset is a header or not. If set to False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
separator
Single byte character to use as separator in the file.
comment_prefix
A string used to indicate the start of a comment line. Comment lines are skipped
during parsing. Common examples of comment prefixes are `#` and `//`.
quote_char
Single byte character used for csv quoting, default = `"`.
Set to None to turn off special handling and escaping of quotes.
skip_rows
Start reading after ``skip_rows`` rows. The header will be parsed at this
offset. Note that we respect CSV escaping/comments when skipping rows.
If you want to skip by newline char only, use `skip_lines`.
skip_lines
Start reading after `skip_lines` lines. The header will be parsed at this
offset. Note that CSV escaping will not be respected when skipping lines.
If you want to skip valid CSV rows, use ``skip_rows``.
schema
Provide the schema. This means that polars doesn't do schema inference.
This argument expects the complete schema, whereas `schema_overrides` can be
used to partially overwrite a schema. Note that the order of the columns in
the provided `schema` must match the order of the columns in the CSV being read.
schema_overrides
Overwrite dtypes during inference; should be a {colname:dtype,} dict or,
if providing a list of strings to `new_columns`, a list of dtypes of
the same length.
null_values
Values to interpret as null values. You can provide a:
- `str`: All values equal to this string will be null.
- `List[str]`: All values equal to any string in this list will be null.
- `Dict[str, str]`: A dictionary that maps column name to a
null value string.
empty_string_is_null
By default a missing string value is considered to be null. If
`empty_string_is_null` is set to False, missing string values are considered to
decoded as empty strings.
ignore_errors
Try to keep reading lines if some lines yield errors.
First try `infer_schema=False` to read all columns as
`pl.String` to check which values might cause an issue.
cache
Cache the result after reading.
.. deprecated:: 1.39.0
File cache is no longer supported.
with_column_names
Apply a function over the column names just in time (when they are determined);
this function will receive (and should return) a list of column names.
infer_schema
When `True`, the schema is inferred from the data using the first
`infer_schema_length` rows.
When `False`, the schema is not inferred and will be `pl.String` if not
specified in `schema` or `schema_overrides`.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data will be scanned into memory
**(this is slow)**.
Alternatively set `infer_schema=False` to read all columns as
`pl.String`.
n_rows
Stop reading from CSV file after reading `n_rows`.
encoding : {'utf8', 'utf8-lossy'}
Lossy means that invalid utf8 values are replaced with `�`
characters. Defaults to "utf8".
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Reallocate to contiguous memory when all chunks/ files are parsed.
skip_rows_after_header
Skip this number of rows when the header is parsed.
row_index_name
If not None, this will insert a row index column with the given name into
the DataFrame.
row_index_offset
Offset to start the row index column (only used if the name is set).
try_parse_dates
Try to automatically parse dates. Most ISO8601-like formats
can be inferred, as well as a handful of others. If this does not succeed,
the column remains of data type `pl.String`.
eol_char
Single byte end of line character (default: ``\n``). When encountering a file
with windows line endings (``\r\n``), one can go with the default ``\n``. The
extra ``\r`` will be removed when processed.
new_columns
Provide an explicit list of string column names to use (for example, when
scanning a headerless CSV file). If the given list is shorter than the width of
the DataFrame the remaining columns will have their original name.
raise_if_empty
When there is no data in the source, `NoDataError` is raised. If this parameter
is set to False, an empty LazyFrame (with no columns) is returned instead.
truncate_ragged_lines
Truncate lines that are longer than the schema.
decimal_comma
Parse floats using a comma as the decimal separator instead of a period.
glob
Expand path given via globbing rules.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \
`{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
file_cache_ttl
Amount of time to keep downloaded cloud files since their last access time,
in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable
(which defaults to 1 hour) if not given.
.. deprecated:: 1.39.0
File cache is no longer supported.
include_file_paths
Include the path of the source file(s) as a column with this name.
missing_columns
Configuration for behavior when columns defined in the schema are
missing from the data:
* ``"insert"``: Insert the missing columns with NULL values.
* ``"raise"``: Raise an error.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Returns
-------
LazyFrame
See Also
--------
read_csv : Read a CSV file into a DataFrame.
Examples
--------
>>> import pathlib
>>>
>>> (
... pl.scan_csv("my_long_file.csv") # lazy, doesn't do a thing
... .select(
... ["a", "c"]
... ) # select only 2 columns (other columns will not be read)
... .filter(
... pl.col("a") > 10
... ) # the filter is pushed down the scan, so less data is read into memory
... .head(100) # constrain number of returned results to 100
... ) # doctest: +SKIP
We can use `with_column_names` to modify the header before scanning:
>>> df = pl.DataFrame(
... {"BrEeZaH": [1, 2, 3, 4], "LaNgUaGe": ["is", "hard", "to", "read"]}
... )
>>> path: pathlib.Path = dirpath / "mydf.csv"
>>> df.write_csv(path)
>>> pl.scan_csv(
... path, with_column_names=lambda cols: [col.lower() for col in cols]
... ).collect()
shape: (4, 2)
┌─────────┬──────────┐
│ breezah ┆ language │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════════╪══════════╡
│ 1 ┆ is │
│ 2 ┆ hard │
│ 3 ┆ to │
│ 4 ┆ read │
└─────────┴──────────┘
You can also simply replace column names (or provide them if the file has none)
by passing a list of new column names to the `new_columns` parameter:
>>> df.write_csv(path)
>>> pl.scan_csv(
... path,
... new_columns=["idx", "txt"],
... schema_overrides=[pl.UInt16, pl.String],
... ).collect()
shape: (4, 2)
┌─────┬──────┐
│ idx ┆ txt │
│ --- ┆ --- │
│ u16 ┆ str │
╞═════╪══════╡
│ 1 ┆ is │
│ 2 ┆ hard │
│ 3 ┆ to │
│ 4 ┆ read │
└─────┴──────┘
scan_delta(source: 'str | Path | DeltaTable', *, version: 'int | str | datetime | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', delta_table_options: 'dict[str, Any] | None' = None, use_pyarrow: 'bool' = False, pyarrow_options: 'dict[str, Any] | None' = None, rechunk: 'bool | None' = None) -> 'LazyFrame'
Lazily read from a Delta lake table.
Parameters
----------
source
DeltaTable or a Path or URI to the root of the Delta lake table.
Note: For Local filesystem, absolute and relative paths are supported but
for the supported object storages - GCS, Azure and S3 full URI must be provided.
version
Numerical version or timestamp version of the Delta lake table.
Note: If `version` is not provided, the latest version of delta lake
table is read.
storage_options
Extra options for the storage backends supported by `deltalake`.
For cloud storages, this may include configurations for authentication etc.
More info is available `here
<https://delta-io.github.io/delta-rs/usage/loading-table/>`__.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
delta_table_options
Additional keyword arguments while reading a Delta lake Table.
use_pyarrow
Flag to enable pyarrow dataset reads.
pyarrow_options
Keyword arguments while converting a Delta lake Table to pyarrow table.
Use this parameter when filtering on partitioned columns or to read
from a 'fsspec' supported filesystem.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
Returns
-------
LazyFrame
Examples
--------
Creates a scan for a Delta table from local filesystem.
Note: Since version is not provided, the latest version of the delta table is read.
>>> table_path = "/path/to/delta-table/"
>>> pl.scan_delta(table_path).collect() # doctest: +SKIP
Creates a scan for a specific version of the Delta table from local filesystem.
Note: This will fail if the provided version of the delta table does not exist.
>>> pl.scan_delta(table_path, version=1).collect() # doctest: +SKIP
Time travel a delta table from local filesystem using a timestamp version.
>>> pl.scan_delta(
... table_path, version=datetime(2020, 1, 1, tzinfo=timezone.utc)
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table from AWS S3.
See a list of supported storage options for S3 `here
<https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html#variants>`__.
>>> table_path = "s3://bucket/path/to/delta-table/"
>>> storage_options = {
... "AWS_REGION": "eu-central-1",
... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
... }
>>> pl.scan_delta(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table from Google Cloud storage (GCS).
See a list of supported storage options for GCS `here
<https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html#variants>`__.
>>> table_path = "gs://bucket/path/to/delta-table/"
>>> storage_options = {"SERVICE_ACCOUNT": "SERVICE_ACCOUNT_JSON_ABSOLUTE_PATH"}
>>> pl.scan_delta(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table from Azure.
Supported options for Azure are available `here
<https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants>`__.
Following type of table paths are supported,
* az://<container>/<path>
* adl://<container>/<path>
* abfs[s]://<container>/<path>
>>> table_path = "az://container/path/to/delta-table/"
>>> storage_options = {
... "AZURE_STORAGE_ACCOUNT_NAME": "AZURE_STORAGE_ACCOUNT_NAME",
... "AZURE_STORAGE_ACCOUNT_KEY": "AZURE_STORAGE_ACCOUNT_KEY",
... }
>>> pl.scan_delta(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table with additional delta specific options.
In the below example, `without_files` option is used which loads the table without
file tracking information.
>>> table_path = "/path/to/delta-table/"
>>> delta_table_options = {"without_files": True}
>>> pl.scan_delta(
... table_path, delta_table_options=delta_table_options
... ).collect() # doctest: +SKIP
scan_iceberg(source: 'str | pyiceberg.table.Table', *, snapshot_id: 'int | None' = None, storage_options: 'StorageOptionsDict | None' = None, catalog: 'pyiceberg.catalog.Catalog | polars.io.iceberg.IcebergCatalogConfig | None' = None, reader_override: "Literal['native', 'pyiceberg'] | None" = None, use_metadata_statistics: 'bool' = True, fast_deletion_count: 'bool | None' = None, use_pyiceberg_filter: 'bool' = True) -> 'LazyFrame'
Lazily read from an Apache Iceberg table.
Parameters
----------
source
A PyIceberg table, or a 'namespace.table_name' identifier string,
or an absolute path to the metadata.
snapshot_id
The snapshot ID to scan from.
storage_options
Extra options for the storage backends supported by `pyiceberg`.
For cloud storages, this may include configurations for authentication etc.
More info is available `here <https://py.iceberg.apache.org/configuration/>`__.
catalog
PyIceberg catalog to load the table from if the provided `target`
was a table name.
reader_override
Overrides the reader used to read the data.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Note that this parameter should not be necessary outside of testing, as
polars will by default automatically select the best reader.
Available options:
* native: Uses polars native reader. This allows for more optimizations to
improve performance.
* pyiceberg: Uses PyIceberg, which may support more features.
use_metadata_statistics
Whether to allow using statistics from Iceberg metadata files.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
When a filter is present, this allows using min/max statistics present
in the Iceberg metadata files can be used to allow the reader to skip
scanning of metadata from data files that are guaranteed to not match
the filter.
If a row-count is requested (i.e. `scan_iceberg().select(pl.len())`), this
allows returning a count directly from Iceberg metadata. Note however that
for datasets containing position delete files, `fast_deletion_count` must
also be enabled for this to work.
fast_deletion_count
Allows returning a row count calculated directly from Iceberg metadata
for datasets that contain position delete files. This will give incorrect
results if position delete files contain duplicated entries.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
use_pyiceberg_filter
Convert and push the filter to PyIceberg where possible.
Returns
-------
LazyFrame
Examples
--------
Creates a scan for an Iceberg table from local filesystem, or object store.
>>> table_path = "file:/path/to/iceberg-table/metadata.json"
>>> pl.scan_iceberg(table_path).collect() # doctest: +SKIP
Creates a scan for an Iceberg table from S3.
See a list of supported storage options for S3 `here
<https://py.iceberg.apache.org/configuration/#fileio>`__.
>>> table_path = "s3://bucket/path/to/iceberg-table/metadata.json"
>>> storage_options = {
... "s3.region": "eu-central-1",
... "s3.access-key-id": "THE_AWS_ACCESS_KEY_ID",
... "s3.secret-access-key": "THE_AWS_SECRET_ACCESS_KEY",
... }
>>> pl.scan_iceberg(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for an Iceberg table from Azure.
Supported options for Azure are available `here
<https://py.iceberg.apache.org/configuration/#azure-data-lake>`__.
Following type of table paths are supported:
* az://<container>/<path>/metadata.json
* adl://<container>/<path>/metadata.json
* abfs[s]://<container>/<path>/metadata.json
>>> table_path = "az://container/path/to/iceberg-table/metadata.json"
>>> storage_options = {
... "adlfs.account-name": "AZURE_STORAGE_ACCOUNT_NAME",
... "adlfs.account-key": "AZURE_STORAGE_ACCOUNT_KEY",
... }
>>> pl.scan_iceberg(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for an Iceberg table from Google Cloud Storage.
Supported options for GCS are available `here
<https://py.iceberg.apache.org/configuration/#google-cloud-storage>`__.
>>> table_path = "s3://bucket/path/to/iceberg-table/metadata.json"
>>> storage_options = {
... "gcs.project-id": "my-gcp-project",
... "gcs.oauth.token": "ya29.dr.AfM...",
... }
>>> pl.scan_iceberg(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for an Iceberg table with additional options.
In the below example, `without_files` option is used which loads the table without
file tracking information.
>>> table_path = "/path/to/iceberg-table/metadata.json"
>>> storage_options = {"py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO"}
>>> pl.scan_iceberg(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for an Iceberg table using a specific snapshot ID.
>>> table_path = "/path/to/iceberg-table/metadata.json"
>>> snapshot_id = 7051579356916758811
>>> pl.scan_iceberg(table_path, snapshot_id=snapshot_id).collect() # doctest: +SKIP
scan_ipc(source: 'str | Path | IO[bytes] | bytes | list[str] | list[Path] | list[IO[bytes]] | list[bytes]', *, n_rows: 'int | None' = None, cache: 'bool | None' = None, rechunk: 'bool' = False, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, glob: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', memory_map: 'bool' = True, retries: 'int | None' = None, file_cache_ttl: 'int | None' = None, hive_partitioning: 'bool | None' = None, hive_schema: 'SchemaDict | None' = None, try_parse_hive_dates: 'bool' = True, include_file_paths: 'str | None' = None, _record_batch_statistics: 'bool' = False) -> 'LazyFrame'
Lazily read from an Arrow IPC (Feather v2) file or multiple files via glob patterns.
This allows the query optimizer to push down predicates and projections to the scan
level, thereby potentially reducing memory overhead.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
n_rows
Stop reading from IPC file after reading `n_rows`.
cache
Cache the result after reading.
.. deprecated:: 1.40.0
File cache is no longer supported.
rechunk
Reallocate to contiguous memory when all chunks/ files are parsed.
row_index_name
If not None, this will insert a row index column with give name into the
DataFrame
row_index_offset
Offset to start the row index column (only use if the name is set)
glob
Expand path given via globbing rules.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
memory_map
Try to memory map the file. This can greatly improve performance on repeated
queries as the OS may cache pages.
Only uncompressed IPC files can be memory mapped.
.. deprecated:: 1.40.0
Controlling memory map behavior is no longer supported.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
file_cache_ttl
Amount of time to keep downloaded cloud files since their last access time,
in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable
(which defaults to 1 hour) if not given.
.. deprecated:: 1.40.0
File cache is no longer supported.
hive_partitioning
Infer statistics and schema from Hive partitioned URL and use them
to prune reads. This is unset by default (i.e. `None`), meaning it is
automatically enabled when a single directory is passed, and otherwise
disabled.
hive_schema
The column names and data types of the columns by which the data is partitioned.
If set to `None` (default), the schema of the Hive partitions is inferred.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
try_parse_hive_dates
Whether to try parsing hive values as date/datetime types.
include_file_paths
Include the path of the source file(s) as a column with this name.
scan_lines(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]]', *, name: 'str' = 'line', n_rows: 'int | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, glob: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', include_file_paths: 'str | None' = None) -> 'LazyFrame'
Construct a LazyFrame which scans lines into a string column from a file.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
name
Name to use for the output column.
n_rows
Stop reading from parquet file after reading `n_rows`.
row_index_name
If not None, this will insert a row index column with the given name into the
DataFrame
row_index_offset
Offset to start the row index column (only used if the name is set)
glob
Expand path given via globbing rules.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \
`{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
include_file_paths
Include the path of the source file(s) as a column with this name.
See Also
--------
read_lines
Examples
--------
>>> pl.scan_lines(b"Hello\nworld").collect()
shape: (2, 1)
┌───────┐
│ line │
│ --- │
│ str │
╞═══════╡
│ Hello │
│ world │
└───────┘
scan_ndjson(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]]', *, schema: 'SchemaDefinition | None' = None, schema_overrides: 'SchemaDefinition | None' = None, infer_schema_length: 'int | None' = 100, batch_size: 'int | None' = 1024, n_rows: 'int | None' = None, low_memory: 'bool' = False, rechunk: 'bool' = False, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, ignore_errors: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, file_cache_ttl: 'int | None' = None, include_file_paths: 'str | None' = None) -> 'LazyFrame'
Lazily read from a newline delimited JSON file or multiple files via glob patterns.
This allows the query optimizer to push down predicates and projections to the scan
level, thereby potentially reducing memory overhead.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path to a file.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the schema param will be overridden.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
batch_size
Number of rows to read in each batch.
n_rows
Stop reading from JSON file after reading `n_rows`.
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Reallocate to contiguous memory when all chunks/ files are parsed.
row_index_name
If not None, this will insert a row index column with give name into the
DataFrame
row_index_offset
Offset to start the row index column (only use if the name is set)
ignore_errors
Return `Null` if parsing fails because of schema mismatches.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
file_cache_ttl
Amount of time to keep downloaded cloud files since their last access time,
in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable
(which defaults to 1 hour) if not given.
.. deprecated:: 1.39.0
File cache is no longer supported.
include_file_paths
Include the path of the source file(s) as a column with this name.
scan_parquet(source: 'FileSource', *, n_rows: 'int | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, parallel: 'ParallelStrategy' = 'auto', use_statistics: 'bool' = True, hive_partitioning: 'bool | None' = None, glob: 'bool' = True, hidden_file_prefix: 'str | Sequence[str] | None' = None, schema: 'SchemaDict | None' = None, hive_schema: 'SchemaDict | None' = None, try_parse_hive_dates: 'bool' = True, rechunk: 'bool' = False, low_memory: 'bool' = False, cache: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, include_file_paths: 'str | None' = None, missing_columns: "Literal['insert', 'raise']" = 'raise', allow_missing_columns: 'bool | None' = None, extra_columns: "Literal['ignore', 'raise']" = 'raise', cast_options: 'ScanCastOptions | None' = None, _column_mapping: 'ColumnMapping | None' = None, _default_values: 'DefaultFieldValues | None' = None, _deletion_files: 'DeletionFiles | None' = None, _table_statistics: 'DataFrame | None' = None, _row_count: 'tuple[int, int] | None' = None) -> 'LazyFrame'
Lazily read from a local or cloud-hosted parquet file (or files).
This function allows the query optimizer to push down predicates and projections to
the scan level, typically increasing performance and reducing memory overhead.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
.. versionchanged:: 1.30.0
* The `allow_missing_columns` is deprecated in favor of `missing_columns`.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
n_rows
Stop reading from parquet file after reading `n_rows`.
row_index_name
If not None, this will insert a row index column with the given name into the
DataFrame
row_index_offset
Offset to start the row index column (only used if the name is set)
parallel : {'auto', 'columns', 'row_groups', 'prefiltered', 'none'}
This determines the direction and strategy of parallelism. 'auto' will
try to determine the optimal direction.
The `prefiltered` strategy first evaluates the pushed-down predicates in
parallel and determines a mask of which rows to read. Then, it
parallelizes over both the columns and the row groups while filtering
out rows that do not need to be read. This can provide significant
speedups for large files (i.e. many row-groups) with a predicate that
filters clustered rows or filters heavily. In other cases,
`prefiltered` may slow down the scan compared other strategies.
The `prefiltered` settings falls back to `auto` if no predicate is
given.
.. warning::
The `prefiltered` strategy is considered **unstable**. It may be
changed at any point without it being considered a breaking change.
use_statistics
Use statistics in the parquet to determine if pages
can be skipped from reading.
hive_partitioning
Infer statistics and schema from hive partitioned URL and use them
to prune reads.
glob
Expand path given via globbing rules.
hidden_file_prefix
Skip reading files whose names begin with the specified prefixes.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
schema
Specify the datatypes of the columns. The datatypes must match the
datatypes in the file(s). If there are extra columns that are not in the
file(s), consider also passing `missing_columns='insert'`.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
hive_schema
The column names and data types of the columns by which the data is partitioned.
If set to `None` (default), the schema of the Hive partitions is inferred.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
try_parse_hive_dates
Whether to try parsing hive values as date/datetime types.
rechunk
In case of reading multiple files via a glob pattern rechunk the final DataFrame
into contiguous memory chunks.
low_memory
Reduce memory pressure at the expense of performance.
cache
Cache the result after reading.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
include_file_paths
Include the path of the source file(s) as a column with this name.
missing_columns
Configuration for behavior when columns defined in the schema
are missing from the data:
* `insert`: Inserts the missing columns using NULLs as the row values.
* `raise`: Raises an error.
allow_missing_columns
When reading a list of parquet files, if a column existing in the first
file cannot be found in subsequent files, the default behavior is to
raise an error. However, if `allow_missing_columns` is set to
`True`, a full-NULL column is returned instead of erroring for the files
that do not contain the column.
.. deprecated:: 1.30.0
Use the parameter `missing_columns` instead and pass one of
`('insert', 'raise')`.
extra_columns
Configuration for behavior when extra columns outside of the
defined schema are encountered in the data:
* `ignore`: Silently ignores.
* `raise`: Raises an error.
cast_options
Configuration for column type-casting during scans. Useful for datasets
containing files that have differing schemas.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
See Also
--------
read_parquet
scan_pyarrow_dataset
Examples
--------
Scan a local Parquet file.
>>> pl.scan_parquet("path/to/file.parquet") # doctest: +SKIP
Scan a file on AWS S3.
>>> source = "s3://bucket/*.parquet"
>>> pl.scan_parquet(source) # doctest: +SKIP
>>> storage_options = {
... "aws_access_key_id": "<secret>",
... "aws_secret_access_key": "<secret>",
... "aws_region": "us-east-1",
... }
>>> pl.scan_parquet(source, storage_options=storage_options) # doctest: +SKIP
scan_pyarrow_dataset(source: 'pa.dataset.Dataset', *, allow_pyarrow_filter: 'bool' = True, batch_size: 'int | None' = None) -> 'LazyFrame'
Scan a pyarrow dataset.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
This can be useful to connect to cloud or partitioned datasets.
Parameters
----------
source
Pyarrow dataset to scan.
allow_pyarrow_filter
Allow predicates to be pushed down to pyarrow. This can lead to different
results if comparisons are done with null values as pyarrow handles this
different than polars does.
batch_size
The maximum row count for scanned pyarrow record batches.
Warnings
--------
Don't use this if you accept untrusted user inputs. Predicates will be evaluated
with python 'eval'. There is sanitation in place, but it is a possible attack
vector.
This method can only can push down predicates that are allowed by PyArrow
(e.g. not the full Polars API).
If :func:`scan_parquet` works for your source, you should use that instead.
Notes
-----
When using partitioning, the appropriate `partitioning` option must be set on
`pyarrow.dataset.dataset` before passing to Polars or the partitioned-on column(s)
may not get passed to Polars.
Examples
--------
>>> import pyarrow.dataset as ds
>>> dset = ds.dataset("s3://my-partitioned-folder/", format="ipc") # doctest: +SKIP
>>> (
... pl.scan_pyarrow_dataset(dset)
... .filter("bools")
... .select("bools", "floats", "date")
... .collect()
... ) # doctest: +SKIP
shape: (1, 3)
┌───────┬────────┬────────────┐
│ bools ┆ floats ┆ date │
│ --- ┆ --- ┆ --- │
│ bool ┆ f64 ┆ date │
╞═══════╪════════╪════════════╡
│ true ┆ 2.0 ┆ 1970-05-04 │
└───────┴────────┴────────────┘
DATA
__all__ = ['defer', 'FileProviderArgs', 'PartitionBy', 'read_avro', 'r...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\io\__init__.py
polars.jemalloc_conf
Error rendering doc: No Python documentation found for 'dirty_decay_ms:500,muzzy_decay_ms:1000'. Use help() to get the interactive help utility. Use help(str) for help on the str class.
polars.json_normalize
Python Library Documentation: function json_normalize in module polars.convert.normalize
json_normalize(data: 'dict[Any, Any] | Sequence[dict[Any, Any] | Any]', *, separator: 'str' = '.', max_level: 'int | None' = None, schema: 'Schema | None' = None, strict: 'bool' = True, infer_schema_length: 'int | None' = 100, encoder: 'JSONEncoder | None' = None) -> 'DataFrame'
Normalize semi-structured deserialized JSON data into a flat table.
Dictionary objects that will not be unnested/normalized are encoded
as json string data. Unlike it pandas' counterpart, this function will
not encode dictionaries as objects at any level.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
data
Deserialized JSON objects.
separator
Nested records will generate names separated by sep. e.g.,
for `separator=".", {"foo": {"bar": 0}}` -> foo.bar.
max_level
Max number of levels(depth of dict) to normalize.
If None, normalizes all levels.
schema
Overwrite the `Schema` when the normalized data is passed to
the `DataFrame` constructor.
strict
Whether Polars should be strict when constructing the DataFrame.
infer_schema_length
Number of rows to take into consideration to determine the schema.
encoder
Custom JSON encoder function; if not given, `json.dumps` is used.
Examples
--------
>>> data = [
... {
... "id": 1,
... "name": "Cole Volk",
... "fitness": {"height": 180, "weight": 85},
... },
... {
... "id": 2,
... "name": "Faye Raker",
... "fitness": {"height": 155, "weight": 58},
... },
... {
... "name": "Mark Reg",
... "fitness": {"height": 170, "weight": 78},
... },
... ]
>>> pl.json_normalize(data, max_level=1)
shape: (3, 4)
┌──────┬────────────┬────────────────┬────────────────┐
│ id ┆ name ┆ fitness.height ┆ fitness.weight │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ i64 ┆ i64 │
╞══════╪════════════╪════════════════╪════════════════╡
│ 1 ┆ Cole Volk ┆ 180 ┆ 85 │
│ 2 ┆ Faye Raker ┆ 155 ┆ 58 │
│ null ┆ Mark Reg ┆ 170 ┆ 78 │
└──────┴────────────┴────────────────┴────────────────┘
Normalize to a specific depth, using a custom JSON encoder
(note that `orjson.dumps` encodes to bytes, not str).
>>> import orjson
>>> pl.json_normalize(data, max_level=0, encoder=orjson.dumps)
shape: (3, 3)
┌──────┬────────────┬───────────────────────────────┐
│ id ┆ name ┆ fitness │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ binary │
╞══════╪════════════╪═══════════════════════════════╡
│ 1 ┆ Cole Volk ┆ b"{"height":180,"weight":85}" │
│ 2 ┆ Faye Raker ┆ b"{"height":155,"weight":58}" │
│ null ┆ Mark Reg ┆ b"{"height":170,"weight":78}" │
└──────┴────────────┴───────────────────────────────┘
polars.last
Python Library Documentation: function last in module polars.functions.lazy
last(*columns: 'str') -> 'Expr'
Get the last column or value.
This function has different behavior depending on the presence of `columns`
values. If none given (the default), returns an expression that takes the last
column of the context; otherwise, takes the last value of the given column(s).
Parameters
----------
*columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "baz"],
... }
... )
Return the last column:
>>> df.select(pl.last())
shape: (3, 1)
┌─────┐
│ c │
│ --- │
│ str │
╞═════╡
│ foo │
│ bar │
│ baz │
└─────┘
Return the last value for the given column(s):
>>> df.select(pl.last("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 3 │
└─────┘
>>> df.select(pl.last("b", "c"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 2 ┆ baz │
└─────┴─────┘
polars.lazyframe
Python Library Documentation: package polars.lazyframe in polars
NAME
polars.lazyframe
PACKAGE CONTENTS
engine_config
frame
group_by
in_process
opt_flags
query_result
CLASSES
builtins.object
polars.lazyframe.engine_config.GPUEngine
polars.lazyframe.frame.LazyFrame
polars.lazyframe.opt_flags.QueryOptFlags
polars.lazyframe.query_result.SingleNodeQueryResult
typing.Protocol(typing.Generic)
polars.lazyframe.query_result.QueryResult
class GPUEngine(builtins.object)
| GPUEngine(*, device: 'int | None' = None, memory_resource: 'Any | None' = None, raise_on_fail: 'bool' = False, **kwargs: 'Any') -> 'None'
|
| Configuration options for the GPU execution engine.
|
| Use this if you want control over details of the execution.
|
| Parameters
| ----------
| device : int, default None
| Select the GPU used to run the query. If not provided, the
| query uses the current CUDA device.
| memory_resource : rmm.mr.DeviceMemoryResource, default None
| Provide a memory resource for GPU memory allocations.
|
| .. warning::
| If passing a `memory_resource`, you must ensure that it is valid
| for the selected `device`. See the `RMM documentation
| <https://github.com/rapidsai/rmm?tab=readme-ov-file#multiple-devices>`_
| for more details.
|
| raise_on_fail : bool, default False
| If True, do not fall back to the Polars CPU engine if the GPU
| engine cannot execute the query, but instead raise an error.
|
| Methods defined here:
|
| __init__(self, *, device: 'int | None' = None, memory_resource: 'Any | None' = None, raise_on_fail: 'bool' = False, **kwargs: 'Any') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'config': 'Mapping[str, Any]', 'device': 'int | Non...
class LazyFrame(builtins.object)
| LazyFrame(data: 'FrameInitTypes | None' = None, schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100, nan_to_null: 'bool' = False, height: 'int | None' = None) -> 'None'
|
| Representation of a Lazy computation graph/query against a DataFrame.
|
| This allows for whole-query optimisation in addition to parallelism, and
| is the preferred (and highest-performance) mode of operation for polars.
|
| Parameters
| ----------
| data : dict, Sequence, ndarray, Series, or pandas.DataFrame
| Two-dimensional data in various forms; dict input must contain Sequences,
| Generators, or a `range`. Sequence may contain Series or other Sequences.
| schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
| The LazyFrame schema may be declared in several ways:
|
| * As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
| * As a list of column names; in this case types are automatically inferred.
| * As a list of (name,type) pairs; this is equivalent to the dictionary form.
|
| The order of the schema determines the column order of the frame.
| When passing a dict, its insertion order is respected. To override specific
| column data types by name without changing column order, use
| ``schema_overrides`` instead.
|
| If you supply a list of column names that does not match the names in the
| underlying data, the names given here will overwrite them. The number
| of names given in the schema should match the underlying data dimensions.
| schema_overrides : dict, default None
| Support type specification or override of one or more columns; note that
| any dtypes inferred from the schema param will be overridden.
|
| The number of entries in the schema should match the underlying data
| dimensions, unless a sequence of dictionaries is being passed, in which case
| a *partial* schema can be declared to prevent specific fields from being loaded.
| strict : bool, default True
| Throw an error if any `data` value does not exactly match the given or inferred
| data type for that column. If set to `False`, values that do not match the data
| type are cast to that data type or, if casting is not possible, set to null
| instead.
| orient : {'col', 'row'}, default None
| Whether to interpret two-dimensional data as columns or as rows. If None,
| the orientation is inferred by matching the columns and data dimensions. If
| this does not yield conclusive results, column orientation is used.
| infer_schema_length : int or None
| The maximum number of rows to scan for schema inference. If set to `None`, the
| full data may be scanned *(this can be slow)*. This parameter only applies if
| the input data is a sequence or generator of rows; other input is read as-is.
| nan_to_null : bool, default False
| If the data comes from one or more numpy arrays, can optionally convert input
| data np.nan values to null instead. This is a no-op for all other input data.
| height : int or None, default None
| Allows constructing DataFrames with 0 width and a specified height. If
| passed with data, ensures the resulting DataFrame has this height.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Notes
| -----
| Initialising `LazyFrame(...)` directly is equivalent to `DataFrame(...).lazy()`.
|
| Examples
| --------
| Constructing a LazyFrame directly from a dictionary:
|
| >>> data = {"a": [1, 2], "b": [3, 4]}
| >>> lf = pl.LazyFrame(data)
| >>> lf.collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
|
| Notice that the dtypes are automatically inferred as Polars Int64:
|
| >>> lf.collect_schema().dtypes()
| [Int64, Int64]
|
| To specify a more detailed/specific frame schema you can supply the `schema`
| parameter with a dictionary of (name,dtype) pairs...
|
| >>> data = {"col1": [0, 2], "col2": [3, 7]}
| >>> lf2 = pl.LazyFrame(data, schema={"col1": pl.Float32, "col2": pl.Int64})
| >>> lf2.collect()
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 0.0 ┆ 3 │
| │ 2.0 ┆ 7 │
| └──────┴──────┘
|
| ...a sequence of (name,dtype) pairs...
|
| >>> data = {"col1": [1, 2], "col2": [3, 4]}
| >>> lf3 = pl.LazyFrame(data, schema=[("col1", pl.Float32), ("col2", pl.Int64)])
| >>> lf3.collect()
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 1.0 ┆ 3 │
| │ 2.0 ┆ 4 │
| └──────┴──────┘
|
| ...or a list of typed Series.
|
| >>> data = [
| ... pl.Series("col1", [1, 2], dtype=pl.Float32),
| ... pl.Series("col2", [3, 4], dtype=pl.Int64),
| ... ]
| >>> lf4 = pl.LazyFrame(data)
| >>> lf4.collect()
| shape: (2, 2)
| ┌──────┬──────┐
| │ col1 ┆ col2 │
| │ --- ┆ --- │
| │ f32 ┆ i64 │
| ╞══════╪══════╡
| │ 1.0 ┆ 3 │
| │ 2.0 ┆ 4 │
| └──────┴──────┘
|
| Constructing a LazyFrame from a numpy ndarray, specifying column names:
|
| >>> import numpy as np
| >>> data = np.array([(1, 2), (3, 4)], dtype=np.int64)
| >>> lf5 = pl.LazyFrame(data, schema=["a", "b"], orient="col")
| >>> lf5.collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
|
| Constructing a LazyFrame from a list of lists, row orientation specified:
|
| >>> data = [[1, 2, 3], [4, 5, 6]]
| >>> lf6 = pl.LazyFrame(data, schema=["a", "b", "c"], orient="row")
| >>> lf6.collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 2 ┆ 3 │
| │ 4 ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| Methods defined here:
|
| __bool__(self) -> 'NoReturn'
|
| __contains__(self, key: 'str') -> 'bool'
|
| __copy__(self) -> 'LazyFrame'
|
| __deepcopy__(self, memo: 'None' = None) -> 'LazyFrame'
|
| __eq__(self, other: 'object') -> 'NoReturn'
| Return self==value.
|
| __ge__(self, other: 'Any') -> 'NoReturn'
| Return self>=value.
|
| __getitem__(self, item: 'slice_') -> 'LazyFrame'
| Support slice syntax, returning a new LazyFrame.
|
| All other forms of subscripting are currently unsupported here; use `select`,
| `filter`, or other standard methods instead.
|
| Notes
| -----
| LazyFrame is designed primarily for efficient computation and does not know
| its own length so, unlike DataFrame, certain slice patterns (such as those
| requiring negative stop/step) may not be supported.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
| >>> lf[:2].collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 5 │
| └─────┴─────┘
| >>> lf[::2].collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 3 ┆ 6 │
| └─────┴─────┘
|
| __getstate__(self) -> 'bytes'
| Helper for pickle.
|
| __gt__(self, other: 'Any') -> 'NoReturn'
| Return self>value.
|
| __init__(self, data: 'FrameInitTypes | None' = None, schema: 'SchemaDefinition | None' = None, *, schema_overrides: 'SchemaDict | None' = None, strict: 'bool' = True, orient: 'Orientation | None' = None, infer_schema_length: 'int | None' = 100, nan_to_null: 'bool' = False, height: 'int | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __le__(self, other: 'Any') -> 'NoReturn'
| Return self<=value.
|
| __lt__(self, other: 'Any') -> 'NoReturn'
| Return self<value.
|
| __ne__(self, other: 'object') -> 'NoReturn'
| Return self!=value.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, state: 'bytes') -> 'None'
|
| __str__(self) -> 'str'
| Return str(self).
|
| approx_n_unique(self) -> 'LazyFrame'
| Approximate count of unique values.
|
| .. deprecated:: 0.20.11
| Use `select(pl.all().approx_n_unique())` instead.
|
| This is done using the HyperLogLog++ algorithm for cardinality estimation.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.approx_n_unique().collect() # doctest: +SKIP
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| bottom_k(self, k: 'int', *, by: 'IntoExpr | Iterable[IntoExpr]', reverse: 'bool | Sequence[bool]' = False) -> 'LazyFrame'
| Return the `k` smallest rows.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| k
| Number of rows to return.
| by
| Column(s) used to determine the bottom rows.
| Accepts expression input. Strings are parsed as column names.
| reverse
| Consider the `k` largest elements of the `by` column(s) (instead of the `k`
| smallest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [2, 1, 1, 3, 2, 1],
| ... }
| ... )
|
| Get the rows which contain the 4 smallest values in column b.
|
| >>> lf.bottom_k(4, by="b").collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 1 │
| │ a ┆ 1 │
| │ c ┆ 1 │
| │ a ┆ 2 │
| └─────┴─────┘
|
| Get the rows which contain the 4 smallest values when sorting on column a and b.
|
| >>> lf.bottom_k(4, by=["a", "b"]).collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ a ┆ 2 │
| │ b ┆ 1 │
| │ b ┆ 2 │
| └─────┴─────┘
|
| cache(self) -> 'LazyFrame'
| Cache the result once the execution of the physical plan hits this node.
|
| It is not recommended using this as the optimizer likely can do a better job.
|
| cast(self, dtypes: 'Mapping[ColumnNameOrSelector | PolarsDataType, PolarsDataType | PythonDataType] | PolarsDataType | pl.DataTypeExpr | Schema', *, strict: 'bool' = True) -> 'LazyFrame'
| Cast LazyFrame column(s) to the specified dtype(s).
|
| Parameters
| ----------
| dtypes
| Mapping of column names (or selector) to dtypes, or a single dtype
| to which all columns will be cast.
| strict
| Throw an error if a cast could not be done (for instance, due to an
| overflow).
|
| Examples
| --------
| >>> from datetime import date
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": [date(2020, 1, 2), date(2021, 3, 4), date(2022, 5, 6)],
| ... }
| ... )
|
| Cast specific frame columns to the specified dtypes:
|
| >>> lf.cast({"foo": pl.Float32, "bar": pl.UInt8}).collect()
| shape: (3, 3)
| ┌─────┬─────┬────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f32 ┆ u8 ┆ date │
| ╞═════╪═════╪════════════╡
| │ 1.0 ┆ 6 ┆ 2020-01-02 │
| │ 2.0 ┆ 7 ┆ 2021-03-04 │
| │ 3.0 ┆ 8 ┆ 2022-05-06 │
| └─────┴─────┴────────────┘
|
| Cast all frame columns matching one dtype (or dtype group) to another dtype:
|
| >>> lf.cast({pl.Date: pl.Datetime}).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────────────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ datetime[μs] │
| ╞═════╪═════╪═════════════════════╡
| │ 1 ┆ 6.0 ┆ 2020-01-02 00:00:00 │
| │ 2 ┆ 7.0 ┆ 2021-03-04 00:00:00 │
| │ 3 ┆ 8.0 ┆ 2022-05-06 00:00:00 │
| └─────┴─────┴─────────────────────┘
|
| Use selectors to define the columns being cast:
|
| >>> import polars.selectors as cs
| >>> lf.cast({cs.numeric(): pl.UInt32, cs.temporal(): pl.String}).collect()
| shape: (3, 3)
| ┌─────┬─────┬────────────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ str │
| ╞═════╪═════╪════════════╡
| │ 1 ┆ 6 ┆ 2020-01-02 │
| │ 2 ┆ 7 ┆ 2021-03-04 │
| │ 3 ┆ 8 ┆ 2022-05-06 │
| └─────┴─────┴────────────┘
|
| Cast all frame columns to the specified dtype:
|
| >>> lf.cast(pl.String).collect().to_dict(as_series=False)
| {'foo': ['1', '2', '3'],
| 'bar': ['6.0', '7.0', '8.0'],
| 'ham': ['2020-01-02', '2021-03-04', '2022-05-06']}
|
| clear(self, n: 'int' = 0) -> 'LazyFrame'
| Create an empty copy of the current LazyFrame, with zero to 'n' rows.
|
| Returns a copy with an identical schema but no data.
|
| Parameters
| ----------
| n
| Number of (empty) rows to return in the cleared frame.
|
| See Also
| --------
| clone : Cheap deepcopy/clone.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> lf.clear().collect()
| shape: (0, 3)
| ┌─────┬─────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞═════╪═════╪══════╡
| └─────┴─────┴──────┘
|
| >>> lf.clear(2).collect()
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ null ┆ null ┆ null │
| └──────┴──────┴──────┘
|
| clone(self) -> 'LazyFrame'
| Create a copy of this LazyFrame.
|
| This is a cheap operation that does not copy data.
|
| See Also
| --------
| clear : Create an empty copy of the current LazyFrame, with identical
| schema but no data.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> lf.clone()
| <LazyFrame at ...>
|
| collect(self, *, type_coercion: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, no_optimization: 'bool' = False, engine: 'EngineType' = 'auto', background: 'bool' = False, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, **_kwargs: 'Any') -> 'DataFrame | InProcessQuery'
| Materialize this `LazyFrame` into a `DataFrame`.
|
| By default, all query optimizations are enabled. Individual optimizations may
| be disabled by setting the corresponding parameter to `False`.
|
| Parameters
| ----------
| type_coercion
| Do type coercion optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| predicate_pushdown
| Do predicate pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| projection_pushdown
| Do projection pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| simplify_expression
| Run simplify expressions optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| slice_pushdown
| Slice pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subplan_elim
| Will try to cache branching subplans that occur on self-joins or unions.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subexpr_elim
| Common subexpressions will be cached and reused.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| cluster_with_columns
| Combine sequential independent calls to with_columns
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| collapse_joins
| Collapse a join and filters into a faster join
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| no_optimization
| Turn off (certain) optimizations.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| background
| Run the query in the background and get a handle to the query.
| This handle can be used to fetch the result or cancel the query.
|
| .. warning::
| Background mode is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| See Also
| --------
| explain : Print the query plan that is evaluated with collect.
| profile : Collect the LazyFrame and time each node in the computation graph.
| polars.collect_all : Collect multiple LazyFrames at the same time.
| polars.Config.set_streaming_chunk_size : Set the size of streaming batches.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a").agg(pl.all().sum()).collect() # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| Collect in streaming mode
|
| >>> lf.group_by("a").agg(pl.all().sum()).collect(
| ... engine="streaming"
| ... ) # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| Collect in GPU mode
|
| >>> lf.group_by("a").agg(pl.all().sum()).collect(engine="gpu") # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 11 ┆ 10 │
| │ a ┆ 4 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| With control over the device used
|
| >>> lf.group_by("a").agg(pl.all().sum()).collect(
| ... engine=pl.GPUEngine(device=1)
| ... ) # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ b ┆ 11 ┆ 10 │
| │ a ┆ 4 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| collect_async(self, *, gevent: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'Awaitable[DataFrame] | _GeventDataFrameResult[DataFrame]'
| Collect DataFrame asynchronously in thread pool.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Collects into a DataFrame (like :func:`collect`) but, instead of returning
| a DataFrame directly, it is scheduled to be collected inside a thread pool,
| while this method returns almost instantly.
|
| This can be useful if you use `gevent` or `asyncio` and want to release
| control to other greenlets/tasks while LazyFrames are being collected.
|
| Parameters
| ----------
| gevent
| Return wrapper to `gevent.event.AsyncResult` instead of Awaitable
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU
| systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| The GPU engine does not support async, or running in the
| background. If either are enabled, then GPU execution is switched off.
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| If `gevent=False` (default) then returns an awaitable.
|
| If `gevent=True` then returns wrapper that has a
| `.get(block=True, timeout=None)` method.
|
| See Also
| --------
| polars.collect_all : Collect multiple LazyFrames at the same time.
| polars.collect_all_async : Collect multiple LazyFrames at the same time lazily.
|
| Notes
| -----
| In case of error `set_exception` is used on
| `asyncio.Future`/`gevent.event.AsyncResult` and will be reraised by them.
|
| Examples
| --------
| >>> import asyncio
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> async def main():
| ... return await (
| ... lf.group_by("a", maintain_order=True)
| ... .agg(pl.all().sum())
| ... .collect_async()
| ... )
| >>> asyncio.run(main())
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| collect_batches(self, *, chunk_size: 'int | None' = None, maintain_order: 'bool' = True, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'Iterator[DataFrame]'
| Evaluate the query in streaming mode and get a generator that returns chunks.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| The query will always be fully executed unless `stop` is called, so you should
| call next until all chunks have been seen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. warning::
| This method is much slower than native sinks. Only use it if you cannot
| implement your logic otherwise.
|
| Parameters
| ----------
| chunk_size
| The number of rows that are buffered before a chunk is given.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
| lazy
| Start the query when first requesting a batch.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> for df in lf.collect_batches():
| ... print(df) # doctest: +SKIP
|
| collect_schema(self) -> 'Schema'
| Resolve the schema of this LazyFrame.
|
| .. caution::
| Computing the schema of a LazyFrame is a potentially expensive operation,
| as it may involve reading metadata from (slow) disk storage, or performing
| network requests if the data is remote.
|
| Examples
| --------
| Determine the schema.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.collect_schema()
| Schema({'foo': Int64, 'bar': Float64, 'ham': String})
|
| Access various properties of the schema.
|
| >>> schema = lf.collect_schema()
| >>> schema["bar"]
| Float64
| >>> schema.names()
| ['foo', 'bar', 'ham']
| >>> schema.dtypes()
| [Int64, Float64, String]
| >>> schema.len()
| 3
|
| count(self) -> 'LazyFrame'
| Return the number of non-null elements for each column.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {"a": [1, 2, 3, 4], "b": [1, 2, 1, None], "c": [None, None, None, None]}
| ... )
| >>> lf.count().collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 4 ┆ 3 ┆ 0 │
| └─────┴─────┴─────┘
|
| describe(self, percentiles: 'Sequence[float] | float | None' = (0.25, 0.5, 0.75), *, interpolation: 'QuantileMethod' = 'nearest') -> 'DataFrame'
| Creates a summary of statistics for a LazyFrame, returning a DataFrame.
|
| Parameters
| ----------
| percentiles
| One or more percentiles to include in the summary statistics.
| All values must be in the range `[0, 1]`.
|
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method used when calculating percentiles.
|
| Returns
| -------
| DataFrame
|
| Notes
| -----
| The median is included by default as the 50% percentile.
|
| Warnings
| --------
| * This method does *not* maintain the laziness of the frame, and will `collect`
| the final result. This could potentially be an expensive operation.
| * We do not guarantee the output of `describe` to be stable. It will show
| statistics that we deem informative, and may be updated in the future.
| Using `describe` programmatically (versus interactive exploration) is
| not recommended for this reason.
|
| Examples
| --------
| >>> from datetime import date, time
| >>> lf = pl.LazyFrame(
| ... {
| ... "float": [1.0, 2.8, 3.0],
| ... "int": [40, 50, None],
| ... "bool": [True, False, True],
| ... "str": ["zz", "xx", "yy"],
| ... "date": [date(2020, 1, 1), date(2021, 7, 5), date(2022, 12, 31)],
| ... "time": [time(10, 20, 30), time(14, 45, 50), time(23, 15, 10)],
| ... }
| ... )
|
| Show default frame statistics:
|
| >>> lf.describe()
| shape: (9, 7)
| ┌────────────┬──────────┬──────────┬──────────┬──────┬─────────────────────┬──────────┐
| │ statistic ┆ float ┆ int ┆ bool ┆ str ┆ date ┆ time │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞════════════╪══════════╪══════════╪══════════╪══════╪═════════════════════╪══════════╡
| │ count ┆ 3.0 ┆ 2.0 ┆ 3.0 ┆ 3 ┆ 3 ┆ 3 │
| │ null_count ┆ 0.0 ┆ 1.0 ┆ 0.0 ┆ 0 ┆ 0 ┆ 0 │
| │ mean ┆ 2.266667 ┆ 45.0 ┆ 0.666667 ┆ null ┆ 2021-07-02 16:00:00 ┆ 16:07:10 │
| │ std ┆ 1.101514 ┆ 7.071068 ┆ null ┆ null ┆ null ┆ null │
| │ min ┆ 1.0 ┆ 40.0 ┆ 0.0 ┆ xx ┆ 2020-01-01 ┆ 10:20:30 │
| │ 25% ┆ 2.8 ┆ 40.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 50% ┆ 2.8 ┆ 50.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 75% ┆ 3.0 ┆ 50.0 ┆ null ┆ null ┆ 2022-12-31 ┆ 23:15:10 │
| │ max ┆ 3.0 ┆ 50.0 ┆ 1.0 ┆ zz ┆ 2022-12-31 ┆ 23:15:10 │
| └────────────┴──────────┴──────────┴──────────┴──────┴─────────────────────┴──────────┘
|
| Customize which percentiles are displayed, applying linear interpolation:
|
| >>> with pl.Config(tbl_rows=12):
| ... lf.describe(
| ... percentiles=[0.1, 0.3, 0.5, 0.7, 0.9],
| ... interpolation="linear",
| ... )
| shape: (11, 7)
| ┌────────────┬──────────┬──────────┬──────────┬──────┬─────────────────────┬──────────┐
| │ statistic ┆ float ┆ int ┆ bool ┆ str ┆ date ┆ time │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞════════════╪══════════╪══════════╪══════════╪══════╪═════════════════════╪══════════╡
| │ count ┆ 3.0 ┆ 2.0 ┆ 3.0 ┆ 3 ┆ 3 ┆ 3 │
| │ null_count ┆ 0.0 ┆ 1.0 ┆ 0.0 ┆ 0 ┆ 0 ┆ 0 │
| │ mean ┆ 2.266667 ┆ 45.0 ┆ 0.666667 ┆ null ┆ 2021-07-02 16:00:00 ┆ 16:07:10 │
| │ std ┆ 1.101514 ┆ 7.071068 ┆ null ┆ null ┆ null ┆ null │
| │ min ┆ 1.0 ┆ 40.0 ┆ 0.0 ┆ xx ┆ 2020-01-01 ┆ 10:20:30 │
| │ 10% ┆ 1.36 ┆ 41.0 ┆ null ┆ null ┆ 2020-04-20 ┆ 11:13:34 │
| │ 30% ┆ 2.08 ┆ 43.0 ┆ null ┆ null ┆ 2020-11-26 ┆ 12:59:42 │
| │ 50% ┆ 2.8 ┆ 45.0 ┆ null ┆ null ┆ 2021-07-05 ┆ 14:45:50 │
| │ 70% ┆ 2.88 ┆ 47.0 ┆ null ┆ null ┆ 2022-02-07 ┆ 18:09:34 │
| │ 90% ┆ 2.96 ┆ 49.0 ┆ null ┆ null ┆ 2022-09-13 ┆ 21:33:18 │
| │ max ┆ 3.0 ┆ 50.0 ┆ 1.0 ┆ zz ┆ 2022-12-31 ┆ 23:15:10 │
| └────────────┴──────────┴──────────┴──────────┴──────┴─────────────────────┴──────────┘
|
| drop(self, *columns: 'ColumnNameOrSelector | Iterable[ColumnNameOrSelector]', strict: 'bool' = True) -> 'LazyFrame'
| Remove columns from the DataFrame.
|
| Parameters
| ----------
| *columns
| Names of the columns that should be removed from the dataframe.
| Accepts column selector input.
| strict
| Validate that all column names exist in the current schema,
| and throw an exception if any do not.
|
| Examples
| --------
| Drop a single column by passing the name of that column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.drop("ham").collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 6.0 │
| │ 2 ┆ 7.0 │
| │ 3 ┆ 8.0 │
| └─────┴─────┘
|
| Drop multiple columns by passing a selector.
|
| >>> import polars.selectors as cs
| >>> lf.drop(cs.numeric()).collect()
| shape: (3, 1)
| ┌─────┐
| │ ham │
| │ --- │
| │ str │
| ╞═════╡
| │ a │
| │ b │
| │ c │
| └─────┘
|
| Use positional arguments to drop multiple columns.
|
| >>> lf.drop("foo", "ham").collect()
| shape: (3, 1)
| ┌─────┐
| │ bar │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 6.0 │
| │ 7.0 │
| │ 8.0 │
| └─────┘
|
| drop_nans(self, subset: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None) -> 'LazyFrame'
| Drop all rows that contain one or more NaN values.
|
| The original order of the remaining rows is preserved.
|
| Parameters
| ----------
| subset
| Column name(s) for which NaN values are considered; if set to `None`
| (default), use all columns (note that only floating-point columns
| can contain NaNs).
|
| See Also
| --------
| drop_nulls
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To drop null values, use :func:`drop_nulls`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [-20.5, float("nan"), 80.0],
| ... "bar": [float("nan"), 110.0, 25.5],
| ... "ham": ["xxx", "yyy", None],
| ... }
| ... )
|
| The default behavior of this method is to drop rows where any single
| value in the row is NaN:
|
| >>> lf.drop_nans().collect()
| shape: (1, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════╪══════╪══════╡
| │ 80.0 ┆ 25.5 ┆ null │
| └──────┴──────┴──────┘
|
| This behaviour can be constrained to consider only a subset of columns, as
| defined by name, or with a selector. For example, dropping rows only if
| there is a NaN in the "bar" column:
|
| >>> lf.drop_nans(subset=["bar"]).collect()
| shape: (2, 3)
| ┌──────┬───────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ str │
| ╞══════╪═══════╪══════╡
| │ NaN ┆ 110.0 ┆ yyy │
| │ 80.0 ┆ 25.5 ┆ null │
| └──────┴───────┴──────┘
|
| Dropping a row only if *all* values are NaN requires a different formulation:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [float("nan"), float("nan"), float("nan"), float("nan")],
| ... "b": [10.0, 2.5, float("nan"), 5.25],
| ... "c": [65.75, float("nan"), float("nan"), 10.5],
| ... }
| ... )
| >>> lf.filter(~pl.all_horizontal(pl.all().is_nan())).collect()
| shape: (3, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞═════╪══════╪═══════╡
| │ NaN ┆ 10.0 ┆ 65.75 │
| │ NaN ┆ 2.5 ┆ NaN │
| │ NaN ┆ 5.25 ┆ 10.5 │
| └─────┴──────┴───────┘
|
| drop_nulls(self, subset: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None) -> 'LazyFrame'
| Drop all rows that contain one or more null values.
|
| The original order of the remaining rows is preserved.
|
| See Also
| --------
| drop_nans
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To drop NaN values, use :func:`drop_nans`.
|
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, None, 8],
| ... "ham": ["a", "b", None],
| ... }
| ... )
|
| The default behavior of this method is to drop rows where any single
| value in the row is null:
|
| >>> lf.drop_nulls().collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| This behaviour can be constrained to consider only a subset of columns, as
| defined by name or with a selector. For example, dropping rows if there is
| a null in any of the integer columns:
|
| >>> import polars.selectors as cs
| >>> lf.drop_nulls(subset=cs.integer()).collect()
| shape: (2, 3)
| ┌─────┬─────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 6 ┆ a │
| │ 3 ┆ 8 ┆ null │
| └─────┴─────┴──────┘
|
| Dropping a row only if *all* values are null requires a different formulation:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [None, None, None, None],
| ... "b": [1, 2, None, 1],
| ... "c": [1, None, None, 1],
| ... }
| ... )
| >>> lf.filter(~pl.all_horizontal(pl.all().is_null())).collect()
| shape: (3, 3)
| ┌──────┬─────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ null ┆ i64 ┆ i64 │
| ╞══════╪═════╪══════╡
| │ null ┆ 1 ┆ 1 │
| │ null ┆ 2 ┆ null │
| │ null ┆ 1 ┆ 1 │
| └──────┴─────┴──────┘
|
| execute(self, *, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, engine: 'EngineType' = 'auto', **_kwargs: 'Any') -> 'QueryResult'
| Execute the query into a `QueryResult`.
|
| This method of materializing a `LazyFrame` makes no guarantees as to where
| the result is materialized. This can be on the GPU for the GPU-engine,
| on the cluster or remote storage for the distributed engine and the streaming
| engine could spill the result if it needed to.
|
| The `QueryResult` can always be consumed as a new `LazyFrame` by calling `.lazy`
|
| Parameters
| ----------
| engine
| Select the engine used to process the query, optional.
| At the moment, if set to `"auto"` (default), the query
| is run using the polars in-memory engine. Polars will also
| attempt to use the engine set by the `POLARS_ENGINE_AFFINITY`
| environment variable. If it cannot run the query using the
| selected engine, the query is run using the polars in-memory
| engine. If set to `"gpu"`, the GPU engine is used. Fine-grained
| control over the GPU engine, for example which device to use
| on a system with multiple devices, is possible by providing a
| :class:`~.GPUEngine` object with configuration options.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| See Also
| --------
| explain : Print the query plan that is evaluated with collect.
| profile : Collect the LazyFrame and time each node in the computation graph.
| polars.collect_all : Collect multiple LazyFrames at the same time.
| polars.Config.set_streaming_chunk_size : Set the size of streaming batches.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a").agg(pl.all().sum()).collect() # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| Collect in streaming mode
|
| >>> lf.group_by("a").agg(pl.all().sum()).collect(
| ... engine="streaming"
| ... ) # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘
|
| Collect in GPU mode
|
| explain(self, *, format: 'ExplainFormat' = 'plain', optimized: 'bool' = True, type_coercion: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, streaming: 'bool' = False, engine: 'EngineType' = 'auto', tree_format: 'bool | None' = None, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'str'
| Create a string representation of the query plan.
|
| Different optimizations can be turned on or off.
|
| Parameters
| ----------
| format : {'plain', 'tree'}
| The format to use for displaying the logical plan.
| optimized
| Return an optimized query plan. Defaults to `True`.
| If this is set to `True` the subsequent
| optimization flags control which optimizations
| run.
| type_coercion
| Do type coercion optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| predicate_pushdown
| Do predicate pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| projection_pushdown
| Do projection pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| simplify_expression
| Run simplify expressions optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| slice_pushdown
| Slice pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subplan_elim
| Will try to cache branching subplans that occur on self-joins or unions.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subexpr_elim
| Common subexpressions will be cached and reused.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| cluster_with_columns
| Combine sequential independent calls to with_columns
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| collapse_joins
| Collapse a join and filters into a faster join
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| streaming
| Unused parameter, kept for backward compatibility.
|
| .. deprecated:: 1.30.0
| Use the `engine` parameter instead.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| tree_format
| Format the output as a tree.
|
| .. deprecated:: 0.20.30
| Use `format="tree"` instead.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort(
| ... "a"
| ... ).explain() # doctest: +SKIP
|
| explode(self, columns: 'ColumnNameOrSelector | Iterable[ColumnNameOrSelector]', *more_columns: 'ColumnNameOrSelector', empty_as_null: 'bool' = <object object at 0x00000280D5528F20>, keep_nulls: 'bool' = True) -> 'LazyFrame'
| Explode the DataFrame to long format by exploding the given columns.
|
| Parameters
| ----------
| columns
| Column names, expressions, or a selector defining them. The underlying
| columns being exploded must be of the `List` or `Array` data type.
| *more_columns
| Additional names of columns to explode, specified as positional arguments.
| empty_as_null
| Explode an empty list/array into a `null`.
| keep_nulls
| Explode a `null` list/array into a `null`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "letters": ["a", "a", "b", "c"],
| ... "numbers": [[1], [2, 3], [4, 5], [6, 7, 8]],
| ... }
| ... )
| >>> lf.explode("numbers", empty_as_null=False).collect()
| shape: (8, 2)
| ┌─────────┬─────────┐
| │ letters ┆ numbers │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════════╡
| │ a ┆ 1 │
| │ a ┆ 2 │
| │ a ┆ 3 │
| │ b ┆ 4 │
| │ b ┆ 5 │
| │ c ┆ 6 │
| │ c ┆ 7 │
| │ c ┆ 8 │
| └─────────┴─────────┘
|
| fetch(self, n_rows: 'int' = 500, **kwargs: 'Any') -> 'DataFrame'
| Collect a small number of rows for debugging purposes.
|
| .. deprecated:: 1.0
| Use :meth:`collect` instead, in conjunction with a call to :meth:`head`.`
|
| Notes
| -----
| This is similar to a :func:`collect` operation, but it overwrites the number of
| rows read by *every* scan operation. Be aware that `fetch` does not guarantee
| the final number of rows in the DataFrame. Filters, join operations and fewer
| rows being available in the scanned data will all influence the final number
| of rows (joins are especially susceptible to this, and may return no data
| at all if `n_rows` is too small as the join keys may not be present).
|
| Warnings
| --------
| This is strictly a utility function that can help to debug queries using a
| smaller number of rows, and should *not* be used in production code.
|
| fill_nan(self, value: 'int | float | Expr | None') -> 'LazyFrame'
| Fill floating point NaN values.
|
| Parameters
| ----------
| value
| Value used to fill NaN values.
|
| See Also
| --------
| fill_null
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To fill null values, use :func:`fill_null`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1.5, 2, float("nan"), 4],
| ... "b": [0.5, 4, float("nan"), 13],
| ... }
| ... )
| >>> lf.fill_nan(99).collect()
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪══════╡
| │ 1.5 ┆ 0.5 │
| │ 2.0 ┆ 4.0 │
| │ 99.0 ┆ 99.0 │
| │ 4.0 ┆ 13.0 │
| └──────┴──────┘
|
| fill_null(self, value: 'Any | Expr | None' = None, strategy: 'FillNullStrategy | None' = None, limit: 'int | None' = None, *, matches_supertype: 'bool' = True) -> 'LazyFrame'
| Fill null values using the specified value or strategy.
|
| Parameters
| ----------
| value
| Value used to fill null values.
| strategy : {None, 'forward', 'backward', 'min', 'max', 'mean', 'zero', 'one'}
| Strategy used to fill null values.
| limit
| Number of consecutive null values to fill when using the 'forward' or
| 'backward' strategy.
| matches_supertype
| Fill all matching supertypes of the fill `value` literal.
|
| See Also
| --------
| fill_nan
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To fill NaN values, use :func:`fill_nan`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, None, 4],
| ... "b": [0.5, 4, None, 13],
| ... }
| ... )
| >>> lf.fill_null(99).collect()
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 99 ┆ 99.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
| >>> lf.fill_null(strategy="forward").collect()
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 2 ┆ 4.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| >>> lf.fill_null(strategy="max").collect()
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 4 ┆ 13.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| >>> lf.fill_null(strategy="zero").collect()
| shape: (4, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪══════╡
| │ 1 ┆ 0.5 │
| │ 2 ┆ 4.0 │
| │ 0 ┆ 0.0 │
| │ 4 ┆ 13.0 │
| └─────┴──────┘
|
| filter(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]', **constraints: 'Any') -> 'LazyFrame'
| Filter rows in the LazyFrame based on a predicate expression.
|
| The original order of the remaining rows is preserved.
|
| Rows where the filter predicate does not evaluate to True are discarded
| (this includes rows where the predicate evaluates as `null`).
|
| Parameters
| ----------
| predicates
| Expression that evaluates to a boolean Series.
| constraints
| Column filters; use `name = value` to filter columns using the supplied
| value. Each constraint behaves the same as `pl.col(name).eq(value)`,
| and is implicitly joined with the other filter conditions using `&`.
|
| Notes
| -----
| If you are transitioning from Pandas, and performing filter operations based on
| the comparison of two or more columns, please note that in Polars any comparison
| involving `null` values will result in a `null` result, *not* boolean True or
| False. As a result, these rows will not be retained. Ensure that null values
| are handled appropriately to avoid unexpected behaviour (see examples below).
|
| See Also
| --------
| remove
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3, None, 4, None, 0],
| ... "bar": [6, 7, 8, None, None, 9, 0],
| ... "ham": ["a", "b", "c", None, "d", "e", "f"],
| ... }
| ... )
|
| Filter on one condition:
|
| >>> lf.filter(pl.col("foo") > 1).collect()
| shape: (3, 3)
| ┌─────┬──────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪══════╪═════╡
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ null ┆ d │
| └─────┴──────┴─────┘
|
| Filter on multiple conditions:
|
| >>> lf.filter((pl.col("foo") < 3) & (pl.col("ham") == "a")).collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| Provide multiple filters using `*args` syntax:
|
| >>> lf.filter(
| ... pl.col("foo") == 1,
| ... pl.col("ham") == "a",
| ... ).collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| Provide multiple filters using `**kwargs` syntax:
|
| >>> lf.filter(foo=1, ham="a").collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| └─────┴─────┴─────┘
|
| Filter on an OR condition:
|
| >>> lf.filter(
| ... (pl.col("foo") == 1) | (pl.col("ham") == "c"),
| ... ).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Filter by comparing two columns against each other
|
| >>> lf.filter(
| ... pl.col("foo") == pl.col("bar"),
| ... ).collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 0 ┆ 0 ┆ f │
| └─────┴─────┴─────┘
|
| >>> lf.filter(
| ... pl.col("foo") != pl.col("bar"),
| ... ).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| Notice how the row with `None` values is filtered out; using `ne_missing`
| ensures that null values compare equal, and we get similar behaviour to Pandas:
|
| >>> lf.filter(
| ... pl.col("foo").ne_missing(pl.col("bar")),
| ... ).collect()
| shape: (5, 3)
| ┌──────┬──────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| │ 4 ┆ null ┆ d │
| │ null ┆ 9 ┆ e │
| └──────┴──────┴─────┘
|
| first(self) -> 'LazyFrame'
| Get the first row of the DataFrame.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> lf.first().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 2 │
| └─────┴─────┘
|
| gather(self, indices: 'int | Sequence[int] | IntoExpr | pl.Series | np.ndarray[Any, Any] | pl.LazyFrame', *, null_on_oob: 'bool' = False) -> 'LazyFrame'
| Selects rows from this LazyFrame at the given indices.
|
| .. warning::
| This functionality is experimental. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| indices
| The indices of the rows to select.
|
| Due to the lack of a ``LazySeries`` it's permitted to pass a single-width
| ``LazyFrame`` as indices as well.
|
| null_on_oob
| If true when an index is out-of-bounds a null row will be generated
| instead of raising an error.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"x": [2, 1, 0], "s": ["foo", "bar", "baz"]})
| >>> lf.gather([2, 0, 0]).collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 0 ┆ baz │
| │ 2 ┆ foo │
| │ 2 ┆ foo │
| └─────┴─────┘
|
| >>> lf.gather([0, 10, 1], null_on_oob=True).collect()
| shape: (3, 2)
| ┌──────┬──────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞══════╪══════╡
| │ 2 ┆ foo │
| │ null ┆ null │
| │ 1 ┆ bar │
| └──────┴──────┘
|
| >>> idxs = pl.LazyFrame({"i": [1, 10, 0], "b": [True, False, True]})
| >>> lf.gather(idxs.filter(pl.col.b).select(pl.col.i)).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ x ┆ s │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ bar │
| │ 2 ┆ foo │
| └─────┴─────┘
|
| gather_every(self, n: 'int', offset: 'int' = 0) -> 'LazyFrame'
| Take every nth row in the LazyFrame and return as a new LazyFrame.
|
| Parameters
| ----------
| n
| Gather every *n*-th row.
| offset
| Starting index.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [5, 6, 7, 8],
| ... }
| ... )
| >>> lf.gather_every(2).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 5 │
| │ 3 ┆ 7 │
| └─────┴─────┘
| >>> lf.gather_every(2, offset=1).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 6 │
| │ 4 ┆ 8 │
| └─────┴─────┘
|
| group_by(self, *by: 'IntoExpr | Iterable[IntoExpr]', maintain_order: 'bool' = False, **named_by: 'IntoExpr') -> 'LazyGroupBy'
| Start a group by operation.
|
| Parameters
| ----------
| *by
| Column(s) to group by. Accepts expression input. Strings are parsed as
| column names.
| maintain_order
| Ensure that the order of the groups is consistent with the input data.
| This is slower than a default group by.
| Setting this to `True` blocks the possibility
| to run on the streaming engine.
| **named_by
| Additional columns to group by, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Examples
| --------
| Group by one column and call `agg` to compute the grouped sum of another
| column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "c"],
| ... "b": [1, 2, 1, 3, 3],
| ... "c": [5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a").agg(pl.col("b").sum()).collect() # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 2 │
| │ b ┆ 5 │
| │ c ┆ 3 │
| └─────┴─────┘
|
| Set `maintain_order=True` to ensure the order of the groups is consistent with
| the input.
|
| >>> lf.group_by("a", maintain_order=True).agg(pl.col("c")).collect()
| shape: (3, 2)
| ┌─────┬───────────┐
| │ a ┆ c │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═════╪═══════════╡
| │ a ┆ [5, 3] │
| │ b ┆ [4, 2] │
| │ c ┆ [1] │
| └─────┴───────────┘
|
| Group by multiple columns by passing a list of column names.
|
| >>> lf.group_by(["a", "b"]).agg(pl.max("c")).collect() # doctest: +SKIP
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 1 ┆ 5 │
| │ b ┆ 2 ┆ 4 │
| │ b ┆ 3 ┆ 2 │
| │ c ┆ 3 ┆ 1 │
| └─────┴─────┴─────┘
|
| Or use positional arguments to group by multiple columns in the same way.
| Expressions are also accepted.
|
| >>> lf.group_by("a", pl.col("b") // 2).agg(
| ... pl.col("c").mean()
| ... ).collect() # doctest: +SKIP
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ f64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 0 ┆ 4.0 │
| │ b ┆ 1 ┆ 3.0 │
| │ c ┆ 1 ┆ 1.0 │
| └─────┴─────┴─────┘
|
| group_by_dynamic(self, index_column: 'IntoExpr', *, every: 'str | timedelta', period: 'str | timedelta | None' = None, offset: 'str | timedelta | None' = None, include_boundaries: 'bool' = False, closed: 'ClosedInterval' = 'left', label: 'Label' = 'left', group_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, start_by: 'StartBy' = 'window') -> 'LazyGroupBy'
| Group based on a time value (or index value of type Int32, Int64).
|
| Time windows are calculated and rows are assigned to windows. Different from a
| normal group by is that a row can be member of multiple groups.
| By default, the windows look like:
|
| - [start, start + period)
| - [start + every, start + every + period)
| - [start + 2*every, start + 2*every + period)
| - ...
|
| where `start` is determined by `start_by`, `offset`, `every`, and the earliest
| datapoint. See the `start_by` argument description for details.
|
| .. warning::
| The index column must be sorted in ascending order. If `group_by` is passed, then
| the index column must be sorted in ascending order within each group.
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order (or, if `group_by` is specified,
| then it must be sorted in ascending order within each group).
|
| In case of a dynamic group by on indices, dtype needs to be one of
| {Int32, Int64}. Note that Int32 gets temporarily cast to Int64, so if
| performance matters use an Int64 column.
| every
| interval of the window
| period
| length of the window, if None it will equal 'every'
| offset
| offset of the window, does not take effect if `start_by` is 'datapoint'.
| Defaults to zero.
| include_boundaries
| Add the lower and upper bound of the window to the "_lower_boundary" and
| "_upper_boundary" columns. This will impact performance because it's harder to
| parallelize
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
| label : {'left', 'right', 'datapoint'}
| Define which label to use for the window:
|
| - 'left': lower boundary of the window
| - 'right': upper boundary of the window
| - 'datapoint': the first value of the index column in the given window.
| If you don't need the label to be at one of the boundaries, choose this
| option for maximum performance
| group_by
| Also group by this column/these columns
| start_by : {'window', 'datapoint', 'monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday', 'sunday'}
| The strategy to determine the start of the first window by.
|
| * 'window': Start by taking the earliest timestamp, truncating it with
| `every`, and then adding `offset`.
| Note that weekly windows start on Monday.
| * 'datapoint': Start from the first encountered data point.
| * a day of the week (only takes effect if `every` contains `'w'`):
|
| * 'monday': Start the window on the Monday before the first data point.
| * 'tuesday': Start the window on the Tuesday before the first data point.
| * ...
| * 'sunday': Start the window on the Sunday before the first data point.
|
| The resulting window is then shifted back until the earliest datapoint
| is in or in front of it.
|
| Returns
| -------
| LazyGroupBy
| Object you can call `.agg` on to aggregate by groups, the result
| of which will be sorted by `index_column` (but note that if `group_by` columns are
| passed, it will only be sorted within each group).
|
| See Also
| --------
| rolling
|
| Notes
| -----
| 1) If you're coming from pandas, then
|
| .. code-block:: python
|
| # polars
| df.group_by_dynamic("ts", every="1d").agg(pl.col("value").sum())
|
| is equivalent to
|
| .. code-block:: python
|
| # pandas
| df.set_index("ts").resample("D")["value"].sum().reset_index()
|
| though note that, unlike pandas, polars doesn't add extra rows for empty
| windows. If you need `index_column` to be evenly spaced, then please combine
| with :func:`DataFrame.upsample`.
|
| 2) The `every`, `period` and `offset` arguments are created with
| the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them (except in `every`):
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| In case of a group_by_dynamic on an integer column, the windows are defined by:
|
| - "1i" # length 1
| - "10i" # length 10
|
| Examples
| --------
| >>> from datetime import datetime
| >>> lf = pl.LazyFrame(
| ... {
| ... "time": pl.datetime_range(
| ... start=datetime(2021, 12, 16),
| ... end=datetime(2021, 12, 16, 3),
| ... interval="30m",
| ... eager=True,
| ... ),
| ... "n": range(7),
| ... }
| ... )
| >>> lf.collect()
| shape: (7, 2)
| ┌─────────────────────┬─────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ i64 │
| ╞═════════════════════╪═════╡
| │ 2021-12-16 00:00:00 ┆ 0 │
| │ 2021-12-16 00:30:00 ┆ 1 │
| │ 2021-12-16 01:00:00 ┆ 2 │
| │ 2021-12-16 01:30:00 ┆ 3 │
| │ 2021-12-16 02:00:00 ┆ 4 │
| │ 2021-12-16 02:30:00 ┆ 5 │
| │ 2021-12-16 03:00:00 ┆ 6 │
| └─────────────────────┴─────┘
|
| Group by windows of 1 hour.
|
| >>> lf.group_by_dynamic("time", every="1h", closed="right").agg(
| ... pl.col("n")
| ... ).collect()
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-15 23:00:00 ┆ [0] │
| │ 2021-12-16 00:00:00 ┆ [1, 2] │
| │ 2021-12-16 01:00:00 ┆ [3, 4] │
| │ 2021-12-16 02:00:00 ┆ [5, 6] │
| └─────────────────────┴───────────┘
|
| The window boundaries can also be added to the aggregation result
|
| >>> lf.group_by_dynamic(
| ... "time", every="1h", include_boundaries=True, closed="right"
| ... ).agg(pl.col("n").mean()).collect()
| shape: (4, 4)
| ┌─────────────────────┬─────────────────────┬─────────────────────┬─────┐
| │ _lower_boundary ┆ _upper_boundary ┆ time ┆ n │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ f64 │
| ╞═════════════════════╪═════════════════════╪═════════════════════╪═════╡
| │ 2021-12-15 23:00:00 ┆ 2021-12-16 00:00:00 ┆ 2021-12-15 23:00:00 ┆ 0.0 │
| │ 2021-12-16 00:00:00 ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 00:00:00 ┆ 1.5 │
| │ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ 3.5 │
| │ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ 5.5 │
| └─────────────────────┴─────────────────────┴─────────────────────┴─────┘
|
| When closed="left", the window excludes the right end of interval:
| [lower_bound, upper_bound)
|
| >>> lf.group_by_dynamic("time", every="1h", closed="left").agg(
| ... pl.col("n")
| ... ).collect()
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-16 00:00:00 ┆ [0, 1] │
| │ 2021-12-16 01:00:00 ┆ [2, 3] │
| │ 2021-12-16 02:00:00 ┆ [4, 5] │
| │ 2021-12-16 03:00:00 ┆ [6] │
| └─────────────────────┴───────────┘
|
| When closed="both" the time values at the window boundaries belong to 2 groups.
|
| >>> lf.group_by_dynamic("time", every="1h", closed="both").agg(
| ... pl.col("n")
| ... ).collect()
| shape: (4, 2)
| ┌─────────────────────┬───────────┐
| │ time ┆ n │
| │ --- ┆ --- │
| │ datetime[μs] ┆ list[i64] │
| ╞═════════════════════╪═══════════╡
| │ 2021-12-16 00:00:00 ┆ [0, 1, 2] │
| │ 2021-12-16 01:00:00 ┆ [2, 3, 4] │
| │ 2021-12-16 02:00:00 ┆ [4, 5, 6] │
| │ 2021-12-16 03:00:00 ┆ [6] │
| └─────────────────────┴───────────┘
|
| Dynamic group bys can also be combined with grouping on normal keys
|
| >>> lf = lf.with_columns(groups=pl.Series(["a", "a", "a", "b", "b", "a", "a"]))
| >>> lf.collect()
| shape: (7, 3)
| ┌─────────────────────┬─────┬────────┐
| │ time ┆ n ┆ groups │
| │ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ str │
| ╞═════════════════════╪═════╪════════╡
| │ 2021-12-16 00:00:00 ┆ 0 ┆ a │
| │ 2021-12-16 00:30:00 ┆ 1 ┆ a │
| │ 2021-12-16 01:00:00 ┆ 2 ┆ a │
| │ 2021-12-16 01:30:00 ┆ 3 ┆ b │
| │ 2021-12-16 02:00:00 ┆ 4 ┆ b │
| │ 2021-12-16 02:30:00 ┆ 5 ┆ a │
| │ 2021-12-16 03:00:00 ┆ 6 ┆ a │
| └─────────────────────┴─────┴────────┘
| >>> lf.group_by_dynamic(
| ... "time",
| ... every="1h",
| ... closed="both",
| ... group_by="groups",
| ... include_boundaries=True,
| ... ).agg(pl.col("n")).collect()
| shape: (6, 5)
| ┌────────┬─────────────────────┬─────────────────────┬─────────────────────┬───────────┐
| │ groups ┆ _lower_boundary ┆ _upper_boundary ┆ time ┆ n │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ datetime[μs] ┆ datetime[μs] ┆ datetime[μs] ┆ list[i64] │
| ╞════════╪═════════════════════╪═════════════════════╪═════════════════════╪═══════════╡
| │ a ┆ 2021-12-16 00:00:00 ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 00:00:00 ┆ [0, 1, 2] │
| │ a ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ [2] │
| │ a ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ [5, 6] │
| │ a ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 04:00:00 ┆ 2021-12-16 03:00:00 ┆ [6] │
| │ b ┆ 2021-12-16 01:00:00 ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 01:00:00 ┆ [3, 4] │
| │ b ┆ 2021-12-16 02:00:00 ┆ 2021-12-16 03:00:00 ┆ 2021-12-16 02:00:00 ┆ [4] │
| └────────┴─────────────────────┴─────────────────────┴─────────────────────┴───────────┘
|
| Dynamic group by on an index column
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "idx": pl.int_range(0, 6, eager=True),
| ... "A": ["A", "A", "B", "B", "B", "C"],
| ... }
| ... )
| >>> lf.group_by_dynamic(
| ... "idx",
| ... every="2i",
| ... period="3i",
| ... include_boundaries=True,
| ... closed="right",
| ... ).agg(pl.col("A").alias("A_agg_list")).collect()
| shape: (4, 4)
| ┌─────────────────┬─────────────────┬─────┬─────────────────┐
| │ _lower_boundary ┆ _upper_boundary ┆ idx ┆ A_agg_list │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ list[str] │
| ╞═════════════════╪═════════════════╪═════╪═════════════════╡
| │ -2 ┆ 1 ┆ -2 ┆ ["A", "A"] │
| │ 0 ┆ 3 ┆ 0 ┆ ["A", "B", "B"] │
| │ 2 ┆ 5 ┆ 2 ┆ ["B", "B", "C"] │
| │ 4 ┆ 7 ┆ 4 ┆ ["C"] │
| └─────────────────┴─────────────────┴─────┴─────────────────┘
|
| head(self, n: 'int' = 5) -> 'LazyFrame'
| Get the first `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [7, 8, 9, 10, 11, 12],
| ... }
| ... )
| >>> lf.head().collect()
| shape: (5, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| │ 4 ┆ 10 │
| │ 5 ┆ 11 │
| └─────┴─────┘
| >>> lf.head(2).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| └─────┴─────┘
|
| inspect(self, fmt: 'str' = '{}') -> 'LazyFrame'
| Inspect a node in the computation graph.
|
| Print the value that this node in the computation graph evaluates to and pass on
| the value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"foo": [1, 1, -2, 3]})
| >>> (
| ... lf.with_columns(pl.col("foo").cum_sum().alias("bar"))
| ... .inspect() # print the node before the filter
| ... .filter(pl.col("bar") == pl.col("foo"))
| ... )
| <LazyFrame at ...>
|
| interpolate(self) -> 'LazyFrame'
| Interpolate intermediate values. The interpolation method is linear.
|
| Nulls at the beginning and end of the series remain null.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, None, 9, 10],
| ... "bar": [6, 7, 9, None],
| ... "baz": [1, None, None, 9],
| ... }
| ... )
| >>> lf.interpolate().collect()
| shape: (4, 3)
| ┌──────┬──────┬──────────┐
| │ foo ┆ bar ┆ baz │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪══════════╡
| │ 1.0 ┆ 6.0 ┆ 1.0 │
| │ 5.0 ┆ 7.0 ┆ 3.666667 │
| │ 9.0 ┆ 9.0 ┆ 6.333333 │
| │ 10.0 ┆ null ┆ 9.0 │
| └──────┴──────┴──────────┘
|
| join(self, other: 'LazyFrame', on: 'str | Expr | Sequence[str | Expr] | None' = None, how: 'JoinStrategy' = 'inner', *, left_on: 'str | Expr | Sequence[str | Expr] | None' = None, right_on: 'str | Expr | Sequence[str | Expr] | None' = None, suffix: 'str' = '_right', validate: 'JoinValidation' = 'm:m', nulls_equal: 'bool' = False, coalesce: 'bool | None' = None, maintain_order: 'MaintainOrderJoin | None' = None, build_side: 'JoinBuildSide' = 'auto', allow_parallel: 'bool' = True, force_parallel: 'bool' = False) -> 'LazyFrame'
| Add a join operation to the Logical Plan.
|
| .. versionchanged:: 1.24
| The `join_nulls` parameter was renamed `nulls_equal`.
|
| Parameters
| ----------
| other
| Lazy DataFrame to join with.
| on
| Name(s) of the join columns in both DataFrames. If set, `left_on` and
| `right_on` should be None. This should not be specified if `how='cross'`.
| how : {'inner','left', 'right', 'full', 'semi', 'anti', 'cross'}
| Join strategy.
|
| .. list-table ::
| :header-rows: 0
|
| * - **inner**
| - *(Default)* Returns rows that have matching values in both tables.
| * - **left**
| - Returns all rows from the left table, and the matched rows from
| the right table.
| * - **right**
| - Returns all rows from the right table, and the matched rows from
| the left table.
| * - **full**
| - Returns all rows from both tables, joining matching rows and
| filling non-matches with null values.
| * - **cross**
| - Returns the Cartesian product of rows from both tables
| * - **semi**
| - Returns rows from the left table that have a match in the right
| table. Does not return columns from the right table.
| * - **anti**
| - Returns rows from the left table that have no match in the right
| table. Does not return columns from the right table.
|
| left_on
| Join column of the left DataFrame.
| right_on
| Join column of the right DataFrame.
| suffix
| Suffix to append to columns with a duplicate name.
| validate: {'m:m', 'm:1', '1:m', '1:1'}
| Checks if join is of specified type.
|
| .. list-table ::
| :header-rows: 0
|
| * - **m:m**
| - *(Default)* Many-to-many. Does not result in checks.
| * - **1:1**
| - One-to-one. Checks if join keys are unique in both left and
| right datasets.
| * - **1:m**
| - One-to-many. Checks if join keys are unique in left dataset.
| * - **m:1**
| - Many-to-one. Check if join keys are unique in right dataset.
| nulls_equal
| Join on null values. By default null values will never produce matches.
| coalesce
| Coalescing behavior (merging of join columns).
|
| .. list-table ::
| :header-rows: 0
|
| * - **None**
| - *(Default)* Coalesce unless `how='full'` is specified.
| * - **True**
| - Always coalesce join columns.
| * - **False**
| - Never coalesce join columns.
|
| .. note::
| Joining on any other expressions than `col`
| will turn off coalescing.
| maintain_order : {'none', 'left', 'right', 'left_right', 'right_left'}
| Which DataFrame row order to preserve, if any.
| Do not rely on any observed ordering without explicitly setting this
| parameter, as your code may break in a future release.
| Not specifying any ordering can improve performance.
|
| .. list-table ::
| :header-rows: 0
|
| * - **none**
| - *(Default)* No specific ordering is desired. The ordering might
| differ across Polars versions or even between different runs.
| * - **left**
| - Preserves the order of the left DataFrame.
| * - **right**
| - Preserves the order of the right DataFrame.
| * - **left_right**
| - First preserves the order of the left DataFrame, then the right.
| * - **right_left**
| - First preserves the order of the right DataFrame, then the left.
| build_side: {'auto', 'prefer_left', 'prefer_right', 'force_left', 'force_right'}
| Which side of the join will be used as the build side. This side will be
| likely be held in memory as a hash table. Note that unless a `force_`
| variant is chosen, the chosen side might differ across Polars versions or
| even between different runs.
|
| .. list-table ::
| :header-rows: 0
|
| * - **auto**
| - *(Default)* Let Polars figure out the build side.
| * - **prefer_left**
| - Unless there's a very good reason to believe that the right side is
| smaller, use the left side.
| * - **prefer_right**
| - Unless there's a very good reason to believe that the left side is
| smaller, use the right side.
| * - **force_left**
| - Always use the left side.
| * - **force_right**
| - Always use the right side.
|
| .. warning::
| This functionality is considered **experimental**. It may be removed or
| changed at any point without it being considered a breaking change.
|
| allow_parallel
| Allow the physical plan to optionally evaluate the computation of both
| DataFrames up to the join in parallel.
| force_parallel
| Force the physical plan to evaluate the computation of both DataFrames up to
| the join in parallel.
|
| See Also
| --------
| join_asof
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> other_lf = pl.LazyFrame(
| ... {
| ... "apple": ["x", "y", "z"],
| ... "ham": ["a", "b", "d"],
| ... }
| ... )
| >>> lf.join(other_lf, on="ham").collect()
| shape: (2, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| └─────┴─────┴─────┴───────┘
| >>> lf.join(other_lf, on="ham", how="full").collect()
| shape: (4, 5)
| ┌──────┬──────┬──────┬───────┬───────────┐
| │ foo ┆ bar ┆ ham ┆ apple ┆ ham_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞══════╪══════╪══════╪═══════╪═══════════╡
| │ 1 ┆ 6.0 ┆ a ┆ x ┆ a │
| │ 2 ┆ 7.0 ┆ b ┆ y ┆ b │
| │ null ┆ null ┆ null ┆ z ┆ d │
| │ 3 ┆ 8.0 ┆ c ┆ null ┆ null │
| └──────┴──────┴──────┴───────┴───────────┘
| >>> lf.join(other_lf, on="ham", how="left", coalesce=True).collect()
| shape: (3, 4)
| ┌─────┬─────┬─────┬───────┐
| │ foo ┆ bar ┆ ham ┆ apple │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╡
| │ 1 ┆ 6.0 ┆ a ┆ x │
| │ 2 ┆ 7.0 ┆ b ┆ y │
| │ 3 ┆ 8.0 ┆ c ┆ null │
| └─────┴─────┴─────┴───────┘
| >>> lf.join(other_lf, on="ham", how="semi").collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 7.0 ┆ b │
| └─────┴─────┴─────┘
| >>> lf.join(other_lf, on="ham", how="anti").collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 3 ┆ 8.0 ┆ c │
| └─────┴─────┴─────┘
|
| >>> lf.join(other_lf, how="cross").collect()
| shape: (9, 5)
| ┌─────┬─────┬─────┬───────┬───────────┐
| │ foo ┆ bar ┆ ham ┆ apple ┆ ham_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str ┆ str ┆ str │
| ╞═════╪═════╪═════╪═══════╪═══════════╡
| │ 1 ┆ 6.0 ┆ a ┆ x ┆ a │
| │ 1 ┆ 6.0 ┆ a ┆ y ┆ b │
| │ 1 ┆ 6.0 ┆ a ┆ z ┆ d │
| │ 2 ┆ 7.0 ┆ b ┆ x ┆ a │
| │ 2 ┆ 7.0 ┆ b ┆ y ┆ b │
| │ 2 ┆ 7.0 ┆ b ┆ z ┆ d │
| │ 3 ┆ 8.0 ┆ c ┆ x ┆ a │
| │ 3 ┆ 8.0 ┆ c ┆ y ┆ b │
| │ 3 ┆ 8.0 ┆ c ┆ z ┆ d │
| └─────┴─────┴─────┴───────┴───────────┘
|
| join_asof(self, other: 'LazyFrame', *, left_on: 'str | None | Expr' = None, right_on: 'str | None | Expr' = None, on: 'str | None | Expr' = None, by_left: 'str | Sequence[str] | None' = None, by_right: 'str | Sequence[str] | None' = None, by: 'str | Sequence[str] | None' = None, strategy: 'AsofJoinStrategy' = 'backward', suffix: 'str' = '_right', tolerance: 'str | int | float | timedelta | None' = None, allow_parallel: 'bool' = True, force_parallel: 'bool' = False, coalesce: 'bool' = True, allow_exact_matches: 'bool' = True, check_sortedness: 'bool' = True) -> 'LazyFrame'
| Perform an asof join.
|
| This is similar to a left-join except that we match on nearest key rather than
| equal keys.
|
| Both DataFrames must be sorted by the `on` key (within each `by` group, if
| specified).
|
| For each row in the left DataFrame:
|
| - A "backward" search selects the last row in the right DataFrame whose
| 'on' key is less than or equal to the left's key.
|
| - A "forward" search selects the first row in the right DataFrame whose
| 'on' key is greater than or equal to the left's key.
|
| A "nearest" search selects the last row in the right DataFrame whose value
| is nearest to the left's key. String keys are not currently supported for a
| nearest search.
|
| The default is "backward".
|
| Parameters
| ----------
| other
| Lazy DataFrame to join with.
| left_on
| Join column of the left DataFrame.
| right_on
| Join column of the right DataFrame.
| on
| Join column of both DataFrames. If set, `left_on` and `right_on` should be
| None.
| by_left
| Join on these columns before doing asof join.
| by_right
| Join on these columns before doing asof join.
| by
| Join on these columns before doing asof join.
| strategy : {'backward', 'forward', 'nearest'}
| Join strategy.
| suffix
| Suffix to append to columns with a duplicate name.
| tolerance
| Numeric tolerance. By setting this the join will only be done if the near
| keys are within this distance. If an asof join is done on columns of dtype
| "Date", "Datetime", "Duration" or "Time", use either a datetime.timedelta
| object or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of
| ambiguity, we follow RFC-5545 and preserve the DST fold of the original
| datetime). Similarly for "calendar week", "calendar month",
| "calendar quarter", and "calendar year".
|
| allow_parallel
| Allow the physical plan to optionally evaluate the computation of both
| DataFrames up to the join in parallel.
| force_parallel
| Force the physical plan to evaluate the computation of both DataFrames up to
| the join in parallel.
| coalesce
| Coalescing behavior (merging of `on` / `left_on` / `right_on` columns):
|
| - True: -> Always coalesce join columns.
| - False: -> Never coalesce join columns.
|
| Note that joining on any other expressions than `col`
| will turn off coalescing.
| allow_exact_matches
| Whether exact matches are valid join predicates.
|
| - If True, allow matching with the same ``on`` value
| (i.e. less-than-or-equal-to / greater-than-or-equal-to)
| - If False, don't match the same ``on`` value
| (i.e., strictly less-than / strictly greater-than).
| check_sortedness
| Check the sortedness of the asof keys. If the keys are not sorted Polars
| will error. Currently, the `in-memory` engine cannot check the sortedness
| if 'by' groups are provided. The `streaming` engine will only check the
| sortedness of the rows it processes.
|
| Notes
| -----
| If 'by' is set, the implementation will compute the asof join over all of the
| groups concurrently. This can potentially lead to high memory usage if there
| are many groups.
|
| This can be mitigated by sorting (via :meth:`.sort() <polars.LazyFrame.sort>`)
| both of the input LazyFrames by the 'by' keys (or using :meth:`.set_sorted()
| <polars.LazyFrame.set_sorted>` if the columns are already sorted)
| before computing the join operation; and using the streaming engine to collect
| the results. For example:
|
| >>> # Compute streaming asof join with 'by' groups
| >>> result = (
| ... left.sort("by", "on").join_asof( # Sort left manually
| ... right.set_sorted("by", "on"), # Set right as already sorted
| ... )
| ... ).collect(streaming=True) # doctest: +SKIP
|
| Examples
| --------
| >>> from datetime import date
| >>> gdp = pl.LazyFrame(
| ... {
| ... "date": pl.date_range(
| ... date(2016, 1, 1),
| ... date(2020, 1, 1),
| ... "1y",
| ... eager=True,
| ... ),
| ... "gdp": [4164, 4411, 4566, 4696, 4827],
| ... }
| ... )
| >>> gdp.collect()
| shape: (5, 2)
| ┌────────────┬──────┐
| │ date ┆ gdp │
| │ --- ┆ --- │
| │ date ┆ i64 │
| ╞════════════╪══════╡
| │ 2016-01-01 ┆ 4164 │
| │ 2017-01-01 ┆ 4411 │
| │ 2018-01-01 ┆ 4566 │
| │ 2019-01-01 ┆ 4696 │
| │ 2020-01-01 ┆ 4827 │
| └────────────┴──────┘
|
| >>> population = pl.LazyFrame(
| ... {
| ... "date": [date(2016, 3, 1), date(2018, 8, 1), date(2019, 1, 1)],
| ... "population": [82.19, 82.66, 83.12],
| ... }
| ... ).sort("date")
| >>> population.collect()
| shape: (3, 2)
| ┌────────────┬────────────┐
| │ date ┆ population │
| │ --- ┆ --- │
| │ date ┆ f64 │
| ╞════════════╪════════════╡
| │ 2016-03-01 ┆ 82.19 │
| │ 2018-08-01 ┆ 82.66 │
| │ 2019-01-01 ┆ 83.12 │
| └────────────┴────────────┘
|
| Note how the dates don't quite match. If we join them using `join_asof` and
| `strategy='backward'`, then each date from `population` which doesn't have an
| exact match is matched with the closest earlier date from `gdp`:
|
| >>> population.join_asof(gdp, on="date", strategy="backward").collect()
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 4566 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2016-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2018-01-01` from `gdp`.
|
| You can verify this by passing `coalesce=False`:
|
| >>> population.join_asof(
| ... gdp, on="date", strategy="backward", coalesce=False
| ... ).collect()
| shape: (3, 4)
| ┌────────────┬────────────┬────────────┬──────┐
| │ date ┆ population ┆ date_right ┆ gdp │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ date ┆ i64 │
| ╞════════════╪════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 2016-01-01 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 2018-01-01 ┆ 4566 │
| │ 2019-01-01 ┆ 83.12 ┆ 2019-01-01 ┆ 4696 │
| └────────────┴────────────┴────────────┴──────┘
|
| If we instead use `strategy='forward'`, then each date from `population` which
| doesn't have an exact match is matched with the closest later date from `gdp`:
|
| >>> population.join_asof(gdp, on="date", strategy="forward").collect()
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4411 │
| │ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2017-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2019-01-01` from `gdp`.
|
| Finally, `strategy='nearest'` gives us a mix of the two results above, as each
| date from `population` which doesn't have an exact match is matched with the
| closest date from `gdp`, regardless of whether it's earlier or later:
|
| >>> population.join_asof(gdp, on="date", strategy="nearest").collect()
| shape: (3, 3)
| ┌────────────┬────────────┬──────┐
| │ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ date ┆ f64 ┆ i64 │
| ╞════════════╪════════════╪══════╡
| │ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ 2019-01-01 ┆ 83.12 ┆ 4696 │
| └────────────┴────────────┴──────┘
|
| Note how:
|
| - date `2016-03-01` from `population` is matched with `2016-01-01` from `gdp`;
| - date `2018-08-01` from `population` is matched with `2019-01-01` from `gdp`.
|
| They `by` argument allows joining on another column first, before the asof join.
| In this example we join by `country` first, then asof join by date, as above.
|
| >>> gdp_dates = pl.date_range( # fmt: skip
| ... date(2016, 1, 1), date(2020, 1, 1), "1y", eager=True
| ... )
| >>> gdp2 = pl.LazyFrame(
| ... {
| ... "country": ["Germany"] * 5 + ["Netherlands"] * 5,
| ... "date": pl.concat([gdp_dates, gdp_dates]),
| ... "gdp": [4164, 4411, 4566, 4696, 4827, 784, 833, 914, 910, 909],
| ... }
| ... ).sort("country", "date")
| >>>
| >>> gdp2.collect()
| shape: (10, 3)
| ┌─────────────┬────────────┬──────┐
| │ country ┆ date ┆ gdp │
| │ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 │
| ╞═════════════╪════════════╪══════╡
| │ Germany ┆ 2016-01-01 ┆ 4164 │
| │ Germany ┆ 2017-01-01 ┆ 4411 │
| │ Germany ┆ 2018-01-01 ┆ 4566 │
| │ Germany ┆ 2019-01-01 ┆ 4696 │
| │ Germany ┆ 2020-01-01 ┆ 4827 │
| │ Netherlands ┆ 2016-01-01 ┆ 784 │
| │ Netherlands ┆ 2017-01-01 ┆ 833 │
| │ Netherlands ┆ 2018-01-01 ┆ 914 │
| │ Netherlands ┆ 2019-01-01 ┆ 910 │
| │ Netherlands ┆ 2020-01-01 ┆ 909 │
| └─────────────┴────────────┴──────┘
| >>> pop2 = pl.LazyFrame(
| ... {
| ... "country": ["Germany"] * 3 + ["Netherlands"] * 3,
| ... "date": [
| ... date(2016, 3, 1),
| ... date(2018, 8, 1),
| ... date(2019, 1, 1),
| ... date(2016, 3, 1),
| ... date(2018, 8, 1),
| ... date(2019, 1, 1),
| ... ],
| ... "population": [82.19, 82.66, 83.12, 17.11, 17.32, 17.40],
| ... }
| ... ).sort("country", "date")
| >>>
| >>> pop2.collect()
| shape: (6, 3)
| ┌─────────────┬────────────┬────────────┐
| │ country ┆ date ┆ population │
| │ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ f64 │
| ╞═════════════╪════════════╪════════════╡
| │ Germany ┆ 2016-03-01 ┆ 82.19 │
| │ Germany ┆ 2018-08-01 ┆ 82.66 │
| │ Germany ┆ 2019-01-01 ┆ 83.12 │
| │ Netherlands ┆ 2016-03-01 ┆ 17.11 │
| │ Netherlands ┆ 2018-08-01 ┆ 17.32 │
| │ Netherlands ┆ 2019-01-01 ┆ 17.4 │
| └─────────────┴────────────┴────────────┘
| >>> pop2.join_asof(gdp2, by="country", on="date", strategy="nearest").collect()
| shape: (6, 4)
| ┌─────────────┬────────────┬────────────┬──────┐
| │ country ┆ date ┆ population ┆ gdp │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ f64 ┆ i64 │
| ╞═════════════╪════════════╪════════════╪══════╡
| │ Germany ┆ 2016-03-01 ┆ 82.19 ┆ 4164 │
| │ Germany ┆ 2018-08-01 ┆ 82.66 ┆ 4696 │
| │ Germany ┆ 2019-01-01 ┆ 83.12 ┆ 4696 │
| │ Netherlands ┆ 2016-03-01 ┆ 17.11 ┆ 784 │
| │ Netherlands ┆ 2018-08-01 ┆ 17.32 ┆ 910 │
| │ Netherlands ┆ 2019-01-01 ┆ 17.4 ┆ 910 │
| └─────────────┴────────────┴────────────┴──────┘
|
| join_where(self, other: 'LazyFrame', *predicates: 'Expr | Iterable[Expr]', suffix: 'str' = '_right') -> 'LazyFrame'
| Perform a join based on one or multiple (in)equality predicates.
|
| This performs an inner join, so only rows where all predicates are true
| are included in the result, and a row from either DataFrame may be included
| multiple times in the result.
|
| .. note::
| The row order of the input DataFrames is not preserved.
|
| .. warning::
| This functionality is experimental. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| other
| DataFrame to join with.
| *predicates
| (In)Equality condition to join the two tables on.
| When a column name occurs in both tables, the proper suffix must
| be applied in the predicate.
| suffix
| Suffix to append to columns with a duplicate name.
|
| Examples
| --------
| Join two lazyframes together based on two predicates which get AND-ed together.
|
| >>> east = pl.LazyFrame(
| ... {
| ... "id": [100, 101, 102],
| ... "dur": [120, 140, 160],
| ... "rev": [12, 14, 16],
| ... "cores": [2, 8, 4],
| ... }
| ... )
| >>> west = pl.LazyFrame(
| ... {
| ... "t_id": [404, 498, 676, 742],
| ... "time": [90, 130, 150, 170],
| ... "cost": [9, 13, 15, 16],
| ... "cores": [4, 2, 1, 4],
| ... }
| ... )
| >>> east.join_where(
| ... west,
| ... pl.col("dur") < pl.col("time"),
| ... pl.col("rev") < pl.col("cost"),
| ... ).collect()
| shape: (5, 8)
| ┌─────┬─────┬─────┬───────┬──────┬──────┬──────┬─────────────┐
| │ id ┆ dur ┆ rev ┆ cores ┆ t_id ┆ time ┆ cost ┆ cores_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╪══════╪══════╪══════╪═════════════╡
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 498 ┆ 130 ┆ 13 ┆ 2 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| └─────┴─────┴─────┴───────┴──────┴──────┴──────┴─────────────┘
|
| To OR them together, use a single expression and the `|` operator.
|
| >>> east.join_where(
| ... west,
| ... (pl.col("dur") < pl.col("time")) | (pl.col("rev") < pl.col("cost")),
| ... ).collect()
| shape: (6, 8)
| ┌─────┬─────┬─────┬───────┬──────┬──────┬──────┬─────────────┐
| │ id ┆ dur ┆ rev ┆ cores ┆ t_id ┆ time ┆ cost ┆ cores_right │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╪══════╪══════╪══════╪═════════════╡
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 498 ┆ 130 ┆ 13 ┆ 2 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 100 ┆ 120 ┆ 12 ┆ 2 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 676 ┆ 150 ┆ 15 ┆ 1 │
| │ 101 ┆ 140 ┆ 14 ┆ 8 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| │ 102 ┆ 160 ┆ 16 ┆ 4 ┆ 742 ┆ 170 ┆ 16 ┆ 4 │
| └─────┴─────┴─────┴───────┴──────┴──────┴──────┴─────────────┘
|
| last(self) -> 'LazyFrame'
| Get the last row of the DataFrame.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 5, 3],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> lf.last().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 6 │
| └─────┴─────┘
|
| lazy(self) -> 'LazyFrame'
| Return lazy representation, i.e. itself.
|
| Useful for writing code that expects either a :class:`DataFrame` or
| :class:`LazyFrame`. On LazyFrame this is a no-op, and returns the same object.
|
| Returns
| -------
| LazyFrame
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [None, 2, 3, 4],
| ... "b": [0.5, None, 2.5, 13],
| ... "c": [True, True, False, None],
| ... }
| ... )
| >>> lf.lazy()
| <LazyFrame at ...>
|
| limit(self, n: 'int' = 5) -> 'LazyFrame'
| Get the first `n` rows.
|
| Alias for :func:`LazyFrame.head`.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [7, 8, 9, 10, 11, 12],
| ... }
| ... )
| >>> lf.limit().collect()
| shape: (5, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| │ 4 ┆ 10 │
| │ 5 ┆ 11 │
| └─────┴─────┘
| >>> lf.limit(2).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| └─────┴─────┘
|
| map_batches(self, function: 'Callable[[DataFrame], DataFrame]', *, predicate_pushdown: 'bool' = False, projection_pushdown: 'bool' = False, slice_pushdown: 'bool' = False, no_optimizations: 'bool | None' = None, schema: 'None | SchemaDict' = None, validate_output_schema: 'bool' = True, streamable: 'bool' = False) -> 'LazyFrame'
| Apply a custom function.
|
| It is important that the function returns a Polars DataFrame.
|
| Parameters
| ----------
| function
| Lambda/ function to apply.
| predicate_pushdown
| Allow predicate pushdown optimization to pass this node.
| projection_pushdown
| Allow projection pushdown optimization to pass this node.
| slice_pushdown
| Allow slice pushdown optimization to pass this node.
| no_optimizations
| .. deprecated:: 1.30.0
| This parameter is deprecated and will be removed in a future version.
| The `_pushdown` parameters now default to `False`, so this parameter
| is no longer needed.
| schema
| Output schema of the function, if set to `None` we assume that the schema
| will remain unchanged by the applied function.
| validate_output_schema
| It is paramount that polars' schema is correct. This flag will ensure that
| the output schema of this function will be checked with the expected schema.
| Setting this to `False` will not do this check, but may lead to hard to
| debug bugs.
| streamable
| Whether the function that is given is eligible to be running with the
| streaming engine. That means that the function must produce the same result
| when it is executed in batches or when it is be executed on the full
| dataset.
|
| Warnings
| --------
| The `schema` of a `LazyFrame` must always be correct. It is up to the caller
| of this function to ensure that this invariant is upheld.
|
| It is important that the optimization flags are correct. If the custom function
| for instance does an aggregation of a column, `predicate_pushdown` should not
| be allowed, as this prunes rows and will influence your aggregation results.
|
| Notes
| -----
| A UDF passed to `map_batches` must be pure, meaning that it cannot modify or
| depend on state other than its arguments.
|
| Examples
| --------
| >>> lf = ( # doctest: +SKIP
| ... pl.LazyFrame(
| ... {
| ... "a": pl.int_range(-100_000, 0, eager=True),
| ... "b": pl.int_range(0, 100_000, eager=True),
| ... }
| ... )
| ... .map_batches(lambda x: 2 * x, streamable=True)
| ... .collect(engine="streaming")
| ... )
| shape: (100_000, 2)
| ┌─────────┬────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════════╪════════╡
| │ -200000 ┆ 0 │
| │ -199998 ┆ 2 │
| │ -199996 ┆ 4 │
| │ -199994 ┆ 6 │
| │ … ┆ … │
| │ -8 ┆ 199992 │
| │ -6 ┆ 199994 │
| │ -4 ┆ 199996 │
| │ -2 ┆ 199998 │
| └─────────┴────────┘
|
| match_to_schema(self, schema: 'SchemaDict | Schema', *, missing_columns: "Literal['insert', 'raise'] | Mapping[str, Literal['insert', 'raise'] | Expr] | Expr" = 'raise', missing_struct_fields: "Literal['insert', 'raise'] | Mapping[str, Literal['insert', 'raise']]" = 'raise', extra_columns: "Literal['ignore', 'raise']" = 'raise', extra_struct_fields: "Literal['ignore', 'raise'] | Mapping[str, Literal['ignore', 'raise']]" = 'raise', integer_cast: "Literal['upcast', 'forbid'] | Mapping[str, Literal['upcast', 'forbid']]" = 'forbid', float_cast: "Literal['upcast', 'forbid'] | Mapping[str, Literal['upcast', 'forbid']]" = 'forbid') -> 'LazyFrame'
| Match or evolve the schema of a LazyFrame into a specific schema.
|
| By default, match_to_schema returns an error if the input schema does not
| exactly match the target schema. It also allows columns to be freely reordered,
| with additional coercion rules available through optional parameters.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| schema
| Target schema to match or evolve to.
| missing_columns
| Raise of insert missing columns from the input with respect to the `schema`.
|
| This can also be an expression per column with what to insert if it is
| missing.
| missing_struct_fields
| Raise of insert missing struct fields from the input with respect to the
| `schema`.
| extra_columns
| Raise of ignore extra columns from the input with respect to the `schema`.
| extra_struct_fields
| Raise of ignore extra struct fields from the input with respect to the
| `schema`.
| integer_cast
| Forbid of upcast for integer columns from the input to the respective column
| in `schema`.
| float_cast
| Forbid of upcast for float columns from the input to the respective column
| in `schema`.
|
| Examples
| --------
| Ensuring the schema matches
|
| >>> lf = pl.LazyFrame({"a": [1, 2, 3], "b": ["A", "B", "C"]})
| >>> lf.match_to_schema({"a": pl.Int64, "b": pl.String}).collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ A │
| │ 2 ┆ B │
| │ 3 ┆ C │
| └─────┴─────┘
| >>> (lf.match_to_schema({"a": pl.Int64}).collect()) # doctest: +SKIP
| polars.exceptions.SchemaError: extra columns in `match_to_schema`: "b"
|
| Adding missing columns
|
| >>> (
| ... pl.LazyFrame({"a": [1, 2, 3]})
| ... .match_to_schema(
| ... {"a": pl.Int64, "b": pl.String},
| ... missing_columns="insert",
| ... )
| ... .collect()
| ... )
| shape: (3, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪══════╡
| │ 1 ┆ null │
| │ 2 ┆ null │
| │ 3 ┆ null │
| └─────┴──────┘
| >>> (
| ... pl.LazyFrame({"a": [1, 2, 3]})
| ... .match_to_schema(
| ... {"a": pl.Int64, "b": pl.String},
| ... missing_columns={"b": pl.col.a.cast(pl.String)},
| ... )
| ... .collect()
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ str │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| └─────┴─────┘
|
| Removing extra columns
|
| >>> (
| ... pl.LazyFrame({"a": [1, 2, 3], "b": ["A", "B", "C"]})
| ... .match_to_schema(
| ... {"a": pl.Int64},
| ... extra_columns="ignore",
| ... )
| ... .collect()
| ... )
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Upcasting integers and floats
|
| >>> (
| ... pl.LazyFrame(
| ... {"a": [1, 2, 3], "b": [1.0, 2.0, 3.0]},
| ... schema={"a": pl.Int32, "b": pl.Float32},
| ... )
| ... .match_to_schema(
| ... {"a": pl.Int64, "b": pl.Float64},
| ... integer_cast="upcast",
| ... float_cast="upcast",
| ... )
| ... .collect()
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 1.0 │
| │ 2 ┆ 2.0 │
| │ 3 ┆ 3.0 │
| └─────┴─────┘
|
| max(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their maximum value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.max().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| mean(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their mean value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.mean().collect()
| shape: (1, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════╡
| │ 2.5 ┆ 1.25 │
| └─────┴──────┘
|
| median(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their median value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.median().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 2.5 ┆ 1.0 │
| └─────┴─────┘
|
| melt(self, id_vars: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, value_vars: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, variable_name: 'str | None' = None, value_name: 'str | None' = None, *, streamable: 'bool' = True) -> 'LazyFrame'
| Unpivot a DataFrame from wide to long format.
|
| Optionally leaves identifiers set.
|
| This function is useful to massage a DataFrame into a format where one or more
| columns are identifier variables (id_vars) while all other columns, considered
| measured variables (value_vars), are "unpivoted" to the row axis leaving just
| two non-identifier columns, 'variable' and 'value'.
|
| .. deprecated:: 1.0.0
| Use the :meth:`.unpivot` method instead.
|
| Parameters
| ----------
| id_vars
| Column(s) or selector(s) to use as identifier variables.
| value_vars
| Column(s) or selector(s) to use as values variables; if `value_vars`
| is empty all columns that are not in `id_vars` will be used.
| variable_name
| Name to give to the `variable` column. Defaults to "variable"
| value_name
| Name to give to the `value` column. Defaults to "value"
| streamable
| Allow this node to run in the streaming engine.
| If this runs in streaming, the output of the unpivot operation
| will not have a stable ordering.
|
| merge_sorted(self, other: 'LazyFrame', key: 'str | Sequence[str]', *, maintain_order: 'bool' = False) -> 'LazyFrame'
| Take two sorted DataFrames and merge them by the sorted key.
|
| The output of this operation will also be sorted.
| It is the callers responsibility that the frames
| are sorted in ascending order by the key(s), with null
| keys at the end, otherwise the order of the output
| will not make sense.
|
| The schemas of both LazyFrames must be equal.
|
| Parameters
| ----------
| other
| Other DataFrame that must be merged
| key
| Key column(s) that the frames are sorted by. A single column name or a
| sequence of column names can be passed. When multiple keys are given the
| frames are merged as if sorted by those keys in order.
| maintain_order
| If ``True``, the output is guaranteed to have left-biased ordering
| for equal keys: rows from the left frame appear before rows from
| the right frame when their keys are equal.
|
| Examples
| --------
| >>> df0 = pl.LazyFrame(
| ... {"name": ["steve", "elise", "bob"], "age": [42, 44, 18]}
| ... ).sort("age")
| >>> df0.collect()
| shape: (3, 2)
| ┌───────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═════╡
| │ bob ┆ 18 │
| │ steve ┆ 42 │
| │ elise ┆ 44 │
| └───────┴─────┘
| >>> df1 = pl.LazyFrame(
| ... {"name": ["anna", "megan", "steve", "thomas"], "age": [21, 33, 42, 20]}
| ... ).sort("age")
| >>> df1.collect()
| shape: (4, 2)
| ┌────────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞════════╪═════╡
| │ thomas ┆ 20 │
| │ anna ┆ 21 │
| │ megan ┆ 33 │
| │ steve ┆ 42 │
| └────────┴─────┘
| >>> df0.merge_sorted(df1, key="age").collect()
| shape: (7, 2)
| ┌────────┬─────┐
| │ name ┆ age │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞════════╪═════╡
| │ bob ┆ 18 │
| │ thomas ┆ 20 │
| │ anna ┆ 21 │
| │ megan ┆ 33 │
| │ steve ┆ 42 │
| │ steve ┆ 42 │
| │ elise ┆ 44 │
| └────────┴─────┘
|
| Multiple keys can be passed to merge frames sorted by a composite key. The
| frames are merged as if sorted by ``key_1``, then ``key_2``.
|
| >>> df0 = pl.LazyFrame({"key_1": [1, 1, 3], "key_2": [1, 4, 2]})
| >>> df1 = pl.LazyFrame({"key_1": [1, 2, 3], "key_2": [2, 1, 1]})
| >>> df0.merge_sorted(df1, key=["key_1", "key_2"]).collect()
| shape: (6, 2)
| ┌───────┬───────┐
| │ key_1 ┆ key_2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪═══════╡
| │ 1 ┆ 1 │
| │ 1 ┆ 2 │
| │ 1 ┆ 4 │
| │ 2 ┆ 1 │
| │ 3 ┆ 1 │
| │ 3 ┆ 2 │
| └───────┴───────┘
|
| Notes
| -----
| Unless ``maintain_order=True``, no guarantee is given over the output
| row order when the key is equal between the both dataframes.
|
| The key(s) must be sorted in ascending order.
|
| min(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their minimum value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.min().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| └─────┴─────┘
|
| null_count(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame as the sum of their null value count.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, None, 3],
| ... "bar": [6, 7, None],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.null_count().collect()
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 1 ┆ 0 │
| └─────┴─────┴─────┘
|
| pipe(self, function: 'Callable[Concatenate[LazyFrame, P], T]', *args: 'P.args', **kwargs: 'P.kwargs') -> 'T'
| Offers a structured way to apply a sequence of user-defined functions (UDFs).
|
| Parameters
| ----------
| function
| Callable; will receive the frame as the first parameter,
| followed by any given args/kwargs.
| *args
| Arguments to pass to the UDF.
| **kwargs
| Keyword arguments to pass to the UDF.
|
| See Also
| --------
| pipe_with_schema
|
| Examples
| --------
| >>> def cast_str_to_int(lf: pl.LazyFrame, col_name: str) -> pl.LazyFrame:
| ... return lf.with_columns(pl.col(col_name).cast(pl.Int64))
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": ["10", "20", "30", "40"],
| ... }
| ... )
| >>> lf.pipe(cast_str_to_int, col_name="b").collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 10 │
| │ 2 ┆ 20 │
| │ 3 ┆ 30 │
| │ 4 ┆ 40 │
| └─────┴─────┘
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "b": [1, 2],
| ... "a": [3, 4],
| ... }
| ... )
| >>> lf.collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ b ┆ a │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| └─────┴─────┘
| >>> lf.pipe(lambda lf: lf.select(sorted(lf.collect_schema()))).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 1 │
| │ 4 ┆ 2 │
| └─────┴─────┘
|
| pipe_with_schema(self, function: 'Callable[[LazyFrame, Schema], LazyFrame]') -> 'LazyFrame'
| Allows to alter the lazy frame during the plan stage with the resolved schema.
|
| In contrast to `pipe`, this method does not execute `function` immediately but
| only during the plan stage. This allows using the resolved schema of the input
| to dynamically alter the lazy frame. This also means that any exceptions raised
| by `function` will only be emitted during the plan stage.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
|
| Parameters
| ----------
| function
| Callable; will receive the frame as the first parameter and the resolved
| schema as the second parameter.
|
| See Also
| --------
| pipe
|
| Examples
| --------
| >>> def cast_to_float_if_necessary(
| ... lf: pl.LazyFrame, schema: pl.Schema
| ... ) -> pl.LazyFrame:
| ... required_casts = [
| ... pl.col(name).cast(pl.Float64)
| ... for name, dtype in schema.items()
| ... if not dtype.is_float()
| ... ]
| ... return lf.with_columns(required_casts)
| >>> lf = pl.LazyFrame(
| ... {"a": [1.0, 2.0], "b": ["1.0", "2.5"], "c": [2.0, 3.0]},
| ... schema={"a": pl.Float64, "b": pl.String, "c": pl.Float32},
| ... )
| >>> lf.pipe_with_schema(cast_to_float_if_necessary).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f32 │
| ╞═════╪═════╪═════╡
| │ 1.0 ┆ 1.0 ┆ 2.0 │
| │ 2.0 ┆ 2.5 ┆ 3.0 │
| └─────┴─────┴─────┘
|
| pivot(self, on: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector]', on_columns: 'Sequence[Any] | pl.Series | pl.DataFrame', *, index: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, values: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, aggregate_function: 'PivotAgg | Expr | None' = None, maintain_order: 'bool' = False, separator: 'str' = '_', column_naming: "Literal['auto', 'combine']" = 'auto') -> 'LazyFrame'
| Create a spreadsheet-style pivot table as a DataFrame.
|
| Parameters
| ----------
| on
| The column(s) whose values will be used as the new columns of the output
| DataFrame.
| on_columns
| What value combinations will be considered for the output table.
| index
| The column(s) that remain from the input to the output. The output DataFrame will have one row
| for each unique combination of the `index`'s values.
| If None, all remaining columns not specified on `on` and `values` will be used. At least one
| of `index` and `values` must be specified.
| values
| The existing column(s) of values which will be moved under the new columns from index. If an
| aggregation is specified, these are the values on which the aggregation will be computed.
| If None, all remaining columns not specified on `on` and `index` will be used.
| At least one of `index` and `values` must be specified.
| aggregate_function
| Choose from:
|
| - None: no aggregation takes place, will raise error if multiple values are in group.
| - A predefined aggregate function string, one of
| {'min', 'max', 'first', 'last', 'sum', 'mean', 'median', 'len', 'item'}
| - An expression to do the aggregation. The expression can only access data from the respective
| 'values' columns as generated by pivot, through `pl.element()`.
| maintain_order
| Ensure the values of `index` are sorted by discovery order.
| separator
| Used as separator/delimiter in generated column names in case of multiple
| `values` columns.
| column_naming : {'auto', 'combine'}
| How resulting column names will be constructed.
|
| * 'auto': The default; combine with separator if there are multiple
| `values` columns, otherwise just use the `on_columns` names.
| * 'combine': Always combine the `values` columns' names with
| the `on_columns` names.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Notes
| -----
| In some other frameworks, you might know this operation as `pivot_wider`.
|
| Examples
| --------
| You can use `pivot` to reshape a dataframe from "long" to "wide" format.
|
| For example, suppose we have a dataframe of test scores achieved by some
| students, where each row represents a distinct test.
|
| >>> df = pl.DataFrame(
| ... {
| ... "name": ["Cady", "Cady", "Karen", "Karen"],
| ... "subject": ["maths", "physics", "maths", "physics"],
| ... "test_1": [98, 99, 61, 58],
| ... "test_2": [100, 100, 60, 60],
| ... }
| ... )
| >>> df
| shape: (4, 4)
| ┌───────┬─────────┬────────┬────────┐
| │ name ┆ subject ┆ test_1 ┆ test_2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 ┆ i64 │
| ╞═══════╪═════════╪════════╪════════╡
| │ Cady ┆ maths ┆ 98 ┆ 100 │
| │ Cady ┆ physics ┆ 99 ┆ 100 │
| │ Karen ┆ maths ┆ 61 ┆ 60 │
| │ Karen ┆ physics ┆ 58 ┆ 60 │
| └───────┴─────────┴────────┴────────┘
|
| Using `pivot`, we can reshape so we have one row per student, with different
| subjects as columns, and their `test_1` scores as values:
|
| >>> df.lazy().pivot(
| ... "subject",
| ... on_columns=["maths", "physics"],
| ... index="name",
| ... values="test_1",
| ... ).collect() # doctest: +IGNORE_RESULT
| shape: (2, 3)
| ┌───────┬───────┬─────────┐
| │ name ┆ maths ┆ physics │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════╪═══════╪═════════╡
| │ Cady ┆ 98 ┆ 99 │
| │ Karen ┆ 61 ┆ 58 │
| └───────┴───────┴─────────┘
|
| You can use selectors too - here we include all test scores in the pivoted table:
|
| >>> import polars.selectors as cs
| >>> df.lazy().pivot(
| ... "subject",
| ... on_columns=["maths", "physics"],
| ... values=cs.starts_with("test"),
| ... ).collect() # doctest: +IGNORE_RESULT
| shape: (2, 5)
| ┌───────┬──────────────┬────────────────┬──────────────┬────────────────┐
| │ name ┆ test_1_maths ┆ test_1_physics ┆ test_2_maths ┆ test_2_physics │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═══════╪══════════════╪════════════════╪══════════════╪════════════════╡
| │ Cady ┆ 98 ┆ 99 ┆ 100 ┆ 100 │
| │ Karen ┆ 61 ┆ 58 ┆ 60 ┆ 60 │
| └───────┴──────────────┴────────────────┴──────────────┴────────────────┘
|
| If you end up with multiple values per cell, you can specify how to aggregate
| them with `aggregate_function`:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "ix": [1, 1, 2, 2, 1, 2],
| ... "col": ["a", "a", "a", "a", "b", "b"],
| ... "foo": [0, 1, 2, 2, 7, 1],
| ... "bar": [0, 2, 0, 0, 9, 4],
| ... }
| ... )
| >>> lf.pivot(
| ... "col", on_columns=["a", "b"], index="ix", aggregate_function="sum"
| ... ).collect() # doctest: +IGNORE_RESULT
| shape: (2, 5)
| ┌─────┬───────┬───────┬───────┬───────┐
| │ ix ┆ foo_a ┆ foo_b ┆ bar_a ┆ bar_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═══════╪═══════╪═══════╪═══════╡
| │ 1 ┆ 1 ┆ 7 ┆ 2 ┆ 9 │
| │ 2 ┆ 4 ┆ 1 ┆ 0 ┆ 4 │
| └─────┴───────┴───────┴───────┴───────┘
|
| You can also pass a custom aggregation function using
| :meth:`polars.element`:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "col1": ["a", "a", "a", "b", "b", "b"],
| ... "col2": ["x", "x", "x", "x", "y", "y"],
| ... "col3": [6, 7, 3, 2, 5, 7],
| ... }
| ... )
| >>> lf.pivot(
| ... "col2",
| ... on_columns=["x", "y"],
| ... index="col1",
| ... values="col3",
| ... aggregate_function=pl.element().tanh().mean(),
| ... ).collect() # doctest: +IGNORE_RESULT
| shape: (2, 3)
| ┌──────┬──────────┬──────────┐
| │ col1 ┆ x ┆ y │
| │ --- ┆ --- ┆ --- │
| │ str ┆ f64 ┆ f64 │
| ╞══════╪══════════╪══════════╡
| │ a ┆ 0.998347 ┆ null │
| │ b ┆ 0.964028 ┆ 0.999954 │
| └──────┴──────────┴──────────┘
|
| profile(self, *, type_coercion: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, no_optimization: 'bool' = False, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, show_plot: 'bool' = False, truncate_nodes: 'int' = 0, figsize: 'tuple[int, int]' = (18, 8), engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, **_kwargs: 'Any') -> 'tuple[DataFrame, DataFrame]'
| Profile a LazyFrame.
|
| .. deprecated:: 1.43.0
| It was made for the older in-memory engine, but from version 2.0,
| Polars uses a streaming engine by default. Due to the concurrent
| nature of the streaming engine, the profiling information from this
| function would be misleading.
|
| This will run the query and return a tuple
| containing the materialized DataFrame and a DataFrame that
| contains profiling information of each node that is executed.
|
| The units of the timings are microseconds.
|
| Parameters
| ----------
| type_coercion
| Do type coercion optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| predicate_pushdown
| Do predicate pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| projection_pushdown
| Do projection pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| simplify_expression
| Run simplify expressions optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| no_optimization
| Turn off (certain) optimizations.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| slice_pushdown
| Slice pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subplan_elim
| Will try to cache branching subplans that occur on self-joins or unions.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subexpr_elim
| Common subexpressions will be cached and reused.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| cluster_with_columns
| Combine sequential independent calls to with_columns
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| collapse_joins
| Collapse a join and filters into a faster join
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| show_plot
| Show a gantt chart of the profiling result
| truncate_nodes
| Truncate the label lengths in the gantt chart to this number of
| characters.
| figsize
| matplotlib figsize of the profiling plot
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort(
| ... "a"
| ... ).profile() # doctest: +SKIP
| (shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ a ┆ 4 ┆ 10 │
| │ b ┆ 11 ┆ 10 │
| │ c ┆ 6 ┆ 1 │
| └─────┴─────┴─────┘,
| shape: (3, 3)
| ┌─────────────────────────┬───────┬──────┐
| │ node ┆ start ┆ end │
| │ --- ┆ --- ┆ --- │
| │ str ┆ u64 ┆ u64 │
| ╞═════════════════════════╪═══════╪══════╡
| │ optimization ┆ 0 ┆ 5 │
| │ group_by_partitioned(a) ┆ 5 ┆ 470 │
| │ sort(a) ┆ 475 ┆ 1964 │
| └─────────────────────────┴───────┴──────┘)
|
| quantile(self, quantile: 'float | Expr', interpolation: 'QuantileMethod' = 'nearest') -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their quantile value.
|
| Parameters
| ----------
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.quantile(0.7).collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 3.0 ┆ 1.0 │
| └─────┴─────┘
|
| remote(self, context: 'pc.ClientContext | None' = None, *, plan_type: 'pc._typing.PlanTypePreference' = 'dot', n_retries: 'int' = 0, engine: 'pc._typing.Engine' = 'auto', scaling_mode: 'pc._typing.ScalingMode' = 'auto') -> 'pc.LazyFrameRemote'
| Run a query remotely on Polars Cloud.
|
| This allows you to run Polars remotely on
| one or more workers via several strategies
| for distributed compute.
|
| Read more in the `Announcement post <https://pola.rs/posts/polars-cloud-what-we-are-building/>`_
|
| Parameters
| ----------
| context
| Compute context in which queries are executed.
| If none given, it will take the default context.
| plan_type: {'plain', 'dot'}
| Whether to give a dot diagram of a plain text
| version of logical plan.
| n_retries:
| How often a stage should be retried on failure.
| engine: {'auto', 'streaming', 'in-memory'}
| This will serve as a hint that tells Polars which engine
| to prefer. It doesn't have to be respected.
| scaling_mode: {'auto', 'single-node', 'distributed'}
| If set to auto, a query that doesn't explicitly specify
| a scaling mode via `remote().distributed()` or
| `remote().single_node()` will run in distributed mode
| if the cluster has more than 1 node.
|
|
| Examples
| --------
| Run a query on a cloud instance.
|
| >>> lf = pl.LazyFrame([1, 2, 3]).sum()
| >>> in_progress = lf.remote().execute(blocking=False) # doctest: +SKIP
| >>> # do some other work
| >>> in_progress.await_result() # doctest: +SKIP
| shape: (1, 1)
| ┌──────────┐
| │ column_0 │
| │ --- │
| │ i64 │
| ╞══════════╡
| │ 6 │
| └──────────┘
|
| Explicitly run a query distributed.
|
| >>> lf = (
| ... pl.scan_parquet("s3://my_bucket/").group_by("key").agg(pl.sum("values"))
| ... )
| >>> result = lf.remote().distributed().execute() # doctest: +SKIP
| shape: (1, 1)
| ┌──────────┐
| │ column_0 │
| │ --- │
| │ i64 │
| ╞══════════╡
| │ 6 │
| └──────────┘
|
| remove(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool | list[bool] | np.ndarray[Any, Any]', **constraints: 'Any') -> 'LazyFrame'
| Remove rows, dropping those that match the given predicate expression(s).
|
| The original order of the remaining rows is preserved.
|
| Rows where the filter predicate does not evaluate to True are retained
| (this includes rows where the predicate evaluates as `null`).
|
| Parameters
| ----------
| predicates
| Expression(s) that evaluate to a boolean Series. When multiple predicates
| are provided they are combined using `&` (logical AND), so a row is only
| removed when *every* predicate evaluates to True for that row.
| constraints
| Column filters; use `name = value` to filter columns using the supplied
| value. Each constraint behaves the same as `pl.col(name).eq(value)`,
| and is implicitly joined with the other filter conditions using `&`.
|
| Notes
| -----
| If you are transitioning from Pandas, and performing filter operations based on
| the comparison of two or more columns, please note that in Polars any comparison
| involving `null` values will result in a `null` result, *not* boolean True or
| False. As a result, these rows will not be removed. Ensure that null values
| are handled appropriately to avoid unexpected behaviour (see examples below).
|
| See Also
| --------
| filter
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [2, 3, None, 4, 0],
| ... "bar": [5, 6, None, None, 0],
| ... "ham": ["a", "b", None, "c", "d"],
| ... }
| ... )
|
| Remove rows matching a condition:
|
| >>> lf.remove(
| ... pl.col("bar") >= 5,
| ... ).collect()
| shape: (3, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| │ 0 ┆ 0 ┆ d │
| └──────┴──────┴──────┘
|
| Discard rows based on multiple conditions, combined with and/or operators:
|
| >>> lf.remove(
| ... (pl.col("foo") >= 0) & (pl.col("bar") >= 0),
| ... ).collect()
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| >>> lf.remove(
| ... (pl.col("foo") >= 0) | (pl.col("bar") >= 0),
| ... ).collect()
| shape: (1, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| └──────┴──────┴──────┘
|
| Provide multiple constraints using `*args` syntax:
|
| >>> lf.remove(
| ... pl.col("ham").is_not_null(),
| ... pl.col("bar") >= 0,
| ... ).collect()
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| Provide constraints(s) using `**kwargs` syntax:
|
| >>> lf.remove(foo=0, bar=0).collect()
| shape: (4, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ 2 ┆ 5 ┆ a │
| │ 3 ┆ 6 ┆ b │
| │ null ┆ null ┆ null │
| │ 4 ┆ null ┆ c │
| └──────┴──────┴──────┘
|
| Remove rows by comparing two columns against each other; in this case, we
| remove rows where the two columns are not equal (using `ne_missing` to
| ensure that null values compare equal):
|
| >>> lf.remove(
| ... pl.col("foo").ne_missing(pl.col("bar")),
| ... ).collect()
| shape: (2, 3)
| ┌──────┬──────┬──────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞══════╪══════╪══════╡
| │ null ┆ null ┆ null │
| │ 0 ┆ 0 ┆ d │
| └──────┴──────┴──────┘
|
| rename(self, mapping: 'Mapping[str, str] | Callable[[str], str]', *, strict: 'bool' = True) -> 'LazyFrame'
| Rename column names.
|
| Parameters
| ----------
| mapping
| Key value pairs that map from old name to new name, or a function
| that takes the old name as input and returns the new name.
| strict
| Validate that all column names exist in the current schema,
| and throw an exception if any do not. (Note that this parameter
| is a no-op when passing a function to `mapping`).
|
| See Also
| --------
| Expr.name.replace
|
| Notes
| -----
| If existing names are swapped (e.g. 'A' points to 'B' and 'B' points to 'A'),
| polars will block projection and predicate pushdowns at this node.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.rename({"foo": "apple"}).collect()
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ apple ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═══════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └───────┴─────┴─────┘
| >>> lf.rename(lambda column_name: "c" + column_name[1:]).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ coo ┆ car ┆ cam │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 6 ┆ a │
| │ 2 ┆ 7 ┆ b │
| │ 3 ┆ 8 ┆ c │
| └─────┴─────┴─────┘
|
| reverse(self) -> 'LazyFrame'
| Reverse the DataFrame.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "key": ["a", "b", "c"],
| ... "val": [1, 2, 3],
| ... }
| ... )
| >>> lf.reverse().collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ key ┆ val │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ c ┆ 3 │
| │ b ┆ 2 │
| │ a ┆ 1 │
| └─────┴─────┘
|
| rolling(self, index_column: 'IntoExpr', *, period: 'str | timedelta', offset: 'str | timedelta | None' = None, closed: 'ClosedInterval' = 'right', group_by: 'IntoExpr | Iterable[IntoExpr] | None' = None) -> 'LazyGroupBy'
| Create rolling groups based on a temporal or integer column.
|
| Different from a `group_by_dynamic` the windows are now determined by the
| individual values and are not of constant intervals. For constant intervals
| use :func:`LazyFrame.group_by_dynamic`.
|
| If you have a time series `<t_0, t_1, ..., t_n>`, then by default the
| windows created will be
|
| * (t_0 - period, t_0]
| * (t_1 - period, t_1]
| * ...
| * (t_n - period, t_n]
|
| whereas if you pass a non-default `offset`, then the windows will be
|
| * (t_0 + offset, t_0 + offset + period]
| * (t_1 + offset, t_1 + offset + period]
| * ...
| * (t_n + offset, t_n + offset + period]
|
| The `period` and `offset` arguments are created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| .. versionchanged:: 0.20.14
| The `by` parameter was renamed `group_by`.
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order (or, if `group_by` is
| specified, then it must be sorted in ascending order within each group).
|
| In case of a rolling group by on indices, dtype needs to be one of
| {UInt32, UInt64, Int32, Int64}. Note that the first three get temporarily
| cast to Int64, so if performance matters use an Int64 column.
| period
| Length of the window - must be non-negative.
| offset
| Offset of the window. Default is `-period`.
| closed : {'right', 'left', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
| group_by
| Also group by this column/these columns
|
| Returns
| -------
| LazyGroupBy
| Object you can call `.agg` on to aggregate by groups, the result
| of which will be sorted by `index_column` (but note that if `group_by`
| columns are passed, it will only be sorted within each group).
|
| See Also
| --------
| group_by_dynamic
|
| Examples
| --------
| >>> dates = [
| ... "2020-01-01 13:45:48",
| ... "2020-01-01 16:42:13",
| ... "2020-01-01 16:45:09",
| ... "2020-01-02 18:12:48",
| ... "2020-01-03 19:45:32",
| ... "2020-01-08 23:16:43",
| ... ]
| >>> df = pl.LazyFrame({"dt": dates, "a": [3, 7, 5, 9, 2, 1]}).with_columns(
| ... pl.col("dt").str.strptime(pl.Datetime).set_sorted()
| ... )
| >>> out = (
| ... df.rolling(index_column="dt", period="2d")
| ... .agg(
| ... pl.sum("a").alias("sum_a"),
| ... pl.min("a").alias("min_a"),
| ... pl.max("a").alias("max_a"),
| ... )
| ... .collect()
| ... )
| >>> out
| shape: (6, 4)
| ┌─────────────────────┬───────┬───────┬───────┐
| │ dt ┆ sum_a ┆ min_a ┆ max_a │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ i64 ┆ i64 │
| ╞═════════════════════╪═══════╪═══════╪═══════╡
| │ 2020-01-01 13:45:48 ┆ 3 ┆ 3 ┆ 3 │
| │ 2020-01-01 16:42:13 ┆ 10 ┆ 3 ┆ 7 │
| │ 2020-01-01 16:45:09 ┆ 15 ┆ 3 ┆ 7 │
| │ 2020-01-02 18:12:48 ┆ 24 ┆ 3 ┆ 9 │
| │ 2020-01-03 19:45:32 ┆ 11 ┆ 2 ┆ 9 │
| │ 2020-01-08 23:16:43 ┆ 1 ┆ 1 ┆ 1 │
| └─────────────────────┴───────┴───────┴───────┘
|
| select(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'LazyFrame'
| Select columns from this LazyFrame.
|
| Parameters
| ----------
| *exprs
| Column(s) to select, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names,
| other non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to select, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Examples
| --------
| Pass the name of a column to select that column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.select("foo").collect()
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Multiple columns can be selected by passing a list of column names.
|
| >>> lf.select(["foo", "bar"]).collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 6 │
| │ 2 ┆ 7 │
| │ 3 ┆ 8 │
| └─────┴─────┘
|
| Multiple columns can also be selected using positional arguments instead of a
| list. Expressions are also accepted.
|
| >>> lf.select(pl.col("foo"), pl.col("bar") + 1).collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ foo ┆ bar │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| └─────┴─────┘
|
| Use keyword arguments to easily name your expression inputs.
|
| >>> lf.select(
| ... threshold=pl.when(pl.col("foo") > 2).then(10).otherwise(0)
| ... ).collect()
| shape: (3, 1)
| ┌───────────┐
| │ threshold │
| │ --- │
| │ i32 │
| ╞═══════════╡
| │ 0 │
| │ 0 │
| │ 10 │
| └───────────┘
|
| select_seq(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'LazyFrame'
| Select columns from this LazyFrame.
|
| This will run all expression sequentially instead of in parallel.
| Use this when the work per expression is cheap.
|
| Parameters
| ----------
| *exprs
| Column(s) to select, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names,
| other non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to select, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| See Also
| --------
| select
|
| serialize(self, file: 'IOBase | str | Path | None' = None, *, format: 'SerializationFormat' = 'binary') -> 'bytes | str | None'
| Serialize the logical plan of this LazyFrame to a file or string in JSON format.
|
| Parameters
| ----------
| file
| File path to which the result should be written. If set to `None`
| (default), the output is returned as a string instead.
| format
| The format in which to serialize. Options:
|
| - `"binary"`: Serialize to binary format (bytes). This is the default.
| - `"json"`: Serialize to JSON format (string) (deprecated).
|
| See Also
| --------
| LazyFrame.deserialize
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| Serialize the logical plan into a binary representation.
|
| >>> lf = pl.LazyFrame({"a": [1, 2, 3]}).sum()
| >>> bytes = lf.serialize()
|
| The bytes can later be deserialized back into a LazyFrame.
|
| >>> import io
| >>> pl.LazyFrame.deserialize(io.BytesIO(bytes)).collect()
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 6 │
| └─────┘
|
| set_sorted(self, column: 'str | list[str]', *more_columns: 'str', descending: 'bool | list[bool]' = False, nulls_last: 'bool | list[bool]' = False) -> 'LazyFrame'
| Flag a column as sorted.
|
| This can speed up future operations.
|
| Parameters
| ----------
| column
| Column(s) that is sorted
| more_columns
| Columns that are sorted over after `column`.
| descending
| Whether the column is sorted in descending order.
| nulls_last
| Whether the nulls are at the end.
|
| Warnings
| --------
| This can lead to incorrect results if the data is NOT sorted!!
| Use with care!
|
| shift(self, n: 'int | IntoExprColumn' = 1, *, fill_value: 'IntoExpr | None' = None) -> 'LazyFrame'
| Shift values by the given number of indices.
|
| Parameters
| ----------
| n
| Number of indices to shift forward. If a negative value is passed, values
| are shifted in the opposite direction instead.
| fill_value
| Fill the resulting null values with this value. Accepts scalar expression
| input. Non-expression inputs are parsed as literals.
|
| Notes
| -----
| This method is similar to the `LAG` operation in SQL when the value for `n`
| is positive. With a negative value for `n`, it is similar to `LEAD`.
|
| Examples
| --------
| By default, values are shifted forward by one index.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [5, 6, 7, 8],
| ... }
| ... )
| >>> lf.shift().collect()
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ null ┆ null │
| │ 1 ┆ 5 │
| │ 2 ┆ 6 │
| │ 3 ┆ 7 │
| └──────┴──────┘
|
| Pass a negative value to shift in the opposite direction instead.
|
| >>> lf.shift(-2).collect()
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ 3 ┆ 7 │
| │ 4 ┆ 8 │
| │ null ┆ null │
| │ null ┆ null │
| └──────┴──────┘
|
| Specify `fill_value` to fill the resulting null values.
|
| >>> lf.shift(-2, fill_value=100).collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 3 ┆ 7 │
| │ 4 ┆ 8 │
| │ 100 ┆ 100 │
| │ 100 ┆ 100 │
| └─────┴─────┘
|
| show(self, limit: 'int | None' = 5, *, ascii_tables: 'bool | None' = None, decimal_separator: 'str | None' = None, thousands_separator: 'str | bool | None' = None, float_precision: 'int | None' = None, fmt_float: 'FloatFmt | None' = None, fmt_str_lengths: 'int | None' = None, fmt_table_cell_list_len: 'int | None' = None, tbl_cell_alignment: 'Alignment | None' = None, tbl_cell_numeric_alignment: 'Alignment | None' = None, tbl_cols: 'int | None' = None, tbl_column_data_type_inline: 'bool | None' = None, tbl_dataframe_shape_below: 'bool | None' = None, tbl_formatting: 'TableFormatNames | None' = None, tbl_hide_column_data_types: 'bool | None' = None, tbl_hide_column_names: 'bool | None' = None, tbl_hide_dtype_separator: 'bool | None' = None, tbl_hide_dataframe_shape: 'bool | None' = None, tbl_width_chars: 'int | None' = None, trim_decimal_zeros: 'bool | None' = True) -> 'None'
| Show the first `n` rows.
|
| Parameters
| ----------
| limit : int
| Number of rows to show. If None is passed, raises a ValueError. This is done
| to match the signature of :func:`DataFrame.show`.
| ascii_tables : bool
| Use ASCII characters to display table outlines. Set False to revert to the
| default UTF8_FULL_CONDENSED formatting style. See
| :func:`Config.set_ascii_tables` for more information.
| decimal_separator : str
| Set the decimal separator character. See
| :func:`Config.set_decimal_separator` for more information.
| thousands_separator : str, bool
| Set the thousands grouping separator character. See
| :func:`Config.set_thousands_separator` for more information.
| float_precision : int
| Number of decimal places to display for floating point values. See
| :func:`Config.set_float_precision` for more information.
| fmt_float : {"mixed", "full"}
| Control how floating point values are displayed. See
| :func:`Config.set_fmt_float` for more information. Supported options are:
|
| * "mixed": Limit the number of decimal places and use scientific notation
| for large/small values.
| * "full": Print the full precision of the floating point number.
|
| fmt_str_lengths : int
| Number of characters to display for string values. See
| :func:`Config.set_fmt_str_lengths` for more information.
| fmt_table_cell_list_len : int
| Number of elements to display for List values. See
| :func:`Config.set_fmt_table_cell_list_len` for more information.
| tbl_cell_alignment : str
| Set table cell alignment. See :func:`Config.set_tbl_cell_alignment` for more
| information. Supported options are:
|
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| tbl_cell_numeric_alignment : str
| Set table cell alignment for numeric columns. See
| :func:`Config.set_tbl_cell_numeric_alignment` for more information.
| Supported options are:
|
| * "LEFT": left aligned
| * "CENTER": center aligned
| * "RIGHT": right aligned
|
| tbl_cols : int
| Number of columns to display. See :func:`Config.set_tbl_cols` for more
| information.
| tbl_column_data_type_inline : bool
| Moves the data type inline with the column name (to the right, in
| parentheses). See :func:`Config.set_tbl_column_data_type_inline` for more
| information.
| tbl_dataframe_shape_below : bool
| Print the DataFrame shape information below the data when displaying tables.
| See :func:`Config.set_tbl_dataframe_shape_below` for more information.
| tbl_formatting : str
| Set table formatting style. See :func:`Config.set_tbl_formatting` for more
| information. Supported options are:
|
| * "ASCII_FULL": ASCII, with all borders and lines, including row dividers.
| * "ASCII_FULL_CONDENSED": Same as ASCII_FULL, but with dense row spacing.
| * "ASCII_NO_BORDERS": ASCII, no borders.
| * "ASCII_BORDERS_ONLY": ASCII, borders only.
| * "ASCII_BORDERS_ONLY_CONDENSED": ASCII, borders only, dense row spacing.
| * "ASCII_HORIZONTAL_ONLY": ASCII, horizontal lines only.
| * "ASCII_MARKDOWN": Markdown format (ascii ellipses for truncated values).
| * "MARKDOWN": Markdown format (utf8 ellipses for truncated values).
| * "UTF8_FULL": UTF8, with all borders and lines, including row dividers.
| * "UTF8_FULL_CONDENSED": Same as UTF8_FULL, but with dense row spacing.
| * "UTF8_NO_BORDERS": UTF8, no borders.
| * "UTF8_BORDERS_ONLY": UTF8, borders only.
| * "UTF8_HORIZONTAL_ONLY": UTF8, horizontal lines only.
| * "NOTHING": No borders or other lines.
|
| tbl_hide_column_data_types : bool
| Hide table column data types (i64, f64, str etc.). See
| :func:`Config.set_tbl_hide_column_data_types` for more information.
| tbl_hide_column_names : bool
| Hide table column names. See :func:`Config.set_tbl_hide_column_names` for
| more information.
| tbl_hide_dtype_separator : bool
| Hide the '---' separator between the column names and column types. See
| :func:`Config.set_tbl_hide_dtype_separator` for more information.
| tbl_hide_dataframe_shape : bool
| Hide the DataFrame shape information when displaying tables. See
| :func:`Config.set_tbl_hide_dataframe_shape` for more information.
| tbl_width_chars : int
| Set the maximum width of a table in characters. See
| :func:`Config.set_tbl_width_chars` for more information.
| trim_decimal_zeros : bool
| Strip trailing zeros from Decimal data type values. See
| :func:`Config.set_trim_decimal_zeros` for more information.
|
| Warnings
| --------
| * This method does *not* maintain the laziness of the frame, and will `collect`
| the final result. This could potentially be an expensive operation.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [7, 8, 9, 10, 11, 12],
| ... }
| ... )
| >>> lf.show()
| shape: (5, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| │ 4 ┆ 10 │
| │ 5 ┆ 11 │
| └─────┴─────┘
| >>> lf.show(2)
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 7 │
| │ 2 ┆ 8 │
| └─────┴─────┘
|
| show_graph(self, *, optimized: 'bool' = True, show: 'bool' = True, output_path: 'str | Path | None' = None, raw_output: 'bool' = False, figsize: 'tuple[float, float]' = (16.0, 12.0), type_coercion: 'bool' = True, _type_check: 'bool' = True, predicate_pushdown: 'bool' = True, projection_pushdown: 'bool' = True, simplify_expression: 'bool' = True, slice_pushdown: 'bool' = True, comm_subplan_elim: 'bool' = True, comm_subexpr_elim: 'bool' = True, cluster_with_columns: 'bool' = True, collapse_joins: 'bool' = True, engine: 'EngineType' = 'auto', plan_stage: 'PlanStage' = 'ir', _check_order: 'bool' = True, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'str | None'
| Show a plot of the query plan.
|
| Note that Graphviz must be installed to render the visualization (if not
| already present, you can download it here: `<https://graphviz.org/download>`_).
|
| Parameters
| ----------
| optimized
| Optimize the query plan.
| show
| Show the figure.
| output_path
| Write the figure to disk.
| raw_output
| Return dot syntax. This cannot be combined with `show` and/or `output_path`.
| figsize
| Passed to matplotlib if `show == True`.
| type_coercion
| Do type coercion optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| predicate_pushdown
| Do predicate pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| projection_pushdown
| Do projection pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| simplify_expression
| Run simplify expressions optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| slice_pushdown
| Slice pushdown optimization.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subplan_elim
| Will try to cache branching subplans that occur on self-joins or unions.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| comm_subexpr_elim
| Common subexpressions will be cached and reused.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| cluster_with_columns
| Combine sequential independent calls to with_columns.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| collapse_joins
| Collapse a join and filters into a faster join.
|
| .. deprecated:: 1.30.0
| Use the `optimizations` parameters.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"in-memory"`` if unset (this default may change in
| a future release).
| * ``"in-memory"``: use the in-memory engine, this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control (e.g. device selection on multi-GPU systems).
|
| If the selected engine cannot run the query, Polars falls back to
| the in-memory engine.
|
| .. note::
| GPU mode is considered **unstable**. Not all queries will run
| successfully on the GPU, however, they should fall back transparently
| to the default engine if execution is not supported.
|
| Running with `POLARS_VERBOSE=1` will provide information if a query
| falls back (and why).
| plan_stage : {'ir', 'physical'}
| Select the stage to display. Currently only the streaming engine has a
| separate physical stage, for the other engines both IR and physical are the
| same.
| optimizations
| The set of the optimizations considered during query optimization.
|
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [1, 2, 3, 4, 5, 6],
| ... "c": [6, 5, 4, 3, 2, 1],
| ... }
| ... )
| >>> lf.group_by("a", maintain_order=True).agg(pl.all().sum()).sort(
| ... "a"
| ... ).show_graph() # doctest: +SKIP
|
| sink_batches(self, function: 'Callable[[DataFrame], bool | None]', *, chunk_size: 'int | None' = None, maintain_order: 'bool' = True, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'pl.LazyFrame | None'
| Evaluate the query and call a user-defined function for every ready batch.
|
| This allows streaming results that are larger than RAM in certain cases.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. warning::
| This method is much slower than native sinks. Only use it if you cannot
| implement your logic otherwise.
|
| Parameters
| ----------
| function
| Function to run with a batch that is ready. If the function returns
| `True`, this signals that no more results are needed, allowing for
| early stopping.
| chunk_size
| The number of rows that are buffered before the callback is called.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
| lazy: bool
| Wait to start execution until `collect` is called.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_batches(lambda df: print(df)) # doctest: +SKIP
|
| sink_csv(self, path: 'str | Path | IO[bytes] | IO[str] | PartitionBy', *, include_bom: 'bool' = False, compression: "Literal['uncompressed', 'gzip', 'zstd']" = 'uncompressed', compression_level: 'int | None' = None, check_extension: 'bool' = True, include_header: 'bool' = True, separator: 'str' = ',', line_terminator: 'str' = '\n', quote_char: 'str' = '"', batch_size: 'int' = 1024, datetime_format: 'str | None' = None, date_format: 'str | None' = None, time_format: 'str | None' = None, float_scientific: 'bool | None' = None, float_precision: 'int | None' = None, decimal_comma: 'bool' = False, null_value: 'str | None' = None, quote_style: 'CsvQuoteStyle | None' = None, maintain_order: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, sync_on_close: 'SyncOnCloseMethod | None' = None, mkdir: 'bool' = False, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'LazyFrame | None'
| Evaluate the query in streaming mode and write to a CSV file.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| Parameters
| ----------
| path
| File path to which the file should be written.
| include_bom
| Whether to include UTF-8 BOM in the CSV output.
| compression
| What compression format to use.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| compression_level
| The compression level to use, typically 0-9 or `None` to let the
| engine choose.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| check_extension
| Whether to check if the filename matches the compression settings.
| Will raise an error if compression is set to 'uncompressed' and the
| filename ends in one of (".gz", ".zst", ".zstd") or if
| compression != 'uncompressed' and the file uses an mismatched
| extension. Only applies if file is a path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| include_header
| Whether to include header in the CSV output.
| separator
| Separate CSV fields with this symbol.
| line_terminator
| String used to end each row.
| quote_char
| Byte to use as quoting character.
| batch_size
| Number of rows that will be processed per thread.
| datetime_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate. If no format specified, the default fractional-second
| precision is inferred from the maximum timeunit found in the frame's
| Datetime cols (if any).
| date_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate.
| time_format
| A format string, with the specifiers defined by the
| `chrono <https://docs.rs/chrono/latest/chrono/format/strftime/index.html>`_
| Rust crate.
| float_scientific
| Whether to use scientific form always (true), never (false), or
| automatically (None) for floating-point datatypes.
| float_precision
| Number of decimal places to write, applied to both floating-point
| datatypes.
| decimal_comma
| Use a comma as the decimal separator instead of a point. Floats will be
| encapsulated in quotes if necessary; set the field separator to override.
| null_value
| A string representing null values (defaulting to the empty string).
| quote_style : {'necessary', 'always', 'non_numeric', 'never'}
| Determines the quoting strategy used.
|
| - necessary (default): This puts quotes around fields only when necessary.
| They are necessary when fields contain a quote,
| delimiter or record terminator.
| Quotes are also necessary when writing an empty record
| (which is indistinguishable from a record with one empty field).
| This is the default.
| - always: This puts quotes around every field. Always.
| - never: This never puts quotes around fields, even if that results in
| invalid CSV data (e.g.: by not quoting strings containing the
| separator).
| - non_numeric: This puts quotes around all fields that are non-numeric.
| Namely, when writing a field that does not parse as a valid float
| or integer, then quotes will be used even if they aren`t strictly
| necessary.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| sync_on_close: { None, 'data', 'all' }
| Sync to disk when before closing a file.
|
| * `None` does not sync.
| * `data` syncs the file contents.
| * `all` syncs the file contents and metadata.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| lazy: bool
| Wait to start execution until `collect` is called.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_csv("out.csv") # doctest: +SKIP
|
| Sink to a `BytesIO` object.
|
| >>> import io
| >>> buf = io.BytesIO() # doctest: +SKIP
| >>> pl.LazyFrame({"x": [1, 2, 1]}).sink_csv(buf) # doctest: +SKIP
|
| Split into a hive-partitioning style partition:
|
| >>> pl.LazyFrame({"x": [1, 2, 1], "y": [3, 4, 5]}).sink_csv(
| ... pl.PartitionBy("./out/", key="x"),
| ... mkdir=True
| ... ) # doctest: +SKIP
|
| See Also
| --------
| PartitionBy
|
| sink_delta(self, target: 'str | Path | deltalake.DeltaTable', *, mode: "Literal['error', 'append', 'overwrite', 'ignore', 'merge']" = 'error', storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', delta_write_options: 'dict[str, Any] | None' = None, delta_merge_options: 'dict[str, Any] | None' = None, optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'deltalake.table.TableMerger | None'
| Sink DataFrame as delta table.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| target
| URI of a table or a DeltaTable object.
| mode : {'error', 'append', 'overwrite', 'ignore', 'merge'}
| How to handle existing data.
|
| - If 'error', throw an error if the table already exists (default).
| - If 'append', will add new data.
| - If 'overwrite', will replace table with new data.
| - If 'ignore', will not write anything if table already exists.
| - If 'merge', return a `TableMerger` object to merge data from the DataFrame
| with the existing data.
| storage_options
| Extra options for the storage backends supported by `deltalake`.
| For cloud storages, this may include configurations for authentication etc.
|
| - See a list of supported storage options for S3 `here <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html#variants>`__.
| - See a list of supported storage options for GCS `here <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html#variants>`__.
| - See a list of supported storage options for Azure `here <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants>`__.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| delta_write_options
| Additional keyword arguments while writing a Delta lake Table.
| See a list of supported write options `here <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake>`__.
| delta_merge_options
| Keyword arguments which are required to `MERGE` a Delta lake Table.
| See a list of supported merge options `here <https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.merge>`__.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Raises
| ------
| TypeError
| If the DataFrame contains unsupported data types.
| ArrowInvalidError
| If the DataFrame contains data types that could not be cast to their
| primitive type.
| TableNotFoundError
| If the delta table doesn't exist and MERGE action is triggered
|
| Notes
| -----
| The Polars data types :class:`Null` and :class:`Time` are not supported
| by the delta protocol specification and will raise a TypeError. Columns
| using The :class:`Categorical` data type will be converted to
| normal (non-categorical) strings when written.
|
| Polars columns are always nullable. To write data to a delta table with
| non-nullable columns, a custom pyarrow schema has to be passed to the
| `delta_write_options`. See the last example below.
|
| Examples
| --------
| Sink a large than fits into memory dataset to a Delta Lake table.
|
| >>> lf = pl.scan_parquet(
| ... "/path/to/my_larger_than_ram_file.parquet"
| ... ) # doctest: +SKIP
| >>> table_path = "/path/to/delta-table/"
| >>> lf.sink_delta(table_path) # doctest: +SKIP
|
|
| Sink a dataframe to the local filesystem as a Delta Lake table.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> table_path = "/path/to/delta-table/"
| >>> df.lazy().sink_delta(table_path) # doctest: +SKIP
|
| Append data to an existing Delta Lake table on the local filesystem.
| Note that this will fail if the schema of the new data does not match the
| schema of the existing table.
|
| >>> df.lazy().sink_delta(table_path, mode="append") # doctest: +SKIP
|
| Overwrite a Delta Lake table as a new version.
| If the schemas of the new and old data are the same, specifying the
| `schema_mode` is not required.
|
| >>> existing_table_path = "/path/to/delta-table/"
| >>> df.lazy().sink_delta(
| ... existing_table_path,
| ... mode="overwrite",
| ... delta_write_options={"schema_mode": "overwrite"},
| ... ) # doctest: +SKIP
|
| Sink a DataFrame as a Delta Lake table to a cloud object store like S3.
|
| >>> table_path = "s3://bucket/prefix/to/delta-table/"
| >>> df.lazy().sink_delta(
| ... table_path,
| ... storage_options={
| ... "AWS_REGION": "THE_AWS_REGION",
| ... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
| ... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
| ... },
| ... ) # doctest: +SKIP
|
| Sink DataFrame as a Delta Lake table with non-nullable columns.
|
| >>> import pyarrow as pa
| >>> existing_table_path = "/path/to/delta-table/"
| >>> df.lazy().sink_delta(
| ... existing_table_path,
| ... delta_write_options={
| ... "schema": pa.schema([pa.field("foo", pa.int64(), nullable=False)])
| ... },
| ... ) # doctest: +SKIP
|
| Sink DataFrame as a Delta Lake table with zstd compression.
| For all `delta_write_options` keyword arguments, check the deltalake docs
| `here
| <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake>`__,
| and for Writer Properties in particular `here
| <https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.WriterProperties>`__.
|
| >>> import deltalake
| >>> df.lazy().sink_delta(
| ... table_path,
| ... delta_write_options={
| ... "writer_properties": deltalake.WriterProperties(compression="zstd"),
| ... },
| ... ) # doctest: +SKIP
|
| Merge the DataFrame with an existing Delta Lake table.
| For all `TableMerger` methods, check the deltalake docs
| `here <https://delta-io.github.io/delta-rs/api/delta_table/delta_table_merger/>`__.
|
| >>> df = pl.DataFrame(
| ... {
| ... "foo": [1, 2, 3, 4, 5],
| ... "bar": [6, 7, 8, 9, 10],
| ... "ham": ["a", "b", "c", "d", "e"],
| ... }
| ... )
| >>> table_path = "/path/to/delta-table/"
| >>> (
| ... df.lazy()
| ... .sink_delta(
| ... "table_path",
| ... mode="merge",
| ... delta_merge_options={
| ... "predicate": "s.foo = t.foo",
| ... "source_alias": "s",
| ... "target_alias": "t",
| ... },
| ... )
| ... .when_matched_update_all()
| ... .when_not_matched_insert_all()
| ... .execute()
| ... ) # doctest: +SKIP
|
| sink_iceberg(self, target: 'str | pyiceberg.table.Table', *, mode: "Literal['append', 'overwrite']", catalog: 'pyiceberg.catalog.Catalog | polars.io.iceberg.IcebergCatalogConfig | None' = None, storage_options: 'StorageOptionsDict | None' = None) -> 'pl.DataFrame'
| Sink a LazyFrame to an Iceberg table.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| target
| A PyIceberg Table object, or a 'namespace.table_name' identifier string.
| mode : {'append', 'overwrite'}
| How to handle existing data.
|
| - If 'append', will add new data.
| - If 'overwrite', will replace table with new data.
| catalog
| PyIceberg catalog to load the table from if the provided `target`
| was a table identifier.
| storage_options
| Extra options for the storage backends supported by `pyiceberg`.
| For cloud storages, this may include configurations for authentication etc.
|
| More info is available `here <https://py.iceberg.apache.org/configuration/>`__.
|
| Returns
| -------
| DataFrame
| Contains the new metadata path.
|
| sink_ipc(self, path: 'str | Path | IO[bytes] | PartitionBy', *, compression: 'IpcCompression | None' = 'uncompressed', compat_level: 'CompatLevel | None' = None, record_batch_size: 'int | None' = None, maintain_order: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, sync_on_close: 'SyncOnCloseMethod | None' = None, mkdir: 'bool' = False, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, _record_batch_statistics: 'bool' = False) -> 'LazyFrame | None'
| Evaluate the query in streaming mode and write to an IPC file.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| Parameters
| ----------
| path
| File path to which the file should be written.
| compression : {'uncompressed', 'lz4', 'zstd'}
| Choose "zstd" for good compression performance.
| Choose "lz4" for fast compression/decompression.
| compat_level
| Compatibility level to use when exporting Polars data structures.
| The default compatibility level is recommended for most users.
| Use ``pl.CompatLevel.oldest()`` for the most compatible level.
| ``pl.CompatLevel.newest()`` uses the highest supported compatibility
| level, but is considered unstable and may change without it being
| considered a breaking change.
| record_batch_size
| Size of the record batches in number of rows.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| sync_on_close: { None, 'data', 'all' }
| Sync to disk when before closing a file.
|
| * `None` does not sync.
| * `data` syncs the file contents.
| * `all` syncs the file contents and metadata.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| lazy: bool
| Wait to start execution until `collect` is called.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: not currently supported for this sink.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
|
| .. note::
| The GPU engine is currently not supported.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_ipc("out.arrow") # doctest: +SKIP
|
| Sink to a `BytesIO` object.
|
| >>> import io
| >>> buf = io.BytesIO() # doctest: +SKIP
| >>> pl.LazyFrame({"x": [1, 2, 1]}).sink_ipc(buf) # doctest: +SKIP
|
| Split into a hive-partitioning style partition:
|
| >>> pl.LazyFrame({"x": [1, 2, 1], "y": [3, 4, 5]}).sink_ipc(
| ... pl.PartitionBy("./out/", key="x"),
| ... mkdir=True
| ... ) # doctest: +SKIP
|
| See Also
| --------
| PartitionBy
|
| sink_ndjson(self, path: 'str | Path | IO[bytes] | IO[str] | PartitionBy', *, compression: "Literal['uncompressed', 'gzip', 'zstd']" = 'uncompressed', compression_level: 'int | None' = None, check_extension: 'bool' = True, maintain_order: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, sync_on_close: 'SyncOnCloseMethod | None' = None, mkdir: 'bool' = False, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>) -> 'LazyFrame | None'
| Evaluate the query in streaming mode and write to an NDJSON file.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| Parameters
| ----------
| path
| File path to which the file should be written.
| compression
| What compression format to use.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| compression_level
| The compression level to use, typically 0-9 or `None` to let the
| engine choose.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| check_extension
| Whether to check if the filename matches the compression settings.
| Will raise an error if compression is set to 'uncompressed' and the
| filename ends in one of (".gz", ".zst", ".zstd") or if
| compression != 'uncompressed' and the file uses an mismatched
| extension. Only applies if file is a path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| sync_on_close: { None, 'data', 'all' }
| Sync to disk when before closing a file.
|
| * `None` does not sync.
| * `data` syncs the file contents.
| * `all` syncs the file contents and metadata.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| lazy: bool
| Wait to start execution until `collect` is called.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_ndjson("out.ndjson") # doctest: +SKIP
|
| Sink to a `BytesIO` object.
|
| >>> import io
| >>> buf = io.BytesIO() # doctest: +SKIP
| >>> pl.LazyFrame({"x": [1, 2, 1]}).sink_ndjson(buf) # doctest: +SKIP
|
| Split into a hive-partitioning style partition:
|
| >>> pl.LazyFrame({"x": [1, 2, 1], "y": [3, 4, 5]}).sink_ndjson(
| ... pl.PartitionBy("./out/", key="x"),
| ... mkdir=True
| ... ) # doctest: +SKIP
|
| See Also
| --------
| PartitionBy
|
| sink_parquet(self, path: 'str | Path | IO[bytes] | PartitionBy', *, compression: 'str' = 'zstd', compression_level: 'int | None' = None, statistics: 'bool | str | dict[str, bool]' = True, row_group_size: 'int | None' = None, data_page_size: 'int | None' = None, maintain_order: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, sync_on_close: 'SyncOnCloseMethod | None' = None, metadata: 'ParquetMetadata | None' = None, arrow_schema: 'ArrowSchemaExportable | None' = None, mkdir: 'bool' = False, lazy: 'bool' = False, engine: 'EngineType' = 'auto', optimizations: 'QueryOptFlags' = <polars.lazyframe.opt_flags.QueryOptFlags object at 0x00000280D8794920>, _sinked_paths_callback: 'SinkedPathsCallback | None' = None) -> 'LazyFrame | None'
| Evaluate the query in streaming mode and write to a Parquet file.
|
| This allows streaming results that are larger than RAM to be written to disk.
|
| Parameters
| ----------
| path
| File path to which the file should be written.
| compression : {'lz4', 'uncompressed', 'snappy', 'gzip', 'brotli', 'zstd'}
| Choose "zstd" for good compression performance.
| Choose "lz4" for fast compression/decompression.
| Choose "snappy" for more backwards compatibility guarantees
| when you deal with older parquet readers.
| compression_level
| The level of compression to use. Higher compression means smaller files on
| disk.
|
| - "gzip" : min-level: 0, max-level: 9, default: 6.
| - "brotli" : min-level: 0, max-level: 11, default: 1.
| - "zstd" : min-level: 1, max-level: 22, default: 3.
| statistics
| Write statistics to the parquet headers. This is the default behavior.
|
| Possible values:
|
| - `True`: enable default set of statistics (default). Some
| statistics may be disabled.
| - `False`: disable all statistics
| - "full": calculate and write all available statistics.
| - `{ "statistic-key": True / False, ... }`. Available keys:
|
| - "min": column minimum value (default: `True`)
| - "max": column maximum value (default: `True`)
| - "distinct_count": number of unique column values (default: `False`)
| - "null_count": number of null values in column (default: `True`)
| row_group_size
| Size of the row groups in number of rows.
| If None (default), the chunks of the `DataFrame` are
| used. Writing in smaller chunks may reduce memory pressure and improve
| writing speeds.
| data_page_size
| Size limit of individual data pages.
| If not set defaults to 1024 * 1024 bytes
| maintain_order
| Maintain the order in which data is processed.
| Setting this to `False` will be slightly faster.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| storage_options
| Options that indicate how to connect to a cloud provider.
|
| The cloud providers currently supported are AWS, GCP, and Azure.
| See supported keys here:
|
| * `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
| * `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
| * `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
| * Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
|
| If `storage_options` is not provided, Polars will try to infer the
| information from environment variables.
| credential_provider
| Provide a function that can be called to provide cloud storage
| credentials. The function is expected to return a dictionary of
| credential keys along with an optional credential expiry time.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
| retries
| Number of retries if accessing a cloud instance fails.
|
| .. deprecated:: 1.37.1
| Pass {"max_retries": n} via `storage_options` instead.
| sync_on_close: { None, 'data', 'all' }
| Sync to disk when before closing a file.
|
| * `None` does not sync.
| * `data` syncs the file contents.
| * `all` syncs the file contents and metadata.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| metadata
| A dictionary or callback to add key-values to the file-level Parquet
| metadata.
|
| .. warning::
| This functionality is considered **experimental**. It may be removed or
| changed at any point without it being considered a breaking change.
| arrow_schema
| Provide a custom arrow schema to write to the file. This allows
| setting custom schema and field-level metadata. Names and dtypes
| must match.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| mkdir: bool
| Recursively create all the directories in the path.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| lazy: bool
| Wait to start execution until `collect` is called.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed at any
| point without it being considered a breaking change.
| engine
| Select the engine used to process the query (default ``"auto"``):
|
| * ``"auto"``: use the engine set by
| :meth:`Config.set_engine_affinity <polars.Config.set_engine_affinity>`
| or the ``POLARS_ENGINE_AFFINITY`` environment variable, falling
| back to ``"streaming"`` if unset.
| * ``"in-memory"``: use the in-memory engine before writing,
| this is the default engine.
| * ``"streaming"``: use the streaming engine, which processes
| queries in batches, reducing memory pressure and often
| outperforming the in-memory engine. This will soon become
| the default engine of Polars.
| * ``"gpu"``: use the CUDA GPU engine (requires an Nvidia GPU and
| ``cudf-polars``). Pass a :class:`~.GPUEngine` object for
| fine-grained control.
|
| If the selected engine cannot run the query, Polars falls back to
| the streaming engine.
| optimizations
| The optimization passes done during query optimization.
|
| This has no effect if `lazy` is set to `True`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv") # doctest: +SKIP
| >>> lf.sink_parquet("out.parquet") # doctest: +SKIP
|
| Sink to a `BytesIO` object.
|
| >>> import io
| >>> buf = io.BytesIO() # doctest: +SKIP
| >>> pl.LazyFrame({"x": [1, 2, 1]}).sink_parquet(buf) # doctest: +SKIP
|
| Split into a hive-partitioning style partition:
|
| >>> pl.LazyFrame({"x": [1, 2, 1], "y": [3, 4, 5]}).sink_parquet(
| ... pl.PartitionBy("./out/", key="x"),
| ... mkdir=True
| ... ) # doctest: +SKIP
|
| See Also
| --------
| PartitionBy
|
| slice(self, offset: 'int', length: 'int | None' = None) -> 'LazyFrame'
| Get a slice of this DataFrame.
|
| Parameters
| ----------
| offset
| Start index. Negative indexing is supported.
| length
| Length of the slice. If set to `None`, all rows starting at the offset
| will be selected.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["x", "y", "z"],
| ... "b": [1, 3, 5],
| ... "c": [2, 4, 6],
| ... }
| ... )
| >>> lf.slice(1, 2).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╡
| │ y ┆ 3 ┆ 4 │
| │ z ┆ 5 ┆ 6 │
| └─────┴─────┴─────┘
|
| sort(self, by: 'IntoExpr | Iterable[IntoExpr]', *more_by: 'IntoExpr', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False, maintain_order: 'bool' = False, multithreaded: 'bool' = True) -> 'LazyFrame'
| Sort the LazyFrame by the given columns.
|
| Parameters
| ----------
| by
| Column(s) to sort by. Accepts expression input, including selectors. Strings
| are parsed as column names.
| *more_by
| Additional columns to sort by, specified as positional arguments.
| descending
| Sort in descending order. When sorting by multiple columns, can be specified
| per column by passing a sequence of booleans.
| nulls_last
| Place null values last; can specify a single boolean applying to all columns
| or a sequence of booleans for per-column control.
| maintain_order
| Whether the order should be maintained if elements are equal.
| Note that if `true` streaming is not possible and performance might be
| worse since this requires a stable search.
| multithreaded
| Sort using multiple threads.
|
| Examples
| --------
| Pass a single column name to sort by that column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": [6.0, 5.0, 4.0],
| ... "c": ["a", "c", "b"],
| ... }
| ... )
| >>> lf.sort("a").collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ null ┆ 4.0 ┆ b │
| │ 1 ┆ 6.0 ┆ a │
| │ 2 ┆ 5.0 ┆ c │
| └──────┴─────┴─────┘
|
| Sorting by expressions is also supported.
|
| >>> lf.sort(pl.col("a") + pl.col("b") * 2, nulls_last=True).collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 2 ┆ 5.0 ┆ c │
| │ 1 ┆ 6.0 ┆ a │
| │ null ┆ 4.0 ┆ b │
| └──────┴─────┴─────┘
|
| Sort by multiple columns by passing a list of columns.
|
| >>> lf.sort(["c", "a"], descending=True).collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 2 ┆ 5.0 ┆ c │
| │ null ┆ 4.0 ┆ b │
| │ 1 ┆ 6.0 ┆ a │
| └──────┴─────┴─────┘
|
| Or use positional arguments to sort by multiple columns in the same way.
|
| >>> lf.sort("c", "a", descending=[False, True]).collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ str │
| ╞══════╪═════╪═════╡
| │ 1 ┆ 6.0 ┆ a │
| │ null ┆ 4.0 ┆ b │
| │ 2 ┆ 5.0 ┆ c │
| └──────┴─────┴─────┘
|
| sql(self, query: 'str', *, table_name: 'str' = 'self') -> 'LazyFrame'
| Execute a SQL query against the LazyFrame.
|
| .. versionadded:: 0.20.23
|
| .. warning::
| This functionality is considered **unstable**, although it is close to
| being considered stable. It may be changed at any point without it being
| considered a breaking change.
|
| Parameters
| ----------
| query
| SQL query to execute.
| table_name
| Optionally provide an explicit name for the table that represents the
| calling frame (defaults to "self").
|
| Notes
| -----
| * The calling LazyFrame is automatically registered as a table in the SQLContext
| under the name "self". If you want access to the DataFrames and LazyFrames
| found in the current globals, use the top-level :meth:`pl.sql <polars.sql>`.
| * More control over registration and execution behaviour is available by
| using the :class:`SQLContext` object.
|
| See Also
| --------
| SQLContext
|
| Examples
| --------
| >>> lf1 = pl.LazyFrame({"a": [1, 2, 3], "b": [6, 7, 8], "c": ["z", "y", "x"]})
| >>> lf2 = pl.LazyFrame({"a": [3, 2, 1], "d": [125, -654, 888]})
|
| Query the LazyFrame using SQL:
|
| >>> lf1.sql("SELECT c, b FROM self WHERE a > 1").collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ c ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ y ┆ 7 │
| │ x ┆ 8 │
| └─────┴─────┘
|
| Apply SQL transforms (aliasing "self" to "frame") then filter
| natively (you can freely mix SQL and native operations):
|
| >>> lf1.sql(
| ... query='''
| ... SELECT
| ... a,
| ... (a % 2 == 0) AS a_is_even,
| ... (b::float4 / 2) AS "b/2",
| ... CONCAT_WS(':', c, c, c) AS c_c_c
| ... FROM frame
| ... ORDER BY a
| ... ''',
| ... table_name="frame",
| ... ).filter(~pl.col("c_c_c").str.starts_with("x")).collect()
| shape: (2, 4)
| ┌─────┬───────────┬─────┬───────┐
| │ a ┆ a_is_even ┆ b/2 ┆ c_c_c │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ bool ┆ f32 ┆ str │
| ╞═════╪═══════════╪═════╪═══════╡
| │ 1 ┆ false ┆ 3.0 ┆ z:z:z │
| │ 2 ┆ true ┆ 3.5 ┆ y:y:y │
| └─────┴───────────┴─────┴───────┘
|
| std(self, ddof: 'int' = 1) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their standard deviation value.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.std().collect()
| shape: (1, 2)
| ┌──────────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════════╪═════╡
| │ 1.290994 ┆ 0.5 │
| └──────────┴─────┘
| >>> lf.std(ddof=0).collect()
| shape: (1, 2)
| ┌──────────┬──────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════════╪══════════╡
| │ 1.118034 ┆ 0.433013 │
| └──────────┴──────────┘
|
| sum(self) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their sum value.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.sum().collect()
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 10 ┆ 5 │
| └─────┴─────┘
|
| tail(self, n: 'int' = 5) -> 'LazyFrame'
| Get the last `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [7, 8, 9, 10, 11, 12],
| ... }
| ... )
| >>> lf.tail().collect()
| shape: (5, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 2 ┆ 8 │
| │ 3 ┆ 9 │
| │ 4 ┆ 10 │
| │ 5 ┆ 11 │
| │ 6 ┆ 12 │
| └─────┴─────┘
| >>> lf.tail(2).collect()
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 5 ┆ 11 │
| │ 6 ┆ 12 │
| └─────┴─────┘
|
| top_k(self, k: 'int', *, by: 'IntoExpr | Iterable[IntoExpr]', reverse: 'bool | Sequence[bool]' = False) -> 'LazyFrame'
| Return the `k` largest rows.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| k
| Number of rows to return.
| by
| Column(s) used to determine the top rows.
| Accepts expression input. Strings are parsed as column names.
| reverse
| Consider the `k` smallest elements of the `by` column(s) (instead of the `k`
| largest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| bottom_k
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["a", "b", "a", "b", "b", "c"],
| ... "b": [2, 1, 1, 3, 2, 1],
| ... }
| ... )
|
| Get the rows which contain the 4 largest values in column b.
|
| >>> lf.top_k(4, by="b").collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 3 │
| │ a ┆ 2 │
| │ b ┆ 2 │
| │ b ┆ 1 │
| └─────┴─────┘
|
| Get the rows which contain the 4 largest values when sorting on column b and a.
|
| >>> lf.top_k(4, by=["b", "a"]).collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ b ┆ 3 │
| │ b ┆ 2 │
| │ a ┆ 2 │
| │ c ┆ 1 │
| └─────┴─────┘
|
| unique(self, subset: 'IntoExpr | Collection[IntoExpr] | None' = None, *, keep: 'UniqueKeepStrategy' = 'any', maintain_order: 'bool' = False) -> 'LazyFrame'
| Drop duplicate rows from this LazyFrame.
|
| Parameters
| ----------
| subset
| Column name(s), selector(s), or expressions to consider when identifying
| duplicate rows. If set to `None` (default), all columns are considered.
| keep : {'first', 'last', 'any', 'none'}
| Which of the duplicate rows to keep.
|
| * 'any': Does not give any guarantee of which row is kept.
| This allows more optimizations.
| * 'none': Don't keep duplicate rows.
| * 'first': Keep the first unique row.
| * 'last': Keep the last unique row.
| maintain_order
| Keep the same order as the original DataFrame. This is more expensive
| to compute. Settings this to `True` blocks the possibility to run on
| the streaming engine.
|
| Returns
| -------
| LazyFrame
| LazyFrame with unique rows.
|
| Notes
| -----
| If you're coming from Pandas, this is similar to
| `pandas.DataFrame.drop_duplicates`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3, 1, 1],
| ... "bar": ["a", "a", "a", "x", "x"],
| ... "ham": ["b", "b", "b", "y", "y"],
| ... }
| ... )
|
| By default, all columns are considered when determining which rows are unique:
|
| >>> lf.unique(maintain_order=True).collect()
| shape: (4, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
|
| We can also consider only a subset of columns when determining uniqueness,
| controlling which row we keep when duplicates are found:
|
| >>> lf.unique(subset="foo", keep="first", maintain_order=True).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| └─────┴─────┴─────┘
| >>> lf.unique(subset="foo", keep="last", maintain_order=True).collect()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
| >>> lf.unique(subset="foo", keep="none", maintain_order=True).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 2 ┆ a ┆ b │
| │ 3 ┆ a ┆ b │
| └─────┴─────┴─────┘
|
| Selectors can be used to define the "subset" parameter:
|
| >>> import polars.selectors as cs
| >>> lf.unique(subset=cs.string(), maintain_order=True).collect()
| shape: (2, 3)
| ┌─────┬─────┬─────┐
| │ foo ┆ bar ┆ ham │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 1 ┆ a ┆ b │
| │ 1 ┆ x ┆ y │
| └─────┴─────┴─────┘
|
| We can also use an arbitrary expression in the "subset" parameter; in this
| example we use the part of the label in front of ":" to determine uniqueness:
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "label": ["xx:1", "xx:2", "yy:3", "yy:4"],
| ... "value": [100, 200, 300, 400],
| ... }
| ... )
| >>> lf.unique(
| ... subset=pl.col("label").str.extract(r"^(\w+):"),
| ... maintain_order=True,
| ... keep="first",
| ... ).collect()
| shape: (2, 2)
| ┌───────┬───────┐
| │ label ┆ value │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═══════╡
| │ xx:1 ┆ 100 │
| │ yy:3 ┆ 300 │
| └───────┴───────┘
|
| unnest(self, columns: 'ColumnNameOrSelector | Collection[ColumnNameOrSelector] | None' = None, *more_columns: 'ColumnNameOrSelector', separator: 'str | None' = None) -> 'LazyFrame'
| Decompose struct columns into separate columns for each of their fields.
|
| The new columns will be inserted into the DataFrame at the location of the
| struct column.
|
| If no columns are provided, all struct columns are unnested.
|
| Parameters
| ----------
| columns
| Name of the struct column(s) that should be unnested.
| *more_columns
| Additional columns to unnest, specified as positional arguments.
| separator
| Rename output column names as combination of the struct column name,
| name separator and field name.
|
| Examples
| --------
| >>> df = pl.LazyFrame(
| ... {
| ... "before": ["foo", "bar"],
| ... "t_a": [1, 2],
| ... "t_b": ["a", "b"],
| ... "t_c": [True, None],
| ... "t_d": [[1, 2], [3]],
| ... "after": ["baz", "womp"],
| ... }
| ... ).select("before", pl.struct(pl.col("^t_.$")).alias("t_struct"), "after")
| >>> df.collect()
| shape: (2, 3)
| ┌────────┬─────────────────────┬───────┐
| │ before ┆ t_struct ┆ after │
| │ --- ┆ --- ┆ --- │
| │ str ┆ struct[4] ┆ str │
| ╞════════╪═════════════════════╪═══════╡
| │ foo ┆ {1,"a",true,[1, 2]} ┆ baz │
| │ bar ┆ {2,"b",null,[3]} ┆ womp │
| └────────┴─────────────────────┴───────┘
| >>> df.unnest("t_struct").collect()
| shape: (2, 6)
| ┌────────┬─────┬─────┬──────┬───────────┬───────┐
| │ before ┆ t_a ┆ t_b ┆ t_c ┆ t_d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪═════╪═════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴─────┴─────┴──────┴───────────┴───────┘
|
| Unnest all struct columns by calling without arguments:
|
| >>> df.unnest().collect()
| shape: (2, 6)
| ┌────────┬─────┬─────┬──────┬───────────┬───────┐
| │ before ┆ t_a ┆ t_b ┆ t_c ┆ t_d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪═════╪═════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴─────┴─────┴──────┴───────────┴───────┘
|
| >>> df = pl.LazyFrame(
| ... {
| ... "before": ["foo", "bar"],
| ... "t_a": [1, 2],
| ... "t_b": ["a", "b"],
| ... "t_c": [True, None],
| ... "t_d": [[1, 2], [3]],
| ... "after": ["baz", "womp"],
| ... }
| ... ).select(
| ... "before",
| ... pl.struct(pl.col("^t_.$").name.map(lambda t: t[2:])).alias("t"),
| ... "after",
| ... )
| >>> df.unnest("t", separator="::").collect()
| shape: (2, 6)
| ┌────────┬──────┬──────┬──────┬───────────┬───────┐
| │ before ┆ t::a ┆ t::b ┆ t::c ┆ t::d ┆ after │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ str ┆ bool ┆ list[i64] ┆ str │
| ╞════════╪══════╪══════╪══════╪═══════════╪═══════╡
| │ foo ┆ 1 ┆ a ┆ true ┆ [1, 2] ┆ baz │
| │ bar ┆ 2 ┆ b ┆ null ┆ [3] ┆ womp │
| └────────┴──────┴──────┴──────┴───────────┴───────┘
|
| unpivot(self, on: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, *, index: 'ColumnNameOrSelector | Sequence[ColumnNameOrSelector] | None' = None, variable_name: 'str | None' = None, value_name: 'str | None' = None, streamable: 'bool' = True) -> 'LazyFrame'
| Unpivot a DataFrame from wide to long format.
|
| Optionally leaves identifiers set.
|
| This function is useful to massage a DataFrame into a format where one or more
| columns are identifier variables (index) while all other columns, considered
| measured variables (on), are "unpivoted" to the row axis leaving just
| two non-identifier columns, 'variable' and 'value'.
|
| Parameters
| ----------
| on
| Column(s) or selector(s) to use as values variables; if `on`
| is empty no columns will be used. If set to `None` (default)
| all columns that are not in `index` will be used.
| index
| Column(s) or selector(s) to use as identifier variables.
| variable_name
| Name to give to the `variable` column. Defaults to "variable"
| value_name
| Name to give to the `value` column. Defaults to "value"
| streamable
| deprecated
|
| Notes
| -----
| If you're coming from pandas, this is similar to `pandas.DataFrame.melt`,
| but with `index` replacing `id_vars` and `on` replacing `value_vars`.
| In other frameworks, you might know this operation as `pivot_longer`.
|
| The resulting row order is unspecified.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": ["x", "y", "z"],
| ... "b": [1, 3, 5],
| ... "c": [2, 4, 6],
| ... }
| ... )
| >>> import polars.selectors as cs
| >>> lf.unpivot(cs.numeric(), index="a").collect()
| shape: (6, 3)
| ┌─────┬──────────┬───────┐
| │ a ┆ variable ┆ value │
| │ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ i64 │
| ╞═════╪══════════╪═══════╡
| │ x ┆ b ┆ 1 │
| │ y ┆ b ┆ 3 │
| │ z ┆ b ┆ 5 │
| │ x ┆ c ┆ 2 │
| │ y ┆ c ┆ 4 │
| │ z ┆ c ┆ 6 │
| └─────┴──────────┴───────┘
|
| update(self, other: 'LazyFrame', on: 'str | Sequence[str] | None' = None, how: "Literal['left', 'inner', 'full']" = 'left', *, left_on: 'str | Sequence[str] | None' = None, right_on: 'str | Sequence[str] | None' = None, include_nulls: 'bool' = False, maintain_order: 'MaintainOrderJoin | None' = 'left') -> 'LazyFrame'
| Update the values in this `LazyFrame` with the values in `other`.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| other
| LazyFrame that will be used to update the values
| on
| Column names that will be joined on. If set to `None` (default),
| the implicit row index of each frame is used as a join key.
| how : {'left', 'inner', 'full'}
| * 'left' will keep all rows from the left table; rows may be duplicated
| if multiple rows in the right frame match the left row's key.
| * 'inner' keeps only those rows where the key exists in both frames.
| * 'full' will update existing rows where the key matches while also
| adding any new rows contained in the given frame.
| left_on
| Join column(s) of the left DataFrame.
| right_on
| Join column(s) of the right DataFrame.
| include_nulls
| Overwrite values in the left frame with null values from the right frame.
| If set to `False` (default), null values in the right frame are ignored.
| maintain_order : {'none', 'left', 'right', 'left_right', 'right_left'}
| Which order of rows from the inputs to preserve. See :func:`~LazyFrame.join`
| for details. Unlike `join` this function preserves the left order by
| default.
|
| Notes
| -----
| This is syntactic sugar for a left/inner join that preserves the order
| of the left `DataFrame` by default, with an optional coalesce when
| `include_nulls = False`.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "A": [1, 2, 3, 4],
| ... "B": [400, 500, 600, 700],
| ... }
| ... )
| >>> lf.collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 400 │
| │ 2 ┆ 500 │
| │ 3 ┆ 600 │
| │ 4 ┆ 700 │
| └─────┴─────┘
| >>> new_lf = pl.LazyFrame(
| ... {
| ... "B": [-66, None, -99],
| ... "C": [5, 3, 1],
| ... }
| ... )
|
| Update `df` values with the non-null values in `new_df`, by row index:
|
| >>> lf.update(new_lf).collect()
| shape: (4, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -66 │
| │ 2 ┆ 500 │
| │ 3 ┆ -99 │
| │ 4 ┆ 700 │
| └─────┴─────┘
|
| Update `df` values with the non-null values in `new_df`, by row index,
| but only keeping those rows that are common to both frames:
|
| >>> lf.update(new_lf, how="inner").collect()
| shape: (3, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -66 │
| │ 2 ┆ 500 │
| │ 3 ┆ -99 │
| └─────┴─────┘
|
| Update `df` values with the non-null values in `new_df`, using a full
| outer join strategy that defines explicit join columns in each frame:
|
| >>> lf.update(new_lf, left_on=["A"], right_on=["C"], how="full").collect()
| shape: (5, 2)
| ┌─────┬─────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ -99 │
| │ 2 ┆ 500 │
| │ 3 ┆ 600 │
| │ 4 ┆ 700 │
| │ 5 ┆ -66 │
| └─────┴─────┘
|
| Update `df` values including null values in `new_df`, using a full
| outer join strategy that defines explicit join columns in each frame:
|
| >>> lf.update(
| ... new_lf, left_on="A", right_on="C", how="full", include_nulls=True
| ... ).collect()
| shape: (5, 2)
| ┌─────┬──────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════╡
| │ 1 ┆ -99 │
| │ 2 ┆ 500 │
| │ 3 ┆ null │
| │ 4 ┆ 700 │
| │ 5 ┆ -66 │
| └─────┴──────┘
|
| var(self, ddof: 'int' = 1) -> 'LazyFrame'
| Aggregate the columns in the LazyFrame to their variance value.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [1, 2, 1, 1],
| ... }
| ... )
| >>> lf.var().collect()
| shape: (1, 2)
| ┌──────────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════════╪══════╡
| │ 1.666667 ┆ 0.25 │
| └──────────┴──────┘
| >>> lf.var(ddof=0).collect()
| shape: (1, 2)
| ┌──────┬────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪════════╡
| │ 1.25 ┆ 0.1875 │
| └──────┴────────┘
|
| with_columns(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'LazyFrame'
| Add columns to this LazyFrame.
|
| Added columns will replace existing columns with the same name.
|
| Parameters
| ----------
| *exprs
| Column(s) to add, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names, other
| non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to add, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| LazyFrame
| A new LazyFrame with the columns added.
|
| Notes
| -----
| Creating a new LazyFrame using this method does not create a new copy of
| existing data.
|
| Examples
| --------
| Pass an expression to add it as a new column.
|
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 2, 3, 4],
| ... "b": [0.5, 4, 10, 13],
| ... "c": [True, True, False, True],
| ... }
| ... )
| >>> lf.with_columns((pl.col("a") ** 2).alias("a^2")).collect()
| shape: (4, 4)
| ┌─────┬──────┬───────┬─────┐
| │ a ┆ b ┆ c ┆ a^2 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 │
| ╞═════╪══════╪═══════╪═════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 │
| │ 2 ┆ 4.0 ┆ true ┆ 4 │
| │ 3 ┆ 10.0 ┆ false ┆ 9 │
| │ 4 ┆ 13.0 ┆ true ┆ 16 │
| └─────┴──────┴───────┴─────┘
|
| Added columns will replace existing columns with the same name.
|
| >>> lf.with_columns(pl.col("a").cast(pl.Float64)).collect()
| shape: (4, 3)
| ┌─────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╡
| │ 1.0 ┆ 0.5 ┆ true │
| │ 2.0 ┆ 4.0 ┆ true │
| │ 3.0 ┆ 10.0 ┆ false │
| │ 4.0 ┆ 13.0 ┆ true │
| └─────┴──────┴───────┘
|
| Multiple columns can be added using positional arguments.
|
| >>> lf.with_columns(
| ... (pl.col("a") ** 2).alias("a^2"),
| ... (pl.col("b") / 2).alias("b/2"),
| ... (pl.col("c").not_()).alias("not c"),
| ... ).collect()
| shape: (4, 6)
| ┌─────┬──────┬───────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ a^2 ┆ b/2 ┆ not c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 ┆ 0.25 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 4 ┆ 2.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 9 ┆ 5.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 16 ┆ 6.5 ┆ false │
| └─────┴──────┴───────┴─────┴──────┴───────┘
|
| Multiple columns can also be added by passing a list of expressions.
|
| >>> lf.with_columns(
| ... [
| ... (pl.col("a") ** 2).alias("a^2"),
| ... (pl.col("b") / 2).alias("b/2"),
| ... (pl.col("c").not_()).alias("not c"),
| ... ]
| ... ).collect()
| shape: (4, 6)
| ┌─────┬──────┬───────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ a^2 ┆ b/2 ┆ not c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ i64 ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪═════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 1 ┆ 0.25 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 4 ┆ 2.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 9 ┆ 5.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 16 ┆ 6.5 ┆ false │
| └─────┴──────┴───────┴─────┴──────┴───────┘
|
| Use keyword arguments to easily name your expression inputs.
|
| >>> lf.with_columns(
| ... ab=pl.col("a") * pl.col("b"),
| ... not_c=pl.col("c").not_(),
| ... ).collect()
| shape: (4, 5)
| ┌─────┬──────┬───────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ ab ┆ not_c │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ f64 ┆ bool │
| ╞═════╪══════╪═══════╪══════╪═══════╡
| │ 1 ┆ 0.5 ┆ true ┆ 0.5 ┆ false │
| │ 2 ┆ 4.0 ┆ true ┆ 8.0 ┆ false │
| │ 3 ┆ 10.0 ┆ false ┆ 30.0 ┆ true │
| │ 4 ┆ 13.0 ┆ true ┆ 52.0 ┆ false │
| └─────┴──────┴───────┴──────┴───────┘
|
| with_columns_seq(self, *exprs: 'IntoExpr | Iterable[IntoExpr]', **named_exprs: 'IntoExpr') -> 'LazyFrame'
| Add columns to this LazyFrame.
|
| Added columns will replace existing columns with the same name.
|
| This will run all expression sequentially instead of in parallel.
| Use this when the work per expression is cheap.
|
| Parameters
| ----------
| *exprs
| Column(s) to add, specified as positional arguments.
| Accepts expression input. Strings are parsed as column names, other
| non-expression inputs are parsed as literals.
| **named_exprs
| Additional columns to add, specified as keyword arguments.
| The columns will be renamed to the keyword used.
|
| Returns
| -------
| LazyFrame
| A new LazyFrame with the columns added.
|
| See Also
| --------
| with_columns
|
| with_context(self, other: 'Self | list[Self]') -> 'LazyFrame'
| Add an external context to the computation graph.
|
| .. deprecated:: 1.0.0
| Use :func:`concat` instead, with `how='horizontal'`
|
| This allows expressions to also access columns from DataFrames
| that are not part of this one.
|
| Parameters
| ----------
| other
| Lazy DataFrame to join with.
|
| Examples
| --------
| >>> lf = pl.LazyFrame({"a": [1, 2, 3], "b": ["a", "c", None]})
| >>> lf_other = pl.LazyFrame({"c": ["foo", "ham"]})
| >>> lf.with_context(lf_other).select( # doctest: +SKIP
| ... pl.col("b") + pl.col("c").first()
| ... ).collect()
| shape: (3, 1)
| ┌──────┐
| │ b │
| │ --- │
| │ str │
| ╞══════╡
| │ afoo │
| │ cfoo │
| │ null │
| └──────┘
|
| Fill nulls with the median from another DataFrame:
|
| >>> train_lf = pl.LazyFrame(
| ... {"feature_0": [-1.0, 0, 1], "feature_1": [-1.0, 0, 1]}
| ... )
| >>> test_lf = pl.LazyFrame(
| ... {"feature_0": [-1.0, None, 1], "feature_1": [-1.0, 0, 1]}
| ... )
| >>> test_lf.with_context( # doctest: +SKIP
| ... train_lf.select(pl.all().name.suffix("_train"))
| ... ).select(
| ... pl.col("feature_0").fill_null(pl.col("feature_0_train").median())
| ... ).collect()
| shape: (3, 1)
| ┌───────────┐
| │ feature_0 │
| │ --- │
| │ f64 │
| ╞═══════════╡
| │ -1.0 │
| │ 0.0 │
| │ 1.0 │
| └───────────┘
|
| with_row_count(self, name: 'str' = 'row_nr', offset: 'int' = 0) -> 'LazyFrame'
| Add a column at index 0 that counts the rows.
|
| .. deprecated:: 0.20.4
| Use the :meth:`with_row_index` method instead.
| Note that the default column name has changed from 'row_nr' to 'index'.
|
| Parameters
| ----------
| name
| Name of the column to add.
| offset
| Start the row count at this offset.
|
| Warnings
| --------
| This can have a negative effect on query performance.
| This may, for instance, block predicate pushdown optimization.
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> lf.with_row_count().collect() # doctest: +SKIP
| shape: (3, 3)
| ┌────────┬─────┬─────┐
| │ row_nr ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └────────┴─────┴─────┘
|
| with_row_index(self, name: 'str' = 'index', offset: 'int' = 0) -> 'LazyFrame'
| Add a row index as the first column in the LazyFrame.
|
| Parameters
| ----------
| name
| Name of the index column.
| offset
| Start the index at this offset. Cannot be negative.
|
| Warnings
| --------
| Using this function can have a negative effect on query performance.
| This may, for instance, block predicate pushdown optimization.
|
| Notes
| -----
| The resulting column does not have any special properties. It is a regular
| column of type `UInt32` (or `UInt64` in `polars[rt64]`).
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "a": [1, 3, 5],
| ... "b": [2, 4, 6],
| ... }
| ... )
| >>> lf.with_row_index().collect()
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ index ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞═══════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └───────┴─────┴─────┘
| >>> lf.with_row_index("id", offset=1000).collect()
| shape: (3, 3)
| ┌──────┬─────┬─────┐
| │ id ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞══════╪═════╪═════╡
| │ 1000 ┆ 1 ┆ 2 │
| │ 1001 ┆ 3 ┆ 4 │
| │ 1002 ┆ 5 ┆ 6 │
| └──────┴─────┴─────┘
|
| An index column can also be created using the expressions :func:`int_range`
| and :func:`len`.
|
| >>> lf.select(
| ... pl.int_range(pl.len(), dtype=pl.UInt32).alias("index"),
| ... pl.all(),
| ... ).collect()
| shape: (3, 3)
| ┌───────┬─────┬─────┐
| │ index ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ i64 ┆ i64 │
| ╞═══════╪═════╪═════╡
| │ 0 ┆ 1 ┆ 2 │
| │ 1 ┆ 3 ┆ 4 │
| │ 2 ┆ 5 ┆ 6 │
| └───────┴─────┴─────┘
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| deserialize(source: 'str | bytes | Path | IOBase', *, format: 'SerializationFormat' = 'binary') -> 'LazyFrame'
| Read a logical plan from a file to construct a LazyFrame.
|
| Parameters
| ----------
| source
| Path to a file or a file-like object (by file-like object, we refer to
| objects that have a `read()` method, such as a file handler (e.g.
| via builtin `open` function) or `BytesIO`).
| format
| The format with which the LazyFrame was serialized. Options:
|
| - `"binary"`: Deserialize from binary format (bytes). This is the default.
| - `"json"`: Deserialize from JSON format (string).
|
| Warnings
| --------
| This function uses :mod:`pickle` if the logical plan contains Python UDFs,
| and as such inherits the security implications. Deserializing can execute
| arbitrary code, so it should only be attempted on trusted data.
|
| See Also
| --------
| LazyFrame.serialize
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| >>> import io
| >>> lf = pl.LazyFrame({"a": [1, 2, 3]}).sum()
| >>> bytes = lf.serialize()
| >>> pl.LazyFrame.deserialize(io.BytesIO(bytes)).collect()
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 6 │
| └─────┘
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| columns
| Get the column names.
|
| Returns
| -------
| list of str
| A list containing the name of each column in order.
|
| Warnings
| --------
| Determining the column names of a LazyFrame requires resolving its schema,
| which is a potentially expensive operation.
| Using :meth:`collect_schema` is the idiomatic way of resolving the schema.
| This property exists only for symmetry with the DataFrame class.
|
| See Also
| --------
| collect_schema
| Schema.names
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6, 7, 8],
| ... "ham": ["a", "b", "c"],
| ... }
| ... ).select("foo", "bar")
| >>> lf.columns # doctest: +SKIP
| ['foo', 'bar']
|
| dtypes
| Get the column data types.
|
| Returns
| -------
| list of DataType
| A list containing the data type of each column in order.
|
| Warnings
| --------
| Determining the data types of a LazyFrame requires resolving its schema,
| which is a potentially expensive operation.
| Using :meth:`collect_schema` is the idiomatic way to resolve the schema.
| This property exists only for symmetry with the DataFrame class.
|
| See Also
| --------
| collect_schema
| Schema.dtypes
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.dtypes # doctest: +SKIP
| [Int64, Float64, String]
|
| schema
| Get an ordered mapping of column names to their data type.
|
| Warnings
| --------
| Resolving the schema of a LazyFrame is a potentially expensive operation.
| Using :meth:`collect_schema` is the idiomatic way to resolve the schema.
| This property exists only for symmetry with the DataFrame class.
|
| See Also
| --------
| collect_schema
| Schema
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [6.0, 7.0, 8.0],
| ... "ham": ["a", "b", "c"],
| ... }
| ... )
| >>> lf.schema # doctest: +SKIP
| Schema({'foo': Int64, 'bar': Float64, 'ham': String})
|
| width
| Get the number of columns.
|
| Returns
| -------
| int
|
| Warnings
| --------
| Determining the width of a LazyFrame requires resolving its schema,
| which is a potentially expensive operation.
| Using :meth:`collect_schema` is the idiomatic way to resolve the schema.
| This property exists only for symmetry with the DataFrame class.
|
| See Also
| --------
| collect_schema
| Schema.len
|
| Examples
| --------
| >>> lf = pl.LazyFrame(
| ... {
| ... "foo": [1, 2, 3],
| ... "bar": [4, 5, 6],
| ... }
| ... )
| >>> lf.width # doctest: +SKIP
| 2
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set[str]]', '_ldf': 'PyLazy...
|
| __hash__ = None
class QueryOptFlags(builtins.object)
| QueryOptFlags(*, predicate_pushdown: 'None | bool' = None, projection_pushdown: 'None | bool' = None, simplify_expression: 'None | bool' = None, slice_pushdown: 'None | bool' = None, comm_subplan_elim: 'None | bool' = None, comm_subexpr_elim: 'None | bool' = None, cluster_with_columns: 'None | bool' = None, collapse_joins: 'None | bool' = None, check_order_observe: 'None | bool' = None, fast_projection: 'None | bool' = None, sort_collapse: 'None | bool' = None, pre_partition_hive: 'None | bool' = None) -> 'None'
|
| The set of the optimizations considered during query optimization.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Methods defined here:
|
| __copy__(self) -> 'QueryOptFlags'
|
| __deepcopy__(self) -> 'QueryOptFlags'
|
| __init__(self, *, predicate_pushdown: 'None | bool' = None, projection_pushdown: 'None | bool' = None, simplify_expression: 'None | bool' = None, slice_pushdown: 'None | bool' = None, comm_subplan_elim: 'None | bool' = None, comm_subexpr_elim: 'None | bool' = None, cluster_with_columns: 'None | bool' = None, collapse_joins: 'None | bool' = None, check_order_observe: 'None | bool' = None, fast_projection: 'None | bool' = None, sort_collapse: 'None | bool' = None, pre_partition_hive: 'None | bool' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __str__(self) -> 'str'
| Return str(self).
|
| no_optimizations(self) -> 'None'
| Remove selected optimizations.
|
| update(self, *, predicate_pushdown: 'None | bool' = None, projection_pushdown: 'None | bool' = None, simplify_expression: 'None | bool' = None, slice_pushdown: 'None | bool' = None, comm_subplan_elim: 'None | bool' = None, comm_subexpr_elim: 'None | bool' = None, cluster_with_columns: 'None | bool' = None, collapse_joins: 'None | bool' = None, check_order_observe: 'None | bool' = None, fast_projection: 'None | bool' = None, sort_collapse: 'None | bool' = None, pre_partition_hive: 'None | bool' = None) -> 'QueryOptFlags'
| Update the current optimization flags.
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| none(*, predicate_pushdown: 'None | bool' = None, projection_pushdown: 'None | bool' = None, simplify_expression: 'None | bool' = None, slice_pushdown: 'None | bool' = None, comm_subplan_elim: 'None | bool' = None, comm_subexpr_elim: 'None | bool' = None, cluster_with_columns: 'None | bool' = None, collapse_joins: 'None | bool' = None, check_order_observe: 'None | bool' = None, fast_projection: 'None | bool' = None, sort_collapse: 'None | bool' = None, pre_partition_hive: 'None | bool' = None) -> 'QueryOptFlags'
| Create new empty set off optimizations.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| check_order_observe
| Do not maintain order if the order would not be observed.
|
| cluster_with_columns
| Cluster sequential `with_columns` calls to independent calls.
|
| comm_subexpr_elim
| Elide duplicate expressions and caches their outputs.
|
| comm_subplan_elim
| Elide duplicate plans and caches their outputs.
|
| fast_projection
| Replace simple projections with a faster inlined projection that skips the expression engine.
|
| pre_partition_hive
| Prepartition hive-partitioned joins on their partition key (requires `predicate_pushdown`).
|
| predicate_pushdown
| Apply predicates/filters as early as possible.
|
| projection_pushdown
| Only read columns that are used later in the query.
|
| simplify_expression
| Run many expression optimization rules until fixed point.
|
| slice_pushdown
| Pushdown slices/limits.
|
| sort_collapse
| Collapse sequential sort nodes into a single sort node.
class QueryResult(typing.Protocol)
| QueryResult(*args, **kwargs)
|
| The result of a Polars query.
|
| .. note::
| This object should not be instantiated directly by the user.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Method resolution order:
| QueryResult
| typing.Protocol
| typing.Generic
| builtins.object
|
| Methods defined here:
|
| __init__ = _no_init_or_replace_init(self, *args, **kwargs) from typing
|
| lazy(self) -> 'LazyFrame'
| Convert the `QueryResult` into a `LazyFrame`.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __subclasshook__ = _proto_hook(other) from typing
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| head
| The first n rows of the result.
|
| n_rows_total
| Total rows that are outputted by the result.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| __non_callable_proto_members__ = {'head', 'n_rows_total'}
|
| __parameters__ = ()
|
| __protocol_attrs__ = {'head', 'lazy', 'n_rows_total'}
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Protocol:
|
| __init_subclass__(*args, **kwargs)
| Function to initialize subclasses.
|
| ----------------------------------------------------------------------
| Class methods inherited from typing.Generic:
|
| __class_getitem__(...)
| Parameterizes a generic class.
|
| At least, parameterizing a generic class is the *main* thing this
| method does. For example, for some generic class `Foo`, this is called
| when we do `Foo[int]` - there, with `cls=Foo` and `params=int`.
|
| However, note that this method is also called when defining generic
| classes in the first place with `class Foo[T]: ...`.
class SingleNodeQueryResult(builtins.object)
| SingleNodeQueryResult(df: 'DataFrame') -> 'None'
|
| The result of a Polars query.
|
| .. note::
| This object should not be instantiated directly by the user.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Methods defined here:
|
| __init__(self, df: 'DataFrame') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| lazy(self) -> 'LazyFrame'
| Convert the `QueryResult` into a `LazyFrame`.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| head
| The first n rows of the result.
|
| n_rows_total
| Total rows that are outputted by the result.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
DATA
__all__ = ['GPUEngine', 'LazyFrame', 'QueryOptFlags', 'QueryResult', '...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\lazyframe\__init__.py
polars.len
Python Library Documentation: function len in module polars.functions.len
len() -> 'Expr'
Return the number of rows in the context.
This is similar to `COUNT(*)` in SQL.
Returns
-------
Expr
Expression of data type :class:`UInt32`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, None],
... "b": [3, None, None],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.len())
shape: (1, 1)
┌─────┐
│ len │
│ --- │
│ u32 │
╞═════╡
│ 3 │
└─────┘
Generate an index column by using `len` in conjunction with :func:`int_range`.
>>> df.select(
... pl.int_range(pl.len(), dtype=pl.UInt32).alias("index"),
... pl.all(),
... )
shape: (3, 4)
┌───────┬──────┬──────┬─────┐
│ index ┆ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- ┆ --- │
│ u32 ┆ i64 ┆ i64 ┆ str │
╞═══════╪══════╪══════╪═════╡
│ 0 ┆ 1 ┆ 3 ┆ foo │
│ 1 ┆ 2 ┆ null ┆ bar │
│ 2 ┆ null ┆ null ┆ foo │
└───────┴──────┴──────┴─────┘
polars.linear_space
Python Library Documentation: function linear_space in module polars.functions.range.linear_space
linear_space(start: 'NumericLiteral | TemporalLiteral | IntoExpr', end: 'NumericLiteral | TemporalLiteral | IntoExpr', num_samples: 'int | IntoExpr', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Expr | Series'
Create sequence of evenly-spaced points.
Parameters
----------
start
Lower bound of the range.
end
Upper bound of the range.
num_samples
Number of samples in the output sequence.
closed : {'both', 'left', 'right', 'none'}
Define which sides of the interval are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
.. warning::
This functionality is experimental. It may be changed at any point without it
being considered a breaking change.
Notes
-----
`linear_space` works with numeric and temporal dtypes. When the `start` and `end`
parameters are `Date` dtypes, the output sequence consists of equally-spaced
`Datetime` elements with millisecond precision.
Returns
-------
Expr or Series
Column of data type `:class:Time`.
Examples
--------
>>> pl.linear_space(start=0, end=1, num_samples=3, eager=True)
shape: (3,)
Series: 'literal' [f64]
[
0.0
0.5
1.0
]
>>> pl.linear_space(start=0, end=1, num_samples=3, closed="left", eager=True)
shape: (3,)
Series: 'literal' [f64]
[
0.0
0.333333
0.666667
]
>>> pl.linear_space(start=0, end=1, num_samples=3, closed="right", eager=True)
shape: (3,)
Series: 'literal' [f64]
[
0.333333
0.666667
1.0
]
>>> pl.linear_space(start=0, end=1, num_samples=3, closed="none", eager=True)
shape: (3,)
Series: 'literal' [f64]
[
0.25
0.5
0.75
]
>>> from datetime import time
>>> pl.linear_space(
... start=time(hour=1), end=time(hour=12), num_samples=3, eager=True
... )
shape: (3,)
Series: 'literal' [time]
[
01:00:00
06:30:00
12:00:00
]
`Date` endpoints generate a sequence of `Datetime` values:
>>> from datetime import date
>>> pl.linear_space(
... start=date(2025, 1, 1),
... end=date(2025, 2, 1),
... num_samples=3,
... closed="right",
... eager=True,
... )
shape: (3,)
Series: 'literal' [datetime[μs]]
[
2025-01-11 08:00:00
2025-01-21 16:00:00
2025-02-01 00:00:00
]
When `eager=False` (default), an expression is produced. You can generate a sequence
using the length of the dataframe:
>>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5]})
>>> df.with_columns(pl.linear_space(0, 1, pl.len()).alias("ls"))
shape: (5, 2)
┌─────┬──────┐
│ a ┆ ls │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪══════╡
│ 1 ┆ 0.0 │
│ 2 ┆ 0.25 │
│ 3 ┆ 0.5 │
│ 4 ┆ 0.75 │
│ 5 ┆ 1.0 │
└─────┴──────┘
polars.linear_spaces
Python Library Documentation: function linear_spaces in module polars.functions.range.linear_space
linear_spaces(start: 'NumericLiteral | TemporalLiteral | IntoExprColumn', end: 'NumericLiteral | TemporalLiteral | IntoExprColumn', num_samples: 'int | IntoExprColumn', *, closed: 'ClosedInterval' = 'both', as_array: 'bool' = False, eager: 'bool' = False) -> 'Expr | Series'
Generate a sequence of evenly-spaced values for each row between `start` and `end`.
The number of values in each sequence is determined by `num_samples`.
Parameters
----------
start
Lower bound of the range.
end
Upper bound of the range.
num_samples
Number of samples in the output sequence.
closed : {'both', 'left', 'right', 'none'}
Define which sides of the interval are closed (inclusive).
as_array
Return result as a fixed-length `Array`. `num_samples` must be a constant.
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
.. warning::
This functionality is experimental. It may be changed at any point without it
being considered a breaking change.
Returns
-------
Expr or Series
Column of data type `List(dtype)`.
See Also
--------
linear_space : Generate a single sequence of linearly-spaced values.
Examples
--------
>>> df = pl.DataFrame({"start": [1, -1], "end": [3, 2], "num_samples": [4, 5]})
>>> df.with_columns(ls=pl.linear_spaces("start", "end", "num_samples"))
shape: (2, 4)
┌───────┬─────┬─────────────┬────────────────────────┐
│ start ┆ end ┆ num_samples ┆ ls │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ list[f64] │
╞═══════╪═════╪═════════════╪════════════════════════╡
│ 1 ┆ 3 ┆ 4 ┆ [1.0, 1.666667, … 3.0] │
│ -1 ┆ 2 ┆ 5 ┆ [-1.0, -0.25, … 2.0] │
└───────┴─────┴─────────────┴────────────────────────┘
>>> df.with_columns(ls=pl.linear_spaces("start", "end", 3, as_array=True))
shape: (2, 4)
┌───────┬─────┬─────────────┬──────────────────┐
│ start ┆ end ┆ num_samples ┆ ls │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ array[f64, 3] │
╞═══════╪═════╪═════════════╪══════════════════╡
│ 1 ┆ 3 ┆ 4 ┆ [1.0, 2.0, 3.0] │
│ -1 ┆ 2 ┆ 5 ┆ [-1.0, 0.5, 2.0] │
└───────┴─────┴─────────────┴──────────────────┘
polars.list
Python Library Documentation: function list in module polars.functions.as_datatype
list(exprs: 'IntoExpr | Iterable[IntoExpr]', *more_exprs: 'IntoExpr') -> 'Expr'
Collect columns into a list column, treating each expression's value as one element.
Unlike :func:`concat_list`, list-typed inputs are not extended — each input's value
becomes a single element of the output list. This means ``List(T)`` inputs produce
``List(List(T))`` output.
Parameters
----------
exprs
Columns to collect into a list. Accepts expression input. Strings are parsed
as column names, other non-expression inputs are parsed as literals.
*more_exprs
Additional columns, specified as positional arguments.
Examples
--------
Wrap scalar columns into a list (same as ``concat_list`` for scalars):
>>> df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
>>> df.with_columns(a_b=pl.list("a", "b"))
shape: (3, 3)
┌─────┬─────┬───────────┐
│ a ┆ b ┆ a_b │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ list[i64] │
╞═════╪═════╪═══════════╡
│ 1 ┆ 4 ┆ [1, 4] │
│ 2 ┆ 5 ┆ [2, 5] │
│ 3 ┆ 6 ┆ [3, 6] │
└─────┴─────┴───────────┘
Collect list columns into a list-of-lists (unlike ``concat_list``, which extends):
>>> df = pl.DataFrame({"a": [[1, 2], [3], [4, 5]], "b": [[6], [7, 8], [9]]})
>>> df.with_columns(a_b=pl.list("a", "b"))
shape: (3, 3)
┌───────────┬───────────┬─────────────────┐
│ a ┆ b ┆ a_b │
│ --- ┆ --- ┆ --- │
│ list[i64] ┆ list[i64] ┆ list[list[i64]] │
╞═══════════╪═══════════╪═════════════════╡
│ [1, 2] ┆ [6] ┆ [[1, 2], [6]] │
│ [3] ┆ [7, 8] ┆ [[3], [7, 8]] │
│ [4, 5] ┆ [9] ┆ [[4, 5], [9]] │
└───────────┴───────────┴─────────────────┘
polars.lit
Python Library Documentation: function lit in module polars.functions.lit
lit(value: 'Any', dtype: 'PolarsDataType | DataTypeExpr | None' = None, *, allow_object: 'bool' = False) -> 'Expr'
Return an expression representing a literal value.
Parameters
----------
value
Value that should be used as a `literal`.
dtype
The data type of the resulting expression.
If set to `None` (default), the data type is inferred from the `value` input.
allow_object
If type is unknown use an 'object' type.
By default, we will raise a `ValueException`
if the type is unknown.
Notes
-----
Expected datatypes:
- `pl.lit([])` -> empty List<Null>
- `pl.lit([1, 2, 3])` -> List<i64>
- `pl.lit(pl.Series([]))`-> empty Series Null
- `pl.lit(pl.Series([1, 2, 3]))` -> Series Int64
- `pl.lit(None)` -> Null
Examples
--------
Literal scalar values:
>>> pl.lit(1) # doctest: +IGNORE_RESULT
>>> pl.lit(5.5) # doctest: +IGNORE_RESULT
>>> pl.lit(None) # doctest: +IGNORE_RESULT
>>> pl.lit("foo_bar") # doctest: +IGNORE_RESULT
>>> pl.lit(date(2021, 1, 20)) # doctest: +IGNORE_RESULT
>>> pl.lit(datetime(2023, 3, 31, 10, 30, 45)) # doctest: +IGNORE_RESULT
Literal list/Series data (1D):
>>> pl.lit([1, 2, 3]) # doctest: +SKIP
>>> pl.lit(pl.Series("x", [1, 2, 3])) # doctest: +IGNORE_RESULT
Literal list/Series data (2D):
>>> pl.lit([[1, 2], [3, 4]]) # doctest: +SKIP
>>> pl.lit(pl.Series("y", [[1, 2], [3, 4]])) # doctest: +IGNORE_RESULT
polars.map_batches
Python Library Documentation: function map_batches in module polars.functions.lazy
map_batches(exprs: 'Sequence[str | Expr]', function: 'Callable[[Sequence[Series]], Series | Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, is_elementwise: 'bool' = False, returns_scalar: 'bool' = False) -> 'Expr'
Map a custom function over multiple columns/expressions.
Produces a single Series result.
.. warning::
This method is much slower than the native expressions API.
Only use it if you cannot implement your logic otherwise.
Parameters
----------
exprs
Expression(s) representing the input Series to the function.
function
Function to apply over the input.
return_dtype
Datatype of the output Series.
It is recommended to set this whenever possible. If this is `None`, it tries
to infer the datatype by calling the function with dummy data and looking at
the output.
is_elementwise
Set to true if the operations is elementwise for better performance
and optimization.
An elementwise operations has unit or equal length for all inputs
and can be ran sequentially on slices without results being affected.
returns_scalar
If the function returns a scalar, by default it will be wrapped in
a list in the output, since the assumption is that the function
always returns something Series-like. If you want to keep the
result as a scalar, set this argument to True.
Notes
-----
A UDF passed to `map_batches` must be pure, meaning that it cannot modify
or depend on state other than its arguments. We may call the function
with arbitrary input data.
Returns
-------
Expr
Expression with the data type given by `return_dtype`.
Examples
--------
>>> def test_func(a, b, c):
... return a + b + c
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3, 4],
... "b": [4, 5, 6, 7],
... }
... )
>>>
>>> df.with_columns(
... (
... pl.struct(["a", "b"]).map_batches(
... lambda x: test_func(x.struct.field("a"), x.struct.field("b"), 1)
... )
... ).alias("a+b+c")
... )
shape: (4, 3)
┌─────┬─────┬───────┐
│ a ┆ b ┆ a+b+c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═══════╡
│ 1 ┆ 4 ┆ 6 │
│ 2 ┆ 5 ┆ 8 │
│ 3 ┆ 6 ┆ 10 │
│ 4 ┆ 7 ┆ 12 │
└─────┴─────┴───────┘
polars.map_groups
Python Library Documentation: function map_groups in module polars.functions.lazy
map_groups(exprs: 'Sequence[str | Expr]', function: 'Callable[[Sequence[Series]], Series | Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, is_elementwise: 'bool' = False, returns_scalar: 'bool' = False) -> 'Expr'
Apply a custom/user-defined function (UDF) in a GroupBy context.
.. warning::
This method is much slower than the native expressions API.
Only use it if you cannot implement your logic otherwise.
Parameters
----------
exprs
Expression(s) representing the input Series to the function.
function
Function to apply over the input; should be of type Callable[[Series], Series].
return_dtype
Datatype of the output Series.
It is recommended to set this whenever possible. If this is `None`, it tries
to infer the datatype by calling the function with dummy data and looking at
the output.
is_elementwise
Set to true if the operations is elementwise for better performance
and optimization.
An elementwise operations has unit or equal length for all inputs
and can be ran sequentially on slices without results being affected.
returns_scalar
If the function returns a single scalar as output.
Notes
-----
A UDF passed to `map_batches` must be pure, meaning that it cannot modify
or depend on state other than its arguments. Polars may call the function
with arbitrary input data.
Returns
-------
Expr
Expression with the data type given by `return_dtype`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "group": [1, 1, 2],
... "a": [1, 3, 3],
... "b": [5, 6, 7],
... }
... )
>>> df
shape: (3, 3)
┌───────┬─────┬─────┐
│ group ┆ a ┆ b │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═══════╪═════╪═════╡
│ 1 ┆ 1 ┆ 5 │
│ 1 ┆ 3 ┆ 6 │
│ 2 ┆ 3 ┆ 7 │
└───────┴─────┴─────┘
>>> (
... df.group_by("group").agg(
... pl.map_groups(
... exprs=["a", "b"],
... function=lambda list_of_series: (
... list_of_series[0] / list_of_series[0].sum() + list_of_series[1]
... ),
... return_dtype=pl.Float64,
... ).alias("my_custom_aggregation")
... )
... ).sort("group")
shape: (2, 2)
┌───────┬───────────────────────┐
│ group ┆ my_custom_aggregation │
│ --- ┆ --- │
│ i64 ┆ list[f64] │
╞═══════╪═══════════════════════╡
│ 1 ┆ [5.25, 6.75] │
│ 2 ┆ [8.0] │
└───────┴───────────────────────┘
The output for group `1` can be understood as follows:
- group `1` contains Series `'a': [1, 3]` and `'b': [5, 6]`
- applying the function to those lists of Series, one gets the output
`[1 / 4 + 5, 3 / 4 + 6]`, i.e. `[5.25, 6.75]`
polars.max
Python Library Documentation: function max in module polars.functions.aggregation.vertical
max(*names: 'str') -> 'Expr'
Get the maximum value.
Syntactic sugar for `col(names).max()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
See Also
--------
max_horizontal
Examples
--------
Get the maximum value of a column.
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.max("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 8 │
└─────┘
Get the maximum value of multiple columns.
>>> df.select(pl.max("^a|b$"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 8 ┆ 5 │
└─────┴─────┘
>>> df.select(pl.max("a", "b"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 8 ┆ 5 │
└─────┴─────┘
polars.max_horizontal
Python Library Documentation: function max_horizontal in module polars.functions.aggregation.horizontal
max_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Get the maximum value horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(max=pl.max_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬─────┐
│ a ┆ b ┆ c ┆ max │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ i64 │
╞═════╪══════╪═════╪═════╡
│ 1 ┆ 4 ┆ x ┆ 4 │
│ 8 ┆ 5 ┆ y ┆ 8 │
│ 3 ┆ null ┆ z ┆ 3 │
└─────┴──────┴─────┴─────┘
polars.mean
Python Library Documentation: function mean in module polars.functions.lazy
mean(*columns: 'str') -> 'Expr'
Get the mean value.
This function is syntactic sugar for `pl.col(columns).mean()`.
Parameters
----------
*columns
One or more column names.
See Also
--------
mean_horizontal
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.mean("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ f64 │
╞═════╡
│ 4.0 │
└─────┘
>>> df.select(pl.mean("a", "b"))
shape: (1, 2)
┌─────┬──────────┐
│ a ┆ b │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞═════╪══════════╡
│ 4.0 ┆ 3.666667 │
└─────┴──────────┘
polars.mean_horizontal
Python Library Documentation: function mean_horizontal in module polars.functions.aggregation.horizontal
mean_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]', ignore_nulls: 'bool' = True) -> 'Expr'
Compute the mean of all values horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
ignore_nulls
Ignore null values (default).
If set to `False`, any null value in the input will lead to a null output.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(mean=pl.mean_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬──────┐
│ a ┆ b ┆ c ┆ mean │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ f64 │
╞═════╪══════╪═════╪══════╡
│ 1 ┆ 4 ┆ x ┆ 2.5 │
│ 8 ┆ 5 ┆ y ┆ 6.5 │
│ 3 ┆ null ┆ z ┆ 3.0 │
└─────┴──────┴─────┴──────┘
polars.median
Python Library Documentation: function median in module polars.functions.lazy
median(*columns: 'str') -> 'Expr'
Get the median value.
This function is syntactic sugar for `pl.col(columns).median()`.
Parameters
----------
columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.median("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ f64 │
╞═════╡
│ 3.0 │
└─────┘
>>> df.select(pl.median("a", "b"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ f64 ┆ f64 │
╞═════╪═════╡
│ 3.0 ┆ 4.0 │
└─────┴─────┘
polars.merge_sorted
Python Library Documentation: function merge_sorted in module polars.functions.eager
merge_sorted(items: 'Iterable[PolarsType]', key: 'str | Sequence[str]', *, maintain_order: 'bool' = False) -> 'PolarsType'
Merge multiple sorted DataFrames or LazyFrames by the sorted key.
The output of this operation will also be sorted.
It is the callers responsibility that the frames
are sorted in ascending order by that key(s) otherwise
the output will not make sense.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
items
DataFrames or LazyFrames to merge.
key
Key column(s) that the frames are sorted by. A single column name or a
sequence of column names can be passed. When multiple keys are given the
frames are merged as if sorted by those keys in order.
maintain_order
If ``True``, the output is guaranteed to have left-biased ordering
for equal keys: rows from the left frame appear before rows from
the right frame when their keys are equal.
Examples
--------
>>> df0 = pl.DataFrame(
... {"name": ["steve", "elise", "bob"], "age": [42, 44, 18]}
... ).sort("age")
>>> df1 = pl.DataFrame(
... {"name": ["anna", "megan", "steve", "thomas"], "age": [21, 33, 17, 20]}
... ).sort("age")
>>> df2 = pl.DataFrame({"name": ["ida", "maya"], "age": [37, 27]}).sort("age")
>>> pl.merge_sorted([df0, df1, df2], key="age")
shape: (9, 2)
┌────────┬─────┐
│ name ┆ age │
│ --- ┆ --- │
│ str ┆ i64 │
╞════════╪═════╡
│ steve ┆ 17 │
│ bob ┆ 18 │
│ thomas ┆ 20 │
│ anna ┆ 21 │
│ maya ┆ 27 │
│ megan ┆ 33 │
│ ida ┆ 37 │
│ steve ┆ 42 │
│ elise ┆ 44 │
└────────┴─────┘
Notes
-----
Unless ``maintain_order=True``, no guarantee is given over the output
row order when the key is equal between dataframes.
The key(s) must be sorted in ascending order.
polars.meta
Python Library Documentation: package polars.meta in polars
NAME
polars.meta - Public functions that provide information about the Polars package or the environment it runs in.
PACKAGE CONTENTS
build
index_type
thread_pool
versions
FUNCTIONS
build_info() -> 'dict[str, Any]'
Return detailed Polars build information.
The dictionary with build information contains the following keys:
- `"compiler"`
- `"time"`
- `"dependencies"`
- `"features"`
- `"host"`
- `"target"`
- `"git"`
- `"version"`
If Polars was compiled without the `build_info` feature flag, only the `"version"`
key is included.
get_index_type() -> 'PolarsIntegerType'
Return the data type used for Polars indexing.
Returns
-------
PolarsIntegerType
:class:`UInt32` in regular Polars, :class:`UInt64` in bigidx Polars.
Examples
--------
>>> pl.get_index_type()
UInt32
show_versions() -> 'None'
Print out the version of Polars and its optional dependencies.
Examples
--------
>>> pl.show_versions() # doctest: +SKIP
--------Version info---------
Polars: 0.20.22
Index type: UInt32
Platform: macOS-14.4.1-arm64-arm-64bit
Python: 3.11.8 (main, Feb 6 2024, 21:21:21) [Clang 15.0.0 (clang-1500.1.0.2.5)]
LTS CPU: False
----Optional dependencies----
adbc_driver_manager: 0.11.0
altair: 5.4.0
cloudpickle: 3.0.0
connectorx: 0.3.2
deltalake: 0.17.1
fastexcel: 0.10.4
fsspec: 2023.12.2
gevent: 24.2.1
matplotlib: 3.8.4
numpy: 1.26.4
openpyxl: 3.1.2
pandas: 2.2.2
pyarrow: 16.0.0
pydantic: 2.7.1
pyiceberg: 0.7.1
sqlalchemy: 2.0.29
torch: 2.2.2
xlsx2csv: 0.8.2
xlsxwriter: 3.2.0
thread_pool_size() -> 'int'
Return the number of threads in the Polars thread pool.
Notes
-----
The thread pool size can be overridden by setting the `POLARS_MAX_THREADS`
environment variable before process start. The thread pool is not behind a
lock, so it cannot be modified once set. A reasonable use case for this might
be temporarily limiting the number of threads before importing Polars in a
PySpark UDF or similar context. Otherwise, it is strongly recommended not to
override this value as it will be set automatically by the engine.
Examples
--------
>>> pl.thread_pool_size() # doctest: +SKIP
16
threadpool_size() -> 'int'
Return the number of threads in the Polars thread pool.
.. deprecated:: 0.20.7
This function has been renamed to :func:`thread_pool_size`.
DATA
__all__ = ['build_info', 'get_index_type', 'show_versions', 'thread_po...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\meta\__init__.py
polars.min
Python Library Documentation: function min in module polars.functions.aggregation.vertical
min(*names: 'str') -> 'Expr'
Get the minimum value.
Syntactic sugar for `col(names).min()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
See Also
--------
min_horizontal
Examples
--------
Get the minimum value of a column.
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.min("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
└─────┘
Get the minimum value of multiple columns.
>>> df.select(pl.min("^a|b$"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 2 │
└─────┴─────┘
>>> df.select(pl.min("a", "b"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 2 │
└─────┴─────┘
polars.min_horizontal
Python Library Documentation: function min_horizontal in module polars.functions.aggregation.horizontal
min_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]') -> 'Expr'
Get the minimum value horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(min=pl.min_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬─────┐
│ a ┆ b ┆ c ┆ min │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ i64 │
╞═════╪══════╪═════╪═════╡
│ 1 ┆ 4 ┆ x ┆ 1 │
│ 8 ┆ 5 ┆ y ┆ 5 │
│ 3 ┆ null ┆ z ┆ 3 │
└─────┴──────┴─────┴─────┘
polars.n_unique
Python Library Documentation: function n_unique in module polars.functions.lazy
n_unique(*columns: 'str') -> 'Expr'
Count unique values.
This function is syntactic sugar for `pl.col(columns).n_unique()`.
`null` is considered to be a unique value for the purposes of this operation.
Parameters
----------
columns
One or more column names.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 1, None],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.n_unique("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ u32 │
╞═════╡
│ 2 │
└─────┘
>>> df.select(pl.n_unique("b", "c"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ u32 ┆ u32 │
╞═════╪═════╡
│ 3 ┆ 2 │
└─────┴─────┘
polars.nth
Python Library Documentation: function nth in module polars.functions.lazy
nth(*indices: 'int | Sequence[int]', strict: 'bool' = True) -> 'Expr'
Get the nth column(s) of the context.
Parameters
----------
indices
One or more indices representing the columns to retrieve.
strict
By default, all specified indices must be valid; if any index is out of bounds,
an error is raised. If set to `False`, out-of-bounds indices are ignored.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "baz"],
... }
... )
>>> df.select(pl.nth(1))
shape: (3, 1)
┌─────┐
│ b │
│ --- │
│ i64 │
╞═════╡
│ 4 │
│ 5 │
│ 2 │
└─────┘
>>> df.select(pl.nth(2, 0))
shape: (3, 2)
┌─────┬─────┐
│ c ┆ a │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═════╡
│ foo ┆ 1 │
│ bar ┆ 8 │
│ baz ┆ 3 │
└─────┴─────┘
polars.ones
Python Library Documentation: function ones in module polars.functions.repeat
ones(n: 'int | Expr', dtype: 'PolarsDataType' = Float64, *, eager: 'bool' = False) -> 'Expr | Series'
Construct a column of length `n` filled with ones.
This is syntactic sugar for the `repeat` function.
Parameters
----------
n
Length of the resulting column.
dtype
Data type of the resulting column. Defaults to Float64.
eager
Evaluate immediately and return a `Series`. If set to `False`,
return an expression instead.
Notes
-----
If you want to construct a column in lazy mode and do not need a pre-determined
length, use :func:`lit` instead.
See Also
--------
repeat
lit
Examples
--------
>>> pl.ones(3, pl.Int8, eager=True)
shape: (3,)
Series: 'ones' [i8]
[
1
1
1
]
polars.os
Python Library Documentation: module os
NAME
os - OS routines for NT or Posix depending on what system we're on.
MODULE REFERENCE
https://docs.python.org/3.12/library/os.html
The following documentation is automatically generated from the Python
source files. It may be incomplete, incorrect or include features that
are considered implementation detail and may vary between Python
implementations. When in doubt, consult the module reference at the
location listed above.
DESCRIPTION
This exports:
- all functions from posix or nt, e.g. unlink, stat, etc.
- os.path is either posixpath or ntpath
- os.name is either 'posix' or 'nt'
- os.curdir is a string representing the current directory (always '.')
- os.pardir is a string representing the parent directory (always '..')
- os.sep is the (or a most common) pathname separator ('/' or '\\')
- os.extsep is the extension separator (always '.')
- os.altsep is the alternate pathname separator (None or '/')
- os.pathsep is the component separator used in $PATH etc
- os.linesep is the line separator in text files ('\r' or '\n' or '\r\n')
- os.defpath is the default search path for executables
- os.devnull is the file path of the null device ('/dev/null', etc.)
Programs that import and use 'os' stand a better chance of being
portable between different platforms. Of course, they must then
only use functions that are defined by all platforms (e.g., unlink
and opendir), and leave all pathname manipulation to os.path
(e.g., split and join).
CLASSES
builtins.Exception(builtins.BaseException)
builtins.OSError
builtins.object
nt.DirEntry
builtins.tuple(builtins.object)
nt.times_result
nt.uname_result
stat_result
statvfs_result
terminal_size
class DirEntry(builtins.object)
| Methods defined here:
|
| __fspath__(self, /)
| Returns the path for the entry.
|
| __repr__(self, /)
| Return repr(self).
|
| inode(self, /)
| Return inode of the entry; cached per entry.
|
| is_dir(self, /, *, follow_symlinks=True)
| Return True if the entry is a directory; cached per entry.
|
| is_file(self, /, *, follow_symlinks=True)
| Return True if the entry is a file; cached per entry.
|
| is_junction(self, /)
| Return True if the entry is a junction; cached per entry.
|
| is_symlink(self, /)
| Return True if the entry is a symbolic link; cached per entry.
|
| stat(self, /, *, follow_symlinks=True)
| Return stat_result object for the entry; cached per entry.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| name
| the entry's base filename, relative to scandir() "path" argument
|
| path
| the entry's full path name; equivalent to os.path.join(scandir_path, entry.name)
error = class OSError(Exception)
| Base class for I/O related errors.
|
| Method resolution order:
| OSError
| Exception
| BaseException
| object
|
| Built-in subclasses:
| BlockingIOError
| ChildProcessError
| ConnectionError
| FileExistsError
| ... and 7 other subclasses
|
| Methods defined here:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __reduce__(...)
| Helper for pickle.
|
| __str__(self, /)
| Return str(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| characters_written
|
| errno
| POSIX exception code
|
| filename
| exception filename
|
| filename2
| second exception filename
|
| strerror
| exception strerror
|
| winerror
| Win32 exception code
|
| ----------------------------------------------------------------------
| Methods inherited from BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class stat_result(builtins.tuple)
| stat_result(iterable=(), /)
|
| stat_result: Result from stat, fstat, or lstat.
|
| This object may be accessed either as a tuple of
| (mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime)
| or via the attributes st_mode, st_ino, st_dev, st_nlink, st_uid, and so on.
|
| Posix/windows: If your platform supports st_blksize, st_blocks, st_rdev,
| or st_flags, they are available as attributes only.
|
| See os.stat for more information.
|
| Method resolution order:
| stat_result
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| st_atime
| time of last access
|
| st_atime_ns
| time of last access in nanoseconds
|
| st_birthtime
| time of creation
|
| st_birthtime_ns
| time of creation in nanoseconds
|
| st_ctime
| time of last change
|
| st_ctime_ns
| time of last change in nanoseconds
|
| st_dev
| device
|
| st_file_attributes
| Windows file attribute bits
|
| st_gid
| group ID of owner
|
| st_ino
| inode
|
| st_mode
| protection bits
|
| st_mtime
| time of last modification
|
| st_mtime_ns
| time of last modification in nanoseconds
|
| st_nlink
| number of hard links
|
| st_reparse_tag
| Windows reparse tag
|
| st_size
| total size, in bytes
|
| st_uid
| user ID of owner
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('st_mode', 'st_ino', 'st_dev', 'st_nlink', 'st_uid',...
|
| n_fields = 20
|
| n_sequence_fields = 10
|
| n_unnamed_fields = 3
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class statvfs_result(builtins.tuple)
| statvfs_result(iterable=(), /)
|
| statvfs_result: Result from statvfs or fstatvfs.
|
| This object may be accessed either as a tuple of
| (bsize, frsize, blocks, bfree, bavail, files, ffree, favail, flag, namemax),
| or via the attributes f_bsize, f_frsize, f_blocks, f_bfree, and so on.
|
| See os.statvfs for more information.
|
| Method resolution order:
| statvfs_result
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| f_bavail
|
| f_bfree
|
| f_blocks
|
| f_bsize
|
| f_favail
|
| f_ffree
|
| f_files
|
| f_flag
|
| f_frsize
|
| f_fsid
|
| f_namemax
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('f_bsize', 'f_frsize', 'f_blocks', 'f_bfree', 'f_bav...
|
| n_fields = 11
|
| n_sequence_fields = 10
|
| n_unnamed_fields = 0
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class terminal_size(builtins.tuple)
| terminal_size(iterable=(), /)
|
| A tuple of (columns, lines) for holding terminal window size
|
| Method resolution order:
| terminal_size
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| columns
| width of the terminal window in characters
|
| lines
| height of the terminal window in characters
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('columns', 'lines')
|
| n_fields = 2
|
| n_sequence_fields = 2
|
| n_unnamed_fields = 0
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class times_result(builtins.tuple)
| times_result(iterable=(), /)
|
| times_result: Result from os.times().
|
| This object may be accessed either as a tuple of
| (user, system, children_user, children_system, elapsed),
| or via the attributes user, system, children_user, children_system,
| and elapsed.
|
| See os.times for more information.
|
| Method resolution order:
| times_result
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| children_system
| system time of children
|
| children_user
| user time of children
|
| elapsed
| elapsed time since an arbitrary point in the past
|
| system
| system time
|
| user
| user time
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('user', 'system', 'children_user', 'children_system'...
|
| n_fields = 5
|
| n_sequence_fields = 5
|
| n_unnamed_fields = 0
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class uname_result(builtins.tuple)
| uname_result(iterable=(), /)
|
| uname_result: Result from os.uname().
|
| This object may be accessed either as a tuple of
| (sysname, nodename, release, version, machine),
| or via the attributes sysname, nodename, release, version, and machine.
|
| See os.uname for more information.
|
| Method resolution order:
| uname_result
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| machine
| hardware identifier
|
| nodename
| name of machine on network (implementation-defined)
|
| release
| operating system release
|
| sysname
| operating system name
|
| version
| operating system version
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('sysname', 'nodename', 'release', 'version', 'machin...
|
| n_fields = 5
|
| n_sequence_fields = 5
|
| n_unnamed_fields = 0
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(self, /)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
FUNCTIONS
_exit(status)
Exit to the system with specified status, without normal exit processing.
abort()
Abort the interpreter immediately.
This function 'dumps core' or otherwise fails in the hardest way possible
on the hosting operating system. This function never returns.
access(path, mode, *, dir_fd=None, effective_ids=False, follow_symlinks=True)
Use the real uid/gid to test for access to a path.
path
Path to be tested; can be string, bytes, or a path-like object.
mode
Operating-system mode bitfield. Can be F_OK to test existence,
or the inclusive-OR of R_OK, W_OK, and X_OK.
dir_fd
If not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that
directory.
effective_ids
If True, access will use the effective uid/gid instead of
the real uid/gid.
follow_symlinks
If False, and the last element of the path is a symbolic link,
access will examine the symbolic link itself instead of the file
the link points to.
dir_fd, effective_ids, and follow_symlinks may not be implemented
on your platform. If they are unavailable, using them will raise a
NotImplementedError.
Note that most operations will use the effective uid/gid, therefore this
routine can be used in a suid/sgid environment to test if the invoking user
has the specified access to the path.
chdir(path)
Change the current working directory to the specified path.
path may always be specified as a string.
On some platforms, path may also be specified as an open file descriptor.
If this functionality is unavailable, using it raises an exception.
chmod(path, mode, *, dir_fd=None, follow_symlinks=True)
Change the access permissions of a file.
path
Path to be modified. May always be specified as a str, bytes, or a path-like object.
On some platforms, path may also be specified as an open file descriptor.
If this functionality is unavailable, using it raises an exception.
mode
Operating-system mode bitfield.
Be careful when using number literals for *mode*. The conventional UNIX notation for
numeric modes uses an octal base, which needs to be indicated with a ``0o`` prefix in
Python.
dir_fd
If not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that
directory.
follow_symlinks
If False, and the last element of the path is a symbolic link,
chmod will modify the symbolic link itself instead of the file
the link points to.
It is an error to use dir_fd or follow_symlinks when specifying path as
an open file descriptor.
dir_fd and follow_symlinks may not be implemented on your platform.
If they are unavailable, using them will raise a NotImplementedError.
close(fd)
Close a file descriptor.
closerange(fd_low, fd_high, /)
Closes all file descriptors in [fd_low, fd_high), ignoring errors.
cpu_count()
Return the number of CPUs in the system; return None if indeterminable.
This number is not equivalent to the number of CPUs the current process can
use. The number of usable CPUs can be obtained with
``len(os.sched_getaffinity(0))``
device_encoding(fd)
Return a string describing the encoding of a terminal's file descriptor.
The file descriptor must be attached to a terminal.
If the device is not a terminal, return None.
dup(fd, /)
Return a duplicate of a file descriptor.
dup2(fd, fd2, inheritable=True)
Duplicate file descriptor.
execl(file, *args)
execl(file, *args)
Execute the executable file with argument list args, replacing the
current process.
execle(file, *args)
execle(file, *args, env)
Execute the executable file with argument list args and
environment env, replacing the current process.
execlp(file, *args)
execlp(file, *args)
Execute the executable file (which is searched for along $PATH)
with argument list args, replacing the current process.
execlpe(file, *args)
execlpe(file, *args, env)
Execute the executable file (which is searched for along $PATH)
with argument list args and environment env, replacing the current
process.
execv(path, argv, /)
Execute an executable path with arguments, replacing current process.
path
Path of executable file.
argv
Tuple or list of strings.
execve(path, argv, env)
Execute an executable path with arguments, replacing current process.
path
Path of executable file.
argv
Tuple or list of strings.
env
Dictionary of strings mapping to strings.
execvp(file, args)
execvp(file, args)
Execute the executable file (which is searched for along $PATH)
with argument list args, replacing the current process.
args may be a list or tuple of strings.
execvpe(file, args, env)
execvpe(file, args, env)
Execute the executable file (which is searched for along $PATH)
with argument list args and environment env, replacing the
current process.
args may be a list or tuple of strings.
fdopen(fd, mode='r', buffering=-1, encoding=None, *args, **kwargs)
fsdecode(filename)
Decode filename (an os.PathLike, bytes, or str) from the filesystem
encoding with 'surrogateescape' error handler, return str unchanged. On
Windows, use 'strict' error handler if the file system encoding is
'mbcs' (which is the default encoding).
fsencode(filename)
Encode filename (an os.PathLike, bytes, or str) to the filesystem
encoding with 'surrogateescape' error handler, return bytes unchanged.
On Windows, use 'strict' error handler if the file system encoding is
'mbcs' (which is the default encoding).
fspath(path)
Return the file system path representation of the object.
If the object is str or bytes, then allow it to pass through as-is. If the
object defines __fspath__(), then return the result of that method. All other
types raise a TypeError.
fstat(fd)
Perform a stat system call on the given file descriptor.
Like stat(), but for an open file descriptor.
Equivalent to os.stat(fd).
fsync(fd)
Force write of fd to disk.
ftruncate(fd, length, /)
Truncate a file, specified by file descriptor, to a specific length.
get_blocking(fd, /)
Get the blocking mode of the file descriptor.
Return False if the O_NONBLOCK flag is set, True if the flag is cleared.
get_exec_path(env=None)
Returns the sequence of directories that will be searched for the
named executable (similar to a shell) when launching a process.
*env* must be an environment variable dict or None. If *env* is None,
os.environ will be used.
get_handle_inheritable(handle, /)
Get the close-on-exe flag of the specified file descriptor.
get_inheritable(fd, /)
Get the close-on-exe flag of the specified file descriptor.
get_terminal_size(...)
Return the size of the terminal window as (columns, lines).
The optional argument fd (default standard output) specifies
which file descriptor should be queried.
If the file descriptor is not connected to a terminal, an OSError
is thrown.
This function will only be defined if an implementation is
available for this system.
shutil.get_terminal_size is the high-level function which should
normally be used, os.get_terminal_size is the low-level implementation.
getcwd()
Return a unicode string representing the current working directory.
getcwdb()
Return a bytes string representing the current working directory.
getenv(key, default=None)
Get an environment variable, return None if it doesn't exist.
The optional second argument can specify an alternate default.
key, default and the result are str.
getlogin()
Return the actual login name.
getpid()
Return the current process id.
getppid()
Return the parent's process id.
If the parent process has already exited, Windows machines will still
return its id; others systems will return the id of the 'init' process (1).
isatty(fd, /)
Return True if the fd is connected to a terminal.
Return True if the file descriptor is an open file descriptor
connected to the slave end of a terminal.
kill(pid, signal, /)
Kill a process with a signal.
link(src, dst, *, src_dir_fd=None, dst_dir_fd=None, follow_symlinks=True)
Create a hard link to a file.
If either src_dir_fd or dst_dir_fd is not None, it should be a file
descriptor open to a directory, and the respective path string (src or dst)
should be relative; the path will then be relative to that directory.
If follow_symlinks is False, and the last element of src is a symbolic
link, link will create a link to the symbolic link itself instead of the
file the link points to.
src_dir_fd, dst_dir_fd, and follow_symlinks may not be implemented on your
platform. If they are unavailable, using them will raise a
NotImplementedError.
listdir(path=None)
Return a list containing the names of the files in the directory.
path can be specified as either str, bytes, or a path-like object. If path is bytes,
the filenames returned will also be bytes; in all other circumstances
the filenames returned will be str.
If path is None, uses the path='.'.
On some platforms, path may also be specified as an open file descriptor;\
the file descriptor must refer to a directory.
If this functionality is unavailable, using it raises NotImplementedError.
The list is in arbitrary order. It does not include the special
entries '.' and '..' even if they are present in the directory.
listdrives()
Return a list containing the names of drives in the system.
A drive name typically looks like 'C:\\'.
listmounts(volume)
Return a list containing mount points for a particular volume.
'volume' should be a GUID path as returned from os.listvolumes.
listvolumes()
Return a list containing the volumes in the system.
Volumes are typically represented as a GUID path.
lseek(fd, position, whence, /)
Set the position of a file descriptor. Return the new position.
fd
An open file descriptor, as returned by os.open().
position
Position, interpreted relative to 'whence'.
whence
The relative position to seek from. Valid values are:
- SEEK_SET: seek from the start of the file.
- SEEK_CUR: seek from the current file position.
- SEEK_END: seek from the end of the file.
The return value is the number of bytes relative to the beginning of the file.
lstat(path, *, dir_fd=None)
Perform a stat system call on the given path, without following symbolic links.
Like stat(), but do not follow symbolic links.
Equivalent to stat(path, follow_symlinks=False).
makedirs(name, mode=511, exist_ok=False)
makedirs(name [, mode=0o777][, exist_ok=False])
Super-mkdir; create a leaf directory and all intermediate ones. Works like
mkdir, except that any intermediate path segment (not just the rightmost)
will be created if it does not exist. If the target directory already
exists, raise an OSError if exist_ok is False. Otherwise no exception is
raised. This is recursive.
mkdir(path, mode=511, *, dir_fd=None)
Create a directory.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
The mode argument is ignored on Windows. Where it is used, the current umask
value is first masked out.
open(path, flags, mode=511, *, dir_fd=None)
Open a file for low level IO. Returns a file descriptor (integer).
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
pipe()
Create a pipe.
Returns a tuple of two file descriptors:
(read_fd, write_fd)
popen(cmd, mode='r', buffering=-1)
putenv(name, value, /)
Change or add an environment variable.
read(fd, length, /)
Read from a file descriptor. Returns a bytes object.
readlink(path, *, dir_fd=None)
Return a string representing the path to which the symbolic link points.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform. If it is unavailable,
using it will raise a NotImplementedError.
remove(path, *, dir_fd=None)
Remove a file (same as unlink()).
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
removedirs(name)
removedirs(name)
Super-rmdir; remove a leaf directory and all empty intermediate
ones. Works like rmdir except that, if the leaf directory is
successfully removed, directories corresponding to rightmost path
segments will be pruned away until either the whole path is
consumed or an error occurs. Errors during this latter phase are
ignored -- they generally mean that a directory was not empty.
rename(src, dst, *, src_dir_fd=None, dst_dir_fd=None)
Rename a file or directory.
If either src_dir_fd or dst_dir_fd is not None, it should be a file
descriptor open to a directory, and the respective path string (src or dst)
should be relative; the path will then be relative to that directory.
src_dir_fd and dst_dir_fd, may not be implemented on your platform.
If they are unavailable, using them will raise a NotImplementedError.
renames(old, new)
renames(old, new)
Super-rename; create directories as necessary and delete any left
empty. Works like rename, except creation of any intermediate
directories needed to make the new pathname good is attempted
first. After the rename, directories corresponding to rightmost
path segments of the old name will be pruned until either the
whole path is consumed or a nonempty directory is found.
Note: this function can fail with the new directory structure made
if you lack permissions needed to unlink the leaf directory or
file.
replace(src, dst, *, src_dir_fd=None, dst_dir_fd=None)
Rename a file or directory, overwriting the destination.
If either src_dir_fd or dst_dir_fd is not None, it should be a file
descriptor open to a directory, and the respective path string (src or dst)
should be relative; the path will then be relative to that directory.
src_dir_fd and dst_dir_fd, may not be implemented on your platform.
If they are unavailable, using them will raise a NotImplementedError.
rmdir(path, *, dir_fd=None)
Remove a directory.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
scandir(path=None)
Return an iterator of DirEntry objects for given path.
path can be specified as either str, bytes, or a path-like object. If path
is bytes, the names of yielded DirEntry objects will also be bytes; in
all other circumstances they will be str.
If path is None, uses the path='.'.
set_blocking(fd, blocking, /)
Set the blocking mode of the specified file descriptor.
Set the O_NONBLOCK flag if blocking is False,
clear the O_NONBLOCK flag otherwise.
set_handle_inheritable(handle, inheritable, /)
Set the inheritable flag of the specified handle.
set_inheritable(fd, inheritable, /)
Set the inheritable flag of the specified file descriptor.
spawnl(mode, file, *args)
spawnl(mode, file, *args) -> integer
Execute file with arguments from args in a subprocess.
If mode == P_NOWAIT return the pid of the process.
If mode == P_WAIT return the process's exit code if it exits normally;
otherwise return -SIG, where SIG is the signal that killed it.
spawnle(mode, file, *args)
spawnle(mode, file, *args, env) -> integer
Execute file with arguments from args in a subprocess with the
supplied environment.
If mode == P_NOWAIT return the pid of the process.
If mode == P_WAIT return the process's exit code if it exits normally;
otherwise return -SIG, where SIG is the signal that killed it.
spawnv(mode, path, argv, /)
Execute the program specified by path in a new process.
mode
Mode of process creation.
path
Path of executable file.
argv
Tuple or list of strings.
spawnve(mode, path, argv, env, /)
Execute the program specified by path in a new process.
mode
Mode of process creation.
path
Path of executable file.
argv
Tuple or list of strings.
env
Dictionary of strings mapping to strings.
startfile(...)
Start a file with its associated application.
When "operation" is not specified or "open", this acts like
double-clicking the file in Explorer, or giving the file name as an
argument to the DOS "start" command: the file is opened with whatever
application (if any) its extension is associated.
When another "operation" is given, it specifies what should be done with
the file. A typical operation is "print".
"arguments" is passed to the application, but should be omitted if the
file is a document.
"cwd" is the working directory for the operation. If "filepath" is
relative, it will be resolved against this directory. This argument
should usually be an absolute path.
"show_cmd" can be used to override the recommended visibility option.
See the Windows ShellExecute documentation for values.
startfile returns as soon as the associated application is launched.
There is no option to wait for the application to close, and no way
to retrieve the application's exit status.
The filepath is relative to the current directory. If you want to use
an absolute path, make sure the first character is not a slash ("/");
the underlying Win32 ShellExecute function doesn't work if it is.
stat(path, *, dir_fd=None, follow_symlinks=True)
Perform a stat system call on the given path.
path
Path to be examined; can be string, bytes, a path-like object or
open-file-descriptor int.
dir_fd
If not None, it should be a file descriptor open to a directory,
and path should be a relative string; path will then be relative to
that directory.
follow_symlinks
If False, and the last element of the path is a symbolic link,
stat will examine the symbolic link itself instead of the file
the link points to.
dir_fd and follow_symlinks may not be implemented
on your platform. If they are unavailable, using them will raise a
NotImplementedError.
It's an error to use dir_fd or follow_symlinks when specifying path as
an open file descriptor.
strerror(code, /)
Translate an error code to a message string.
symlink(src, dst, target_is_directory=False, *, dir_fd=None)
Create a symbolic link pointing to src named dst.
target_is_directory is required on Windows if the target is to be
interpreted as a directory. (On Windows, symlink requires
Windows 6.0 or greater, and raises a NotImplementedError otherwise.)
target_is_directory is ignored on non-Windows platforms.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
system(command)
Execute the command in a subshell.
times()
Return a collection containing process timing information.
The object returned behaves like a named tuple with these fields:
(utime, stime, cutime, cstime, elapsed_time)
All fields are floating-point numbers.
truncate(path, length)
Truncate a file, specified by path, to a specific length.
On some platforms, path may also be specified as an open file descriptor.
If this functionality is unavailable, using it raises an exception.
umask(mask, /)
Set the current numeric umask and return the previous umask.
unlink(path, *, dir_fd=None)
Remove a file (same as remove()).
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
dir_fd may not be implemented on your platform.
If it is unavailable, using it will raise a NotImplementedError.
unsetenv(name, /)
Delete an environment variable.
urandom(size, /)
Return a bytes object containing random bytes suitable for cryptographic use.
utime(...)
Set the access and modified time of path.
path may always be specified as a string.
On some platforms, path may also be specified as an open file descriptor.
If this functionality is unavailable, using it raises an exception.
If times is not None, it must be a tuple (atime, mtime);
atime and mtime should be expressed as float seconds since the epoch.
If ns is specified, it must be a tuple (atime_ns, mtime_ns);
atime_ns and mtime_ns should be expressed as integer nanoseconds
since the epoch.
If times is None and ns is unspecified, utime uses the current time.
Specifying tuples for both times and ns is an error.
If dir_fd is not None, it should be a file descriptor open to a directory,
and path should be relative; path will then be relative to that directory.
If follow_symlinks is False, and the last element of the path is a symbolic
link, utime will modify the symbolic link itself instead of the file the
link points to.
It is an error to use dir_fd or follow_symlinks when specifying path
as an open file descriptor.
dir_fd and follow_symlinks may not be available on your platform.
If they are unavailable, using them will raise a NotImplementedError.
waitpid(pid, options, /)
Wait for completion of a given process.
Returns a tuple of information regarding the process:
(pid, status << 8)
The options argument is ignored on Windows.
waitstatus_to_exitcode(status)
Convert a wait status to an exit code.
On Unix:
* If WIFEXITED(status) is true, return WEXITSTATUS(status).
* If WIFSIGNALED(status) is true, return -WTERMSIG(status).
* Otherwise, raise a ValueError.
On Windows, return status shifted right by 8 bits.
On Unix, if the process is being traced or if waitpid() was called with
WUNTRACED option, the caller must first check if WIFSTOPPED(status) is true.
This function must not be called if WIFSTOPPED(status) is true.
walk(top, topdown=True, onerror=None, followlinks=False)
Directory tree generator.
For each directory in the directory tree rooted at top (including top
itself, but excluding '.' and '..'), yields a 3-tuple
dirpath, dirnames, filenames
dirpath is a string, the path to the directory. dirnames is a list of
the names of the subdirectories in dirpath (including symlinks to directories,
and excluding '.' and '..').
filenames is a list of the names of the non-directory files in dirpath.
Note that the names in the lists are just names, with no path components.
To get a full path (which begins with top) to a file or directory in
dirpath, do os.path.join(dirpath, name).
If optional arg 'topdown' is true or not specified, the triple for a
directory is generated before the triples for any of its subdirectories
(directories are generated top down). If topdown is false, the triple
for a directory is generated after the triples for all of its
subdirectories (directories are generated bottom up).
When topdown is true, the caller can modify the dirnames list in-place
(e.g., via del or slice assignment), and walk will only recurse into the
subdirectories whose names remain in dirnames; this can be used to prune the
search, or to impose a specific order of visiting. Modifying dirnames when
topdown is false has no effect on the behavior of os.walk(), since the
directories in dirnames have already been generated by the time dirnames
itself is generated. No matter the value of topdown, the list of
subdirectories is retrieved before the tuples for the directory and its
subdirectories are generated.
By default errors from the os.scandir() call are ignored. If
optional arg 'onerror' is specified, it should be a function; it
will be called with one argument, an OSError instance. It can
report the error to continue with the walk, or raise the exception
to abort the walk. Note that the filename is available as the
filename attribute of the exception object.
By default, os.walk does not follow symbolic links to subdirectories on
systems that support them. In order to get this functionality, set the
optional argument 'followlinks' to true.
Caution: if you pass a relative pathname for top, don't change the
current working directory between resumptions of walk. walk never
changes the current directory, and assumes that the client doesn't
either.
Example:
import os
from os.path import join, getsize
for root, dirs, files in os.walk('python/Lib/email'):
print(root, "consumes ")
print(sum(getsize(join(root, name)) for name in files), end=" ")
print("bytes in", len(files), "non-directory files")
if 'CVS' in dirs:
dirs.remove('CVS') # don't visit CVS directories
write(fd, data, /)
Write a bytes object to a file descriptor.
DATA
EX_OK = 0
F_OK = 0
O_APPEND = 8
O_BINARY = 32768
O_CREAT = 256
O_EXCL = 1024
O_NOINHERIT = 128
O_RANDOM = 16
O_RDONLY = 0
O_RDWR = 2
O_SEQUENTIAL = 32
O_SHORT_LIVED = 4096
O_TEMPORARY = 64
O_TEXT = 16384
O_TRUNC = 512
O_WRONLY = 1
P_DETACH = 4
P_NOWAIT = 1
P_NOWAITO = 3
P_OVERLAY = 2
P_WAIT = 0
R_OK = 4
SEEK_CUR = 1
SEEK_END = 2
SEEK_SET = 0
TMP_MAX = 2147483647
W_OK = 2
X_OK = 1
__all__ = ['altsep', 'curdir', 'pardir', 'sep', 'pathsep', 'linesep', ...
altsep = '/'
curdir = '.'
defpath = r'.;C:\bin'
devnull = 'nul'
environ = environ({'ALLUSERSPROFILE': 'C:\\ProgramData', '...CONF': 'd...
extsep = '.'
linesep = '\r\n'
name = 'nt'
pardir = '..'
pathsep = ';'
sep = r'\'
supports_bytes_environ = False
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\os.py
polars.override
Python Library Documentation: NoneType object class NoneType(object) | Methods defined here: | | __bool__(self, /) | True if self else False | | __eq__(self, value, /) | Return self==value. | | __ge__(self, value, /) | Return self>=value. | | __gt__(self, value, /) | Return self>value. | | __hash__(self, /) | Return hash(self). | | __le__(self, value, /) | Return self<=value. | | __lt__(self, value, /) | Return self<value. | | __ne__(self, value, /) | Return self!=value. | | __repr__(self, /) | Return repr(self). | | ---------------------------------------------------------------------- | Static methods defined here: | | __new__(*args, **kwargs) | Create and return a new object. See help(type) for accurate signature.
polars.plr
Python Library Documentation: module _polars_runtime_32._polars_runtime in _polars_runtime_32
NAME
_polars_runtime_32._polars_runtime
CLASSES
builtins.BaseException(builtins.object)
pyo3_runtime.PanicException
builtins.Exception(builtins.BaseException)
polars.exceptions.PolarsError
polars.exceptions.ColumnNotFoundError
polars.exceptions.ComputeError
polars.exceptions.DuplicateError
polars.exceptions.InvalidOperationError
polars.exceptions.NoDataError
polars.exceptions.OutOfBoundsError
polars.exceptions.SQLInterfaceError
polars.exceptions.SQLSyntaxError
polars.exceptions.SchemaError
polars.exceptions.SchemaFieldNotFoundError
polars.exceptions.ShapeError
polars.exceptions.StringCacheMismatchError
polars.exceptions.StructFieldNotFoundError
builtins.Warning(builtins.Exception)
polars.exceptions.PolarsWarning
polars.exceptions.MapWithoutReturnDtypeWarning
polars.exceptions.PerformanceWarning
polars.exceptions.CategoricalRemappingWarning
builtins.object
builtins.PyArrowCStreamReader
builtins.PyCatalogClient
builtins.PyCategories
builtins.PyDataFrame
builtins.PyDataTypeExpr
builtins.PyExpr
builtins.PyInProcessQuery
builtins.PyLazyFrame
builtins.PyLazyGroupBy
builtins.PyOptFlags
builtins.PySQLContext
builtins.PySelector
builtins.PySeries
class CategoricalRemappingWarning(PerformanceWarning)
| Method resolution order:
| CategoricalRemappingWarning
| PerformanceWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ColumnNotFoundError(PolarsError)
| Method resolution order:
| ColumnNotFoundError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ComputeError(PolarsError)
| Method resolution order:
| ComputeError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DuplicateError(PolarsError)
| Method resolution order:
| DuplicateError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class InvalidOperationError(PolarsError)
| Method resolution order:
| InvalidOperationError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class MapWithoutReturnDtypeWarning(PolarsWarning)
| Method resolution order:
| MapWithoutReturnDtypeWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NoDataError(PolarsError)
| Method resolution order:
| NoDataError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class OutOfBoundsError(PolarsError)
| Method resolution order:
| OutOfBoundsError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PanicException(builtins.BaseException)
| The exception raised when Rust code called from Python panics.
|
| Like SystemExit, this exception is derived from BaseException so that
| it will typically propagate all the way through the stack and cause the
| Python interpreter to exit.
|
| Method resolution order:
| PanicException
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.BaseException:
|
| __new__(*args, **kwargs) class method of builtins.BaseException
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PerformanceWarning(PolarsWarning)
| Method resolution order:
| PerformanceWarning
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsWarning:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PolarsError(builtins.Exception)
| Method resolution order:
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PolarsWarning(builtins.Warning)
| Method resolution order:
| PolarsWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PyArrowCStreamReader(object)
| PyArrowCStreamReader(ob)
|
| Methods defined here:
|
| next_batch(self, /, with_columns)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| schema
class PyCatalogClient(object)
| Methods defined here:
|
| create_catalog(self, /, catalog_name, comment, storage_root)
|
| create_namespace(self, /, catalog_name, namespace, comment, storage_root)
|
| create_table(self, /, catalog_name, namespace, table_name, schema, table_type, data_source_format, comment, storage_root, properties)
|
| delete_catalog(self, /, catalog_name, force)
|
| delete_namespace(self, /, catalog_name, namespace, force)
|
| delete_table(self, /, catalog_name, namespace, table_name)
|
| get_table_credentials(self, /, table_id, write)
|
| get_table_info(self, /, table_name, catalog_name, namespace)
|
| list_catalogs(self, /)
|
| list_namespaces(self, /, catalog_name)
|
| list_tables(self, /, catalog_name, namespace)
|
| scan_table(self, /, catalog_name, namespace, table_name, cloud_options, credential_provider)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| init_classes(catalog_info_cls, namespace_info_cls, table_info_cls, column_info_cls)
|
| new(workspace_url, bearer_token)
|
| type_json_to_polars_type(type_json)
class PyCategories(object)
| PyCategories(name, namespace, physical)
|
| Methods defined here:
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| cat_to_str(self, /, cat)
|
| get_cat(self, /, s)
|
| is_global(self, /)
|
| name(self, /)
|
| namespace(self, /)
|
| num_cats_upper_bound(self, /)
|
| physical(self, /)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| global_categories()
|
| random(namespace, physical)
class PyDataFrame(object)
| PyDataFrame(columns)
|
| Methods defined here:
|
| __arrow_c_stream__(self, /, requested_schema)
|
| add(self, /, s)
|
| add_df(self, /, s)
|
| as_str(self, /)
| Format `DataFrame` as String
|
| clear(self, /)
|
| clone(self, /)
|
| columns(self, /)
| Get column names
|
| div(self, /, s)
|
| div_df(self, /, s)
|
| drop_in_place(self, /, name)
|
| dtype_strings(self, /)
|
| dtypes(self, /)
| Get datatypes
|
| equals(self, /, other, null_equal)
|
| estimated_size(self, /)
|
| extend(self, /, other)
|
| gather(self, /, indices)
|
| gather_with_series(self, /, indices)
|
| get_column(self, /, name)
|
| get_column_index(self, /, name)
|
| get_columns(self, /)
|
| group_by_map_groups(...)
|
| hash_rows(self, /, k0, k1, k2, k3)
|
| head(self, /, n)
|
| height(self, /)
|
| hstack(self, /, columns)
|
| hstack_mut(self, /, columns)
|
| insert_column(self, /, index, column)
|
| is_duplicated(self, /)
|
| is_empty(self, /)
|
| is_sorted(self, /, by, descending, nulls_last)
|
| is_unique(self, /)
|
| lazy(self, /)
|
| map_rows(...)
|
| mul(self, /, s)
|
| mul_df(self, /, s)
|
| n_chunks(self, /)
|
| null_count(self, /)
|
| partition_by(self, /, by, maintain_order, include_key)
|
| rechunk(self, /)
|
| rem(self, /, s)
|
| rem_df(self, /, s)
|
| replace(self, /, column, new_col)
|
| replace_column(self, /, index, new_column)
|
| row_tuple(self, /, idx)
|
| row_tuples(self, /)
|
| sample_frac(self, /, frac, with_replacement, shuffle, seed=None)
|
| sample_n(self, /, n, with_replacement, shuffle, seed=None)
|
| select(self, /, columns)
|
| serialize_binary(self, /, py_f)
| Serialize into binary data.
|
| serialize_json(self, /, py_f)
| Serialize into a JSON string.
|
| set_column_names(self, /, names)
| set column names
|
| shape(self, /)
|
| shrink_to_fit(self, /)
|
| slice(self, /, offset, length)
|
| sub(self, /, s)
|
| sub_df(self, /, s)
|
| tail(self, /, n)
|
| to_arrow(self, /, compat_level)
|
| to_dummies(self, /, columns, separator, drop_first, drop_nulls)
|
| to_numpy(self, /, order, writable, allow_copy)
| Convert this DataFrame to a NumPy ndarray.
|
| to_pandas(self, /)
| Create a `Vec` of PyArrow RecordBatch instances.
|
| Note this will give bad results for columns with dtype `pl.Object`,
| since those can't be converted correctly via PyArrow. The calling Python
| code should make sure these are not included.
|
| to_series(self, /, index)
|
| to_struct(self, /, name, invalid_indices)
|
| transpose(self, /, keep_names_as, column_names)
|
| unpivot(self, /, on, index, value_name=None, variable_name=None)
|
| upsample(self, /, by, index_column, every, stable)
|
| vstack(self, /, other)
|
| vstack_mut(self, /, other)
|
| width(self, /)
|
| with_row_index(self, /, name, offset=None)
|
| write_avro(self, /, py_f, compression, name)
|
| write_ipc_stream(self, /, py_f, compression, compat_level)
|
| write_json(self, /, py_f)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(*args, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| deserialize_binary(py_f)
| Deserialize a file-like object containing binary data into a DataFrame.
|
| deserialize_json(py_f)
| Deserialize a file-like object containing JSON string data into a DataFrame.
|
| empty_with_height(height)
|
| from_arrow_record_batches(rb, schema)
|
| from_dicts(data, schema=None, schema_overrides=None, strict=True, infer_schema_length=None)
|
| from_rows(data, schema=None, infer_schema_length=None)
|
| read_avro(py_f, columns, projection, n_rows)
|
| read_csv(py_f, infer_schema_length, chunk_size, has_header, ignore_errors, n_rows, skip_rows, skip_lines, projection, separator, rechunk, columns, encoding, n_threads, path, overwrite_dtype, overwrite_dtype_slice, low_memory, comment_prefix, quote_char, null_values, empty_string_is_null, try_parse_dates, skip_rows_after_header, row_index, eol_char, raise_if_empty, truncate_ragged_lines, decimal_comma, schema)
|
| read_ipc(py_f, columns, projection, n_rows, row_index, memory_map)
|
| read_ipc_stream(py_f, columns, projection, n_rows, row_index, rechunk)
|
| read_json(py_f, infer_schema_length, schema, schema_overrides)
class PyDataTypeExpr(object)
| Methods defined here:
|
| arr_inner_dtype(self, /)
|
| arr_shape(self, /)
|
| arr_width(self, /)
|
| collect_dtype(self, /, schema)
|
| default_value(self, /, n, numeric_to_one, num_list_values)
|
| display(self, /)
|
| equals(self, /, other)
|
| inner_dtype(self, /)
|
| list_inner_dtype(self, /)
|
| matches(self, /, selector)
|
| struct_field_dtype_by_index(self, /, index)
|
| struct_field_dtype_by_name(self, /, name)
|
| struct_field_names(self, /)
|
| to_signed_integer(self, /)
|
| to_unsigned_integer(self, /)
|
| wrap_in_array(self, /, width)
|
| wrap_in_list(self, /)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| from_dtype(datatype)
|
| of_expr(expr)
|
| self_dtype()
|
| struct_with_fields(fields)
class PyExpr(object)
| Methods defined here:
|
| __add__(self, value, /)
| Return self+value.
|
| __eq__(self, value, /)
| Return self==value.
|
| __floordiv__(self, value, /)
| Return self//value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getstate__(self, /)
| Helper for pickle.
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __lt__(self, value, /)
| Return self<value.
|
| __mod__(self, value, /)
| Return self%value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __neg__(self, /)
| -self
|
| __radd__(self, value, /)
| Return value+self.
|
| __rfloordiv__(self, value, /)
| Return value//self.
|
| __rmod__(self, value, /)
| Return value%self.
|
| __rmul__(self, value, /)
| Return value*self.
|
| __rsub__(self, value, /)
| Return value-self.
|
| __rtruediv__(self, value, /)
| Return value/self.
|
| __setstate__(self, /, state)
|
| __sub__(self, value, /)
| Return self-value.
|
| __truediv__(self, value, /)
| Return self/value.
|
| abs(self, /)
|
| agg_groups(self, /)
|
| alias(self, /, name)
|
| all(self, /, ignore_nulls)
|
| and_(self, /, expr)
|
| any(self, /, ignore_nulls)
|
| append(self, /, other, upcast)
|
| approx_n_unique(self, /)
|
| arccos(self, /)
|
| arccosh(self, /)
|
| arcsin(self, /)
|
| arcsinh(self, /)
|
| arctan(self, /)
|
| arctan2(self, /, y)
|
| arctanh(self, /)
|
| arg_max(self, /)
|
| arg_min(self, /)
|
| arg_sort(self, /, descending, nulls_last)
|
| arg_unique(self, /)
|
| arr_agg(self, /, expr)
|
| arr_arg_max(self, /)
|
| arr_arg_min(self, /)
|
| arr_contains(self, /, other, nulls_equal)
|
| arr_count_matches(self, /, expr)
|
| arr_eval(self, /, expr, as_list)
|
| arr_explode(self, /, empty_as_null, keep_nulls)
|
| arr_get(self, /, index, null_on_oob)
|
| arr_join(self, /, separator, ignore_nulls)
|
| arr_len(self, /)
|
| arr_max(self, /)
|
| arr_mean(self, /)
|
| arr_median(self, /)
|
| arr_min(self, /)
|
| arr_shift(self, /, n)
|
| arr_slice(self, /, offset, length, as_array)
|
| arr_sort(self, /, descending, nulls_last)
|
| arr_std(self, /, ddof)
|
| arr_sum(self, /)
|
| arr_tail(self, /, n, as_array)
|
| arr_to_list(self, /)
|
| arr_to_struct(self, /, name_gen)
|
| arr_var(self, /, ddof)
|
| bin_base64_decode(self, /, strict)
|
| bin_base64_encode(self, /)
|
| bin_contains(self, /, lit)
|
| bin_ends_with(self, /, sub)
|
| bin_get(self, /, index, null_on_oob)
|
| bin_head(self, /, n)
|
| bin_hex_decode(self, /, strict)
|
| bin_hex_encode(self, /)
|
| bin_reinterpret(self, /, dtype, kind)
|
| bin_size_bytes(self, /)
|
| bin_slice(self, /, offset, length)
|
| bin_starts_with(self, /, sub)
|
| bin_tail(self, /, n)
|
| bitwise_and(self, /)
|
| bitwise_count_ones(self, /)
|
| bitwise_count_zeros(self, /)
|
| bitwise_leading_ones(self, /)
|
| bitwise_leading_zeros(self, /)
|
| bitwise_or(self, /)
|
| bitwise_trailing_ones(self, /)
|
| bitwise_trailing_zeros(self, /)
|
| bitwise_xor(self, /)
|
| bottom_k(self, /, k)
|
| bottom_k_by(self, /, by, k, reverse)
|
| cast(self, /, dtype, strict, wrap_numerical)
|
| cat_ends_with(self, /, suffix)
|
| cat_get_categories(self, /)
|
| cat_len_bytes(self, /)
|
| cat_len_chars(self, /)
|
| cat_physical(self, /)
|
| cat_slice(self, /, offset, length=None)
|
| cat_starts_with(self, /, prefix)
|
| cat_to(self, /, dtype, strict)
|
| cbrt(self, /)
|
| ceil(self, /)
|
| clip(self, /, min, max)
|
| compute_tree_format(self, /, display_as_dot, schema)
|
| cos(self, /)
|
| cosh(self, /)
|
| cot(self, /)
|
| count(self, /)
|
| cum_count(self, /, reverse)
|
| cum_max(self, /, reverse)
|
| cum_min(self, /, reverse)
|
| cum_prod(self, /, reverse)
|
| cum_sum(self, /, reverse)
|
| cumulative_eval(self, /, expr, min_samples)
|
| cut(self, /, breaks, labels, left_closed, include_breaks)
|
| degrees(self, /)
|
| diff(self, /, n, null_behavior)
|
| dot(self, /, other)
|
| drop_nans(self, /)
|
| drop_nulls(self, /)
|
| dt_add_business_days(self, /, n, week_mask, holidays, roll)
|
| dt_base_utc_offset(self, /)
|
| dt_cast_time_unit(self, /, time_unit)
|
| dt_century(self, /)
|
| dt_combine(self, /, time, time_unit)
|
| dt_convert_time_zone(self, /, time_zone)
|
| dt_date(self, /)
|
| dt_datetime(self, /)
|
| dt_day(self, /)
|
| dt_days_in_month(self, /)
|
| dt_dst_offset(self, /)
|
| dt_hour(self, /)
|
| dt_is_business_day(self, /, week_mask, holidays)
|
| dt_is_leap_year(self, /)
|
| dt_iso_year(self, /)
|
| dt_microsecond(self, /)
|
| dt_millennium(self, /)
|
| dt_millisecond(self, /)
|
| dt_minute(self, /)
|
| dt_month(self, /)
|
| dt_month_end(self, /)
|
| dt_month_start(self, /)
|
| dt_nanosecond(self, /)
|
| dt_offset_by(self, /, by)
|
| dt_ordinal_day(self, /)
|
| dt_quarter(self, /)
|
| dt_replace(self, /, year, month, day, hour, minute, second, microsecond, ambiguous)
|
| dt_replace_time_zone(self, /, time_zone, ambiguous, non_existent)
|
| dt_round(self, /, every)
|
| dt_second(self, /)
|
| dt_time(self, /)
|
| dt_timestamp(self, /, time_unit)
|
| dt_to_string(self, /, format)
|
| dt_total_days(self, /, fractional)
|
| dt_total_hours(self, /, fractional)
|
| dt_total_microseconds(self, /, fractional)
|
| dt_total_milliseconds(self, /, fractional)
|
| dt_total_minutes(self, /, fractional)
|
| dt_total_nanoseconds(self, /, fractional)
|
| dt_total_seconds(self, /, fractional)
|
| dt_truncate(self, /, every)
|
| dt_week(self, /)
|
| dt_weekday(self, /)
|
| dt_with_time_unit(self, /, time_unit)
|
| dt_year(self, /)
|
| entropy(self, /, base, normalize)
|
| eq(self, /, other)
|
| eq_missing(self, /, other)
|
| ewm_mean(self, /, alpha, adjust, min_periods, ignore_nulls)
|
| ewm_mean_by(self, /, times, half_life)
|
| ewm_std(self, /, alpha, adjust, bias, min_periods, ignore_nulls)
|
| ewm_sum(self, /, alpha, min_periods, ignore_nulls)
|
| ewm_sum_by(self, /, times, half_life)
|
| ewm_var(self, /, alpha, adjust, bias, min_periods, ignore_nulls)
|
| exp(self, /)
|
| explode(self, /, empty_as_null, keep_nulls)
|
| ext_storage(self, /)
|
| ext_to(self, /, dtype)
|
| extend_constant(self, /, value, n)
|
| fill_nan(self, /, expr)
|
| fill_null(self, /, expr)
|
| fill_null_with_strategy(self, /, strategy, limit)
|
| filter(self, /, predicate)
|
| first(self, /, ignore_nulls)
|
| floor(self, /)
|
| gather(self, /, idx, null_on_oob=False)
|
| gather_every(self, /, n, offset)
|
| get(self, /, idx, null_on_oob=False)
|
| gt(self, /, other)
|
| gt_eq(self, /, other)
|
| has_nulls(self, /)
|
| hash(self, /, seed, seed_1, seed_2, seed_3)
|
| hist(self, /, bins, bin_count, include_category, include_breakpoint)
|
| implode(self, /, maintain_order)
|
| index_of(self, /, element)
|
| interpolate(self, /, method)
|
| interpolate_by(self, /, by)
|
| into_selector(self, /)
|
| is_between(self, /, lower, upper, closed)
|
| is_close(self, /, other, abs_tol, rel_tol, nans_equal)
|
| is_duplicated(self, /)
|
| is_empty(self, /, ignore_nulls)
|
| is_finite(self, /)
|
| is_first_distinct(self, /)
|
| is_in(self, /, expr, nulls_equal)
|
| is_infinite(self, /)
|
| is_last_distinct(self, /)
|
| is_nan(self, /)
|
| is_not_nan(self, /)
|
| is_not_null(self, /)
|
| is_null(self, /)
|
| is_sorted(self, /, descending, nulls_last)
|
| is_unique(self, /)
|
| item(self, /, allow_empty)
|
| kurtosis(self, /, fisher, bias)
|
| last(self, /, ignore_nulls)
|
| len(self, /)
|
| list_agg(self, /, expr)
|
| list_arg_max(self, /)
|
| list_arg_min(self, /)
|
| list_contains(self, /, other, nulls_equal)
|
| list_count_matches(self, /, expr)
|
| list_diff(self, /, n, null_behavior)
|
| list_drop_nulls(self, /)
|
| list_eval(self, /, expr, _parallel)
|
| list_filter(self, /, predicate)
|
| list_gather(self, /, index, null_on_oob)
|
| list_gather_every(self, /, n, offset)
|
| list_get(self, /, index, null_on_oob)
|
| list_join(self, /, separator, ignore_nulls)
|
| list_len(self, /)
|
| list_max(self, /)
|
| list_mean(self, /)
|
| list_median(self, /)
|
| list_min(self, /)
|
| list_sample_fraction(self, /, fraction, with_replacement, shuffle, seed=None)
|
| list_sample_n(self, /, n, with_replacement, shuffle, seed=None)
|
| list_set_operation(self, /, other, operation)
|
| list_shift(self, /, periods)
|
| list_slice(self, /, offset, length=None)
|
| list_sort(self, /, descending, nulls_last)
|
| list_std(self, /, ddof)
|
| list_sum(self, /)
|
| list_tail(self, /, n)
|
| list_to_array(self, /, width)
|
| list_to_struct(self, /, names)
|
| list_var(self, /, ddof)
|
| log(self, /, base)
|
| log1p(self, /)
|
| lower_bound(self, /)
|
| lt(self, /, other)
|
| lt_eq(self, /, other)
|
| max(self, /)
|
| max_by(self, /, by)
|
| mean(self, /)
|
| median(self, /)
|
| meta_eq(self, /, other)
|
| meta_has_multiple_outputs(self, /)
|
| meta_is_column(self, /)
|
| meta_is_column_selection(self, /, allow_aliasing)
|
| meta_is_literal(self, /, allow_aliasing)
|
| meta_is_regex_projection(self, /)
|
| meta_output_name(self, /)
|
| meta_pop(self, /, schema)
|
| meta_root_names(self, /)
|
| meta_show_graph(self, /, schema)
|
| meta_tree_format(self, /, schema)
|
| meta_undo_aliases(self, /)
|
| min(self, /)
|
| min_by(self, /, by)
|
| mode(self, /, maintain_order)
|
| n_unique(self, /)
|
| name_keep(self, /)
|
| name_map(...)
|
| name_map_fields(self, /, name_mapper)
|
| name_prefix(self, /, prefix)
|
| name_prefix_fields(self, /, prefix)
|
| name_replace(self, /, pattern, value, literal)
|
| name_suffix(self, /, suffix)
|
| name_suffix_fields(self, /, suffix)
|
| name_to_lowercase(self, /)
|
| name_to_uppercase(self, /)
|
| nan_max(self, /)
|
| nan_min(self, /)
|
| neq(self, /, other)
|
| neq_missing(self, /, other)
|
| not_(self, /)
|
| null_count(self, /)
|
| or_(self, /, expr)
|
| over(self, /, partition_by, order_by, order_by_descending, order_by_nulls_last, mapping_strategy)
|
| pct_change(self, /, n)
|
| peak_max(self, /)
|
| peak_min(self, /)
|
| pow(self, /, exponent)
|
| product(self, /)
|
| qcut(self, /, probs, labels, left_closed, allow_duplicates, include_breaks)
|
| qcut_uniform(self, /, n_bins, labels, left_closed, allow_duplicates, include_breaks)
|
| quantile(self, /, quantile, interpolation)
|
| radians(self, /)
|
| rank(self, /, method, descending, seed)
|
| rechunk(self, /)
|
| reinterpret(self, /, signed, dtype)
|
| repeat_by(self, /, by)
|
| replace(self, /, old, new)
|
| replace_strict(self, /, old, new, default, return_dtype)
|
| reshape(self, /, dims)
|
| reverse(self, /)
|
| rle(self, /)
|
| rle_id(self, /)
|
| rolling(self, /, index_column, period, offset, closed)
|
| rolling_kurtosis(self, /, window_size, fisher, bias, min_periods, center)
|
| rolling_map(...)
|
| rolling_max(self, /, window_size, weights, min_periods, center)
|
| rolling_max_by(self, /, by, window_size, min_periods, closed)
|
| rolling_mean(self, /, window_size, weights, min_periods, center)
|
| rolling_mean_by(self, /, by, window_size, min_periods, closed)
|
| rolling_median(self, /, window_size, weights, min_periods, center)
|
| rolling_median_by(self, /, by, window_size, min_periods, closed)
|
| rolling_min(self, /, window_size, weights, min_periods, center)
|
| rolling_min_by(self, /, by, window_size, min_periods, closed)
|
| rolling_quantile(self, /, quantile, interpolation, window_size, weights, min_periods, center)
|
| rolling_quantile_by(self, /, by, quantile, interpolation, window_size, min_periods, closed)
|
| rolling_rank(self, /, window_size, method, seed, min_samples, center)
|
| rolling_rank_by(self, /, by, window_size, method, seed, min_samples, closed)
|
| rolling_skew(self, /, window_size, bias, min_periods, center)
|
| rolling_std(self, /, window_size, weights, min_periods, center, ddof)
|
| rolling_std_by(self, /, by, window_size, min_periods, closed, ddof)
|
| rolling_sum(self, /, window_size, weights, min_periods, center)
|
| rolling_sum_by(self, /, by, window_size, min_periods, closed)
|
| rolling_var(self, /, window_size, weights, min_periods, center, ddof)
|
| rolling_var_by(self, /, by, window_size, min_periods, closed, ddof)
|
| round(self, /, decimals, mode)
|
| round_sig_figs(self, /, digits)
|
| row_decode_ordered(self, /, names, datatypes, descending, nulls_last)
|
| row_decode_unordered(self, /, names, datatypes)
|
| sample_frac(self, /, frac, with_replacement, shuffle, seed)
|
| sample_n(self, /, n, with_replacement, shuffle, seed)
|
| search_sorted(self, /, element, side, descending)
|
| serialize_binary(self, /, py_f)
| Serialize into binary data.
|
| serialize_json(self, /, py_f)
| Serialize into a JSON string.
|
| set_sorted_flag(self, /, descending, nulls_last)
|
| shift(self, /, n, fill_value)
|
| shuffle(self, /, seed)
|
| sign(self, /)
|
| sin(self, /)
|
| sinh(self, /)
|
| skew(self, /, bias)
|
| skip_batch_predicate(self, /, schema)
|
| slice(self, /, offset, length)
|
| sort_by(self, /, by, descending, nulls_last, multithreaded, maintain_order)
|
| sort_with(self, /, descending, nulls_last)
|
| sqrt(self, /)
|
| std(self, /, ddof)
|
| str_base64_decode(self, /, strict)
|
| str_base64_encode(self, /)
|
| str_contains(self, /, pat, literal, strict)
|
| str_contains_any(self, /, patterns, ascii_case_insensitive)
|
| str_count_matches(self, /, pat, literal)
|
| str_ends_with(self, /, sub)
|
| str_escape_regex(self, /)
|
| str_extract(self, /, pat, group_index)
|
| str_extract_all(self, /, pat)
|
| str_extract_groups(self, /, pat)
|
| str_extract_many(self, /, patterns, ascii_case_insensitive, overlapping, leftmost)
|
| str_find(self, /, pat, literal, strict)
|
| str_find_many(self, /, patterns, ascii_case_insensitive, overlapping, leftmost)
|
| str_head(self, /, n)
|
| str_hex_decode(self, /, strict)
|
| str_hex_encode(self, /)
|
| str_join(self, /, delimiter, ignore_nulls)
|
| str_json_decode(self, /, dtype)
|
| str_json_path_match(self, /, pat)
|
| str_len_bytes(self, /)
|
| str_len_chars(self, /)
|
| str_normalize(self, /, form)
|
| str_pad_end(self, /, length, fill_char)
|
| str_pad_start(self, /, length, fill_char)
|
| str_replace_all(self, /, pat, val, literal)
|
| str_replace_many(self, /, patterns, replace_with, ascii_case_insensitive, leftmost)
|
| str_replace_n(self, /, pat, val, literal, n)
|
| str_reverse(self, /)
|
| str_slice(self, /, offset, length)
|
| str_split(self, /, by)
|
| str_split_exact(self, /, by, n)
|
| str_split_exact_inclusive(self, /, by, n)
|
| str_split_inclusive(self, /, by)
|
| str_split_regex(self, /, by, strict)
|
| str_split_regex_inclusive(self, /, by, strict)
|
| str_split_regex_inclusive_with_strict(self, /, by, strict)
|
| str_split_regex_with_strict(self, /, by, strict)
|
| str_splitn(self, /, by, n)
|
| str_starts_with(self, /, sub)
|
| str_strip_chars(self, /, matches)
|
| str_strip_chars_end(self, /, matches)
|
| str_strip_chars_start(self, /, matches)
|
| str_strip_prefix(self, /, prefix)
|
| str_strip_suffix(self, /, suffix)
|
| str_tail(self, /, n)
|
| str_to_date(self, /, format, strict, exact, cache)
|
| str_to_datetime(self, /, format, time_unit, time_zone, strict, exact, cache, ambiguous)
|
| str_to_decimal(self, /, scale)
|
| str_to_integer(self, /, base, dtype=Ellipsis, strict=True)
|
| str_to_lowercase(self, /)
|
| str_to_time(self, /, format, strict, cache)
|
| str_to_titlecase(self, /)
|
| str_to_uppercase(self, /)
|
| str_zfill(self, /, length)
|
| struct_field_by_index(self, /, index)
|
| struct_field_by_name(self, /, name)
|
| struct_json_encode(self, /)
|
| struct_multiple_fields(self, /, names)
|
| struct_rename_fields(self, /, names)
|
| struct_with_fields(self, /, fields)
|
| sum(self, /)
|
| tan(self, /)
|
| tanh(self, /)
|
| to_physical(self, /)
|
| to_str(self, /)
|
| top_k(self, /, k)
|
| top_k_by(self, /, by, k, reverse)
|
| truncate(self, /, decimals)
|
| unique(self, /)
|
| unique_counts(self, /)
|
| unique_stable(self, /)
|
| upper_bound(self, /)
|
| value_counts(self, /, sort, parallel, name, normalize)
|
| var(self, /, ddof)
|
| xor_(self, /, expr)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| deserialize_binary(py_f)
| Deserialize a file-like object containing binary data into an Expr.
|
| deserialize_json(py_f)
| Deserialize a file-like object containing JSON string data into an Expr.
|
| new_selector(selector)
|
| row_encode_ordered(exprs, descending, nulls_last)
|
| row_encode_unordered(exprs)
|
| str_format(f_string, exprs)
class PyInProcessQuery(object)
| Methods defined here:
|
| cancel(self, /)
|
| fetch(self, /)
|
| fetch_blocking(self, /)
class PyLazyFrame(object)
| Methods defined here:
|
| bottom_k(self, /, k, by, reverse)
|
| cache(self, /)
|
| cast(self, /, dtypes, strict)
|
| cast_all(self, /, dtype, strict)
|
| clone(self, /)
|
| collect(self, /, engine, lambda_post_opt)
|
| collect_batches(self, /, engine, maintain_order, chunk_size, lazy)
|
| collect_concurrently(self, /)
|
| collect_schema(self, /)
|
| collect_with_callback(...)
|
| count(self, /)
|
| describe_optimized_plan(self, /)
|
| describe_optimized_plan_tree(self, /)
|
| describe_plan(self, /)
|
| describe_plan_tree(self, /)
|
| drop(self, /, columns)
|
| drop_nans(self, /, subset)
|
| drop_nulls(self, /, subset)
|
| explode(self, /, subset, empty_as_null, keep_nulls)
|
| fill_nan(self, /, fill_value)
|
| filter(self, /, predicate)
|
| gather(self, /, idxs, null_on_oob)
|
| group_by(self, /, by, maintain_order)
|
| group_by_dynamic(self, /, index_column, every, period, offset, label, include_boundaries, closed, group_by, start_by)
|
| hint_sorted(self, /, columns, descending, nulls_last)
|
| join(self, /, other, left_on, right_on, allow_parallel, force_parallel, nulls_equal, how, suffix, validate, maintain_order, build_side, coalesce=None)
|
| join_asof(self, /, other, left_on, right_on, left_by, right_by, allow_parallel, force_parallel, suffix, strategy, tolerance, tolerance_str, coalesce, allow_eq, check_sortedness)
|
| join_where(self, /, other, predicates, suffix)
|
| map_batches(self, /, function, predicate_pushdown, projection_pushdown, slice_pushdown, streamable, schema, validate_output)
|
| match_to_schema(self, /, schema, missing_columns, missing_struct_fields, extra_columns, extra_struct_fields, integer_cast, float_cast)
|
| max(self, /)
|
| mean(self, /)
|
| median(self, /)
|
| merge_sorted(self, /, other, key, maintain_order)
|
| min(self, /)
|
| null_count(self, /)
|
| pipe_with_schema(self, /, callback)
|
| pivot(self, /, on, on_columns, index, values, agg, maintain_order, separator, column_naming)
|
| profile(self, /, lambda_post_opt)
|
| quantile(self, /, quantile, interpolation)
|
| remove(self, /, predicate)
|
| rename(self, /, existing, new, strict)
|
| reverse(self, /)
|
| rolling(self, /, index_column, period, offset, closed, by)
|
| select(self, /, exprs)
|
| select_seq(self, /, exprs)
|
| serialize_binary(self, /, py_f)
| Serialize into binary data.
|
| serialize_json(self, /, py_f)
| Serialize into a JSON string.
|
| shift(self, /, n, fill_value=None)
|
| sink_batches(self, /, function, maintain_order, chunk_size)
|
| sink_csv(self, /, target, sink_options, include_bom, compression, compression_level, check_extension, include_header, separator, line_terminator, quote_char, batch_size, datetime_format, date_format, time_format, float_scientific, float_precision, decimal_comma, null_value, quote_style)
|
| sink_iceberg(self, /, sink_state_obj)
|
| sink_ipc(self, /, target, sink_options, compression, compat_level, record_batch_size, record_batch_statistics)
|
| sink_ndjson(self, /, target, compression, compression_level, check_extension, sink_options)
|
| sink_parquet(self, /, target, sink_options, compression, compression_level, statistics, row_group_size, data_page_size, metadata, arrow_schema)
|
| slice(self, /, offset, len=None)
|
| sort(self, /, by_column, descending, nulls_last, maintain_order, multithreaded)
|
| sort_by_exprs(self, /, by, descending, nulls_last, maintain_order, multithreaded)
|
| std(self, /, ddof)
|
| sum(self, /)
|
| tail(self, /, n)
|
| to_dot(self, /, optimized)
|
| to_dot_streaming_phys(self, /, optimized)
|
| top_k(self, /, k, by, reverse)
|
| unique(self, /, maintain_order, subset, keep)
|
| unnest(self, /, columns, separator)
|
| unpivot(self, /, on, index, value_name, variable_name)
|
| var(self, /, ddof)
|
| visit(self, /)
|
| with_columns(self, /, exprs)
|
| with_columns_seq(self, /, exprs)
|
| with_context(self, /, contexts)
|
| with_optimizations(self, /, optflags)
|
| with_row_index(self, /, name, offset=None)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| deserialize_binary(py_f)
| Deserialize a file-like object containing binary data into a LazyFrame.
|
| deserialize_json(py_f)
| Deserialize a file-like object containing JSON string data into a LazyFrame.
|
| new_from_csv(source, sources, separator, has_header, ignore_errors, skip_rows, skip_lines, n_rows, cache, overwrite_dtype, overwrite_dtype_slice, low_memory, comment_prefix, quote_char, null_values, empty_string_is_null, infer_schema_length, new_columns, with_schema_modify, rechunk, skip_rows_after_header, encoding, row_index, try_parse_dates, eol_char, raise_if_empty, truncate_ragged_lines, decimal_comma, glob, schema, cloud_options, credential_provider, include_file_paths, missing_columns)
|
| new_from_dataset_object(dataset_object)
|
| new_from_expand_paths(sources, scan_options, name)
|
| new_from_ipc(sources, record_batch_statistics, scan_options)
|
| new_from_ndjson(source, sources, infer_schema_length, schema, schema_overrides, batch_size, n_rows, low_memory, rechunk, row_index, ignore_errors, include_file_paths, cloud_options, credential_provider)
|
| new_from_parquet(sources, schema, scan_options, parallel, low_memory, use_statistics)
|
| new_from_scan_lines(sources, scan_options, name)
|
| scan_from_python_function_arrow_schema(schema, scan_fn, pyarrow, validate_schema, is_pure)
|
| scan_from_python_function_pl_schema(schema, scan_fn, pyarrow, validate_schema, is_pure)
|
| scan_from_python_function_schema_function(schema_fn, scan_fn, validate_schema, is_pure)
class PyLazyGroupBy(object)
| Methods defined here:
|
| agg(self, /, aggs)
|
| having(self, /, predicates)
|
| head(self, /, n)
|
| map_groups(...)
|
| tail(self, /, n)
class PyOptFlags(object)
| Methods defined here:
|
| copy(self, /)
|
| no_optimizations(self, /)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| default()
|
| empty()
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| check_order_observe
|
| cluster_with_columns
|
| comm_subexpr_elim
|
| comm_subplan_elim
|
| eager
|
| fast_projection
|
| gpu
|
| pre_partition_hive
|
| predicate_pushdown
|
| projection_pushdown
|
| simplify_expression
|
| slice_pushdown
|
| sort_collapse
|
| streaming
|
| type_check
|
| type_coercion
class PySQLContext(object)
| Methods defined here:
|
| execute(self, /, query)
| Execute a SQL query in the current SQLContext.
|
| get_tables(self, /)
| Get a list of table names registered in the current SQLContext.
|
| register(self, /, name, lf)
| Register a table in the current SQLContext.
|
| unregister(self, /, name)
| Unregister a table from the current SQLContext.
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| new()
|
| table_identifiers(query, include_schema=True, unique=False)
| Extract table identifiers from a SQL query string.
class PySelector(object)
| Methods defined here:
|
| difference(self, /, other)
|
| exclusive_or(self, /, other)
|
| hash(self, /)
|
| intersect(self, /, other)
|
| union(self, /, other)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| all()
|
| array(inner_dst, width)
|
| by_dtype(dtypes)
|
| by_index(indices, strict)
|
| by_name(names, strict, expand_patterns)
|
| categorical()
|
| datetime(tu, tz)
|
| decimal()
|
| duration(tu)
|
| empty()
|
| enum_()
|
| first(strict)
|
| float()
|
| integer()
|
| last(strict)
|
| list(inner_dst)
|
| matches(pattern)
|
| nested()
|
| numeric()
|
| object()
|
| signed_integer()
|
| struct_()
|
| temporal()
|
| unsigned_integer()
class PySeries(object)
| Methods defined here:
|
| __arrow_c_stream__(self, /, requested_schema=None)
|
| __getstate__(self, /)
| Helper for pickle.
|
| __setstate__(self, /, state)
|
| add(self, /, other)
|
| add_datetime(self, /, other)
|
| add_duration(self, /, other)
|
| add_f16(self, /, other)
|
| add_f16_rhs(self, /, other)
|
| add_f32(self, /, other)
|
| add_f32_rhs(self, /, other)
|
| add_f64(self, /, other)
|
| add_f64_rhs(self, /, other)
|
| add_i16(self, /, other)
|
| add_i16_rhs(self, /, other)
|
| add_i32(self, /, other)
|
| add_i32_rhs(self, /, other)
|
| add_i64(self, /, other)
|
| add_i64_rhs(self, /, other)
|
| add_i8(self, /, other)
|
| add_i8_rhs(self, /, other)
|
| add_u16(self, /, other)
|
| add_u16_rhs(self, /, other)
|
| add_u32(self, /, other)
|
| add_u32_rhs(self, /, other)
|
| add_u64(self, /, other)
|
| add_u64_rhs(self, /, other)
|
| add_u8(self, /, other)
|
| add_u8_rhs(self, /, other)
|
| all(self, /, ignore_nulls)
|
| any(self, /, ignore_nulls)
|
| append(self, /, other)
|
| apply_ufunc_f32(...)
|
| apply_ufunc_f64(...)
|
| apply_ufunc_i16(...)
|
| apply_ufunc_i32(...)
|
| apply_ufunc_i64(...)
|
| apply_ufunc_i8(...)
|
| apply_ufunc_u16(...)
|
| apply_ufunc_u32(...)
|
| apply_ufunc_u64(...)
|
| apply_ufunc_u8(...)
|
| approx_n_unique(self, /)
|
| arg_max(self, /)
|
| arg_min(self, /)
|
| as_single_ptr(self, /)
| Rechunk and return a pointer to the start of the Series.
| Only implemented for numeric types
|
| as_str(self, /)
|
| bitand(self, /, other)
|
| bitor(self, /, other)
|
| bitwise_and(self, /)
|
| bitwise_or(self, /)
|
| bitwise_xor(self, /)
|
| bitxor(self, /, other)
|
| can_fast_explode_flag(self, /)
|
| cast(self, /, dtype, strict, wrap_numerical)
|
| cat_is_local(self, /)
|
| cat_to_local(self, /)
|
| cat_uses_lexical_ordering(self, /)
|
| chunk_lengths(self, /)
|
| clear(self, /)
|
| clone(self, /)
|
| div(self, /, other)
|
| div_f16(self, /, other)
|
| div_f16_rhs(self, /, other)
|
| div_f32(self, /, other)
|
| div_f32_rhs(self, /, other)
|
| div_f64(self, /, other)
|
| div_f64_rhs(self, /, other)
|
| div_i16(self, /, other)
|
| div_i16_rhs(self, /, other)
|
| div_i32(self, /, other)
|
| div_i32_rhs(self, /, other)
|
| div_i64(self, /, other)
|
| div_i64_rhs(self, /, other)
|
| div_i8(self, /, other)
|
| div_i8_rhs(self, /, other)
|
| div_u16(self, /, other)
|
| div_u16_rhs(self, /, other)
|
| div_u32(self, /, other)
|
| div_u32_rhs(self, /, other)
|
| div_u64(self, /, other)
|
| div_u64_rhs(self, /, other)
|
| div_u8(self, /, other)
|
| div_u8_rhs(self, /, other)
|
| dot(self, /, other)
|
| dtype(self, /)
|
| eq(self, /, rhs)
|
| eq_decimal(self, /, rhs)
|
| eq_f16(self, /, rhs)
|
| eq_f32(self, /, rhs)
|
| eq_f64(self, /, rhs)
|
| eq_i128(self, /, rhs)
|
| eq_i16(self, /, rhs)
|
| eq_i32(self, /, rhs)
|
| eq_i64(self, /, rhs)
|
| eq_i8(self, /, rhs)
|
| eq_str(self, /, rhs)
|
| eq_u128(self, /, rhs)
|
| eq_u16(self, /, rhs)
|
| eq_u32(self, /, rhs)
|
| eq_u64(self, /, rhs)
|
| eq_u8(self, /, rhs)
|
| equals(self, /, other, check_dtypes, check_names, null_equal)
|
| estimated_size(self, /)
|
| ext_storage(self, /)
|
| ext_to(self, /, dtype)
|
| extend(self, /, other)
|
| filter(self, /, filter)
|
| first(self, /, ignore_nulls)
|
| floor(self, /)
|
| gather_with_series(self, /, indices)
|
| get_chunks(self, /)
|
| get_date(self, /, index)
|
| get_datetime(self, /, index)
|
| get_duration(self, /, index)
|
| get_f32(self, /, index)
|
| get_f64(self, /, index)
|
| get_fmt(self, /, index, str_len_limit)
| Returns the string format of a single element of the Series.
|
| get_i16(self, /, index)
|
| get_i32(self, /, index)
|
| get_i64(self, /, index)
|
| get_i8(self, /, index)
|
| get_index(self, /, index)
| Get a value by index.
|
| get_index_signed(self, /, index)
| Get a value by index, allowing negative indices.
|
| get_list(self, /, index)
|
| get_object(self, /, index)
|
| get_str(self, /, index)
|
| get_u16(self, /, index)
|
| get_u32(self, /, index)
|
| get_u64(self, /, index)
|
| get_u8(self, /, index)
|
| gt(self, /, rhs)
|
| gt_decimal(self, /, rhs)
|
| gt_eq(self, /, rhs)
|
| gt_eq_decimal(self, /, rhs)
|
| gt_eq_f16(self, /, rhs)
|
| gt_eq_f32(self, /, rhs)
|
| gt_eq_f64(self, /, rhs)
|
| gt_eq_i128(self, /, rhs)
|
| gt_eq_i16(self, /, rhs)
|
| gt_eq_i32(self, /, rhs)
|
| gt_eq_i64(self, /, rhs)
|
| gt_eq_i8(self, /, rhs)
|
| gt_eq_str(self, /, rhs)
|
| gt_eq_u128(self, /, rhs)
|
| gt_eq_u16(self, /, rhs)
|
| gt_eq_u32(self, /, rhs)
|
| gt_eq_u64(self, /, rhs)
|
| gt_eq_u8(self, /, rhs)
|
| gt_f16(self, /, rhs)
|
| gt_f32(self, /, rhs)
|
| gt_f64(self, /, rhs)
|
| gt_i128(self, /, rhs)
|
| gt_i16(self, /, rhs)
|
| gt_i32(self, /, rhs)
|
| gt_i64(self, /, rhs)
|
| gt_i8(self, /, rhs)
|
| gt_str(self, /, rhs)
|
| gt_u128(self, /, rhs)
|
| gt_u16(self, /, rhs)
|
| gt_u32(self, /, rhs)
|
| gt_u64(self, /, rhs)
|
| gt_u8(self, /, rhs)
|
| has_nulls(self, /)
|
| head(self, /, n)
|
| is_empty(self, /, ignore_nulls)
|
| is_sorted(self, /, descending, nulls_last)
|
| is_sorted_ascending_flag(self, /)
|
| is_sorted_descending_flag(self, /)
|
| kurtosis(self, /, fisher, bias)
|
| last(self, /, ignore_nulls)
|
| len(self, /)
|
| list_to_struct(self, /, width_strat, name_gen)
|
| lt(self, /, rhs)
|
| lt_decimal(self, /, rhs)
|
| lt_eq(self, /, rhs)
|
| lt_eq_decimal(self, /, rhs)
|
| lt_eq_f16(self, /, rhs)
|
| lt_eq_f32(self, /, rhs)
|
| lt_eq_f64(self, /, rhs)
|
| lt_eq_i128(self, /, rhs)
|
| lt_eq_i16(self, /, rhs)
|
| lt_eq_i32(self, /, rhs)
|
| lt_eq_i64(self, /, rhs)
|
| lt_eq_i8(self, /, rhs)
|
| lt_eq_str(self, /, rhs)
|
| lt_eq_u128(self, /, rhs)
|
| lt_eq_u16(self, /, rhs)
|
| lt_eq_u32(self, /, rhs)
|
| lt_eq_u64(self, /, rhs)
|
| lt_eq_u8(self, /, rhs)
|
| lt_f16(self, /, rhs)
|
| lt_f32(self, /, rhs)
|
| lt_f64(self, /, rhs)
|
| lt_i128(self, /, rhs)
|
| lt_i16(self, /, rhs)
|
| lt_i32(self, /, rhs)
|
| lt_i64(self, /, rhs)
|
| lt_i8(self, /, rhs)
|
| lt_str(self, /, rhs)
|
| lt_u128(self, /, rhs)
|
| lt_u16(self, /, rhs)
|
| lt_u32(self, /, rhs)
|
| lt_u64(self, /, rhs)
|
| lt_u8(self, /, rhs)
|
| map_elements(self, /, function, return_dtype, skip_nulls)
|
| max(self, /)
|
| mean(self, /)
|
| median(self, /)
|
| min(self, /)
|
| mul(self, /, other)
|
| mul_f16(self, /, other)
|
| mul_f16_rhs(self, /, other)
|
| mul_f32(self, /, other)
|
| mul_f32_rhs(self, /, other)
|
| mul_f64(self, /, other)
|
| mul_f64_rhs(self, /, other)
|
| mul_i16(self, /, other)
|
| mul_i16_rhs(self, /, other)
|
| mul_i32(self, /, other)
|
| mul_i32_rhs(self, /, other)
|
| mul_i64(self, /, other)
|
| mul_i64_rhs(self, /, other)
|
| mul_i8(self, /, other)
|
| mul_i8_rhs(self, /, other)
|
| mul_u16(self, /, other)
|
| mul_u16_rhs(self, /, other)
|
| mul_u32(self, /, other)
|
| mul_u32_rhs(self, /, other)
|
| mul_u64(self, /, other)
|
| mul_u64_rhs(self, /, other)
|
| mul_u8(self, /, other)
|
| mul_u8_rhs(self, /, other)
|
| n_chunks(self, /)
|
| n_unique(self, /)
|
| name(self, /)
|
| neq(self, /, rhs)
|
| neq_decimal(self, /, rhs)
|
| neq_f16(self, /, rhs)
|
| neq_f32(self, /, rhs)
|
| neq_f64(self, /, rhs)
|
| neq_i128(self, /, rhs)
|
| neq_i16(self, /, rhs)
|
| neq_i32(self, /, rhs)
|
| neq_i64(self, /, rhs)
|
| neq_i8(self, /, rhs)
|
| neq_str(self, /, rhs)
|
| neq_u128(self, /, rhs)
|
| neq_u16(self, /, rhs)
|
| neq_u32(self, /, rhs)
|
| neq_u64(self, /, rhs)
|
| neq_u8(self, /, rhs)
|
| new_from_index(self, /, index, length)
|
| not_(self, /)
|
| null_count(self, /)
|
| product(self, /)
|
| quantile(self, /, quantile, interpolation)
|
| rechunk(self, /, in_place)
|
| rem(self, /, other)
|
| rem_f16(self, /, other)
|
| rem_f16_rhs(self, /, other)
|
| rem_f32(self, /, other)
|
| rem_f32_rhs(self, /, other)
|
| rem_f64(self, /, other)
|
| rem_f64_rhs(self, /, other)
|
| rem_i16(self, /, other)
|
| rem_i16_rhs(self, /, other)
|
| rem_i32(self, /, other)
|
| rem_i32_rhs(self, /, other)
|
| rem_i64(self, /, other)
|
| rem_i64_rhs(self, /, other)
|
| rem_i8(self, /, other)
|
| rem_i8_rhs(self, /, other)
|
| rem_u16(self, /, other)
|
| rem_u16_rhs(self, /, other)
|
| rem_u32(self, /, other)
|
| rem_u32_rhs(self, /, other)
|
| rem_u64(self, /, other)
|
| rem_u64_rhs(self, /, other)
|
| rem_u8(self, /, other)
|
| rem_u8_rhs(self, /, other)
|
| rename(self, /, name)
|
| reshape(self, /, dims)
|
| scatter(self, /, idx, values)
|
| set(self, /, mask, value)
|
| set_sorted_flag(self, /, descending)
|
| shrink_dtype(self, /)
|
| shrink_to_fit(self, /)
|
| skew(self, /, bias)
|
| slice(self, /, offset, length)
|
| sort(self, /, descending, nulls_last, multithreaded)
|
| std(self, /, ddof)
|
| str_json_decode(self, /, infer_schema_length)
|
| str_to_datetime_infer(self, /, time_unit, strict, exact, ambiguous)
|
| str_to_decimal_infer(self, /, inference_length)
|
| struct_fields(self, /)
|
| struct_unnest(self, /)
|
| sub(self, /, other)
|
| sub_datetime(self, /, other)
|
| sub_duration(self, /, other)
|
| sub_f16(self, /, other)
|
| sub_f16_rhs(self, /, other)
|
| sub_f32(self, /, other)
|
| sub_f32_rhs(self, /, other)
|
| sub_f64(self, /, other)
|
| sub_f64_rhs(self, /, other)
|
| sub_i16(self, /, other)
|
| sub_i16_rhs(self, /, other)
|
| sub_i32(self, /, other)
|
| sub_i32_rhs(self, /, other)
|
| sub_i64(self, /, other)
|
| sub_i64_rhs(self, /, other)
|
| sub_i8(self, /, other)
|
| sub_i8_rhs(self, /, other)
|
| sub_u16(self, /, other)
|
| sub_u16_rhs(self, /, other)
|
| sub_u32(self, /, other)
|
| sub_u32_rhs(self, /, other)
|
| sub_u64(self, /, other)
|
| sub_u64_rhs(self, /, other)
|
| sub_u8(self, /, other)
|
| sub_u8_rhs(self, /, other)
|
| sum(self, /)
|
| tail(self, /, n)
|
| to_arrow(self, /, compat_level)
| Return the underlying Arrow array.
|
| to_dummies(self, /, separator, drop_first, drop_nulls)
|
| to_list(self, /)
| Convert this Series to a Python list.
| This operation copies data.
|
| to_numpy(self, /, writable, allow_copy)
| Convert this Series to a NumPy ndarray.
|
| This method copies data only when necessary. Set `allow_copy` to raise an error if copy
| is required. Set `writable` to make sure the resulting array is writable, possibly requiring
| copying the data.
|
| to_numpy_view(self, /)
| Create a view of the data as a NumPy ndarray.
|
| WARNING: The resulting view will show the underlying value for nulls,
| which may be any value. The caller is responsible for handling nulls
| appropriately.
|
| value_counts(self, /, sort, parallel, name, normalize)
|
| var(self, /, ddof)
|
| zip_with(self, /, mask, other)
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_arrow_c_array(ob)
|
| from_arrow_c_stream(ob)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| from_arrow(name, array)
|
| new_array(name, values, strict, dtype)
|
| new_binary(name, values, _strict)
|
| new_bool(name, array, _strict)
|
| new_decimal(name, values, strict)
|
| new_f16(name, array, nan_is_null)
|
| new_f32(name, array, nan_is_null)
|
| new_f64(name, array, nan_is_null)
|
| new_from_any_values(name, values, strict)
|
| new_from_any_values_and_dtype(name, values, dtype, strict)
|
| new_i16(name, array, _strict)
|
| new_i32(name, array, _strict)
|
| new_i64(name, array, _strict)
|
| new_i8(name, array, _strict)
|
| new_null(name, values, _strict)
|
| new_object(name, values, _strict)
|
| new_opt_bool(name, values, _strict)
|
| new_opt_f16(name, values, _strict)
|
| new_opt_f32(name, obj, strict)
|
| new_opt_f64(name, obj, strict)
|
| new_opt_i128(name, obj, strict)
|
| new_opt_i16(name, obj, strict)
|
| new_opt_i32(name, obj, strict)
|
| new_opt_i64(name, obj, strict)
|
| new_opt_i8(name, obj, strict)
|
| new_opt_u128(name, obj, strict)
|
| new_opt_u16(name, obj, strict)
|
| new_opt_u32(name, obj, strict)
|
| new_opt_u64(name, obj, strict)
|
| new_opt_u8(name, obj, strict)
|
| new_series_list(name, values, _strict)
|
| new_str(name, values, _strict)
|
| new_u16(name, array, _strict)
|
| new_u32(name, array, _strict)
|
| new_u64(name, array, _strict)
|
| new_u8(name, array, _strict)
class SQLInterfaceError(PolarsError)
| Method resolution order:
| SQLInterfaceError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class SQLSyntaxError(PolarsError)
| Method resolution order:
| SQLSyntaxError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class SchemaError(PolarsError)
| Method resolution order:
| SchemaError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class SchemaFieldNotFoundError(PolarsError)
| Method resolution order:
| SchemaFieldNotFoundError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ShapeError(PolarsError)
| Method resolution order:
| ShapeError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class StringCacheMismatchError(PolarsError)
| Method resolution order:
| StringCacheMismatchError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class StructFieldNotFoundError(PolarsError)
| Method resolution order:
| StructFieldNotFoundError
| PolarsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from PolarsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
FUNCTIONS
__register_startup_deps(warn_function)
_bench_parquet_metadata_bincode_size(path, projection, predicate)
Decode a parquet footer, optionally apply `FileMetadata::pruned`, then
bincode-encode and return the byte length of the wire form.
Exposed for out-of-tree measurement of the IR-plan-borne metadata wire
form (the `bincode(FileMetadata)` blob shipped to workers in distributed
execution); no caller in py-polars itself.
`projection = None` ⇒ encode the full `FileMetadata`. `projection =
Some(cols)` ⇒ apply `pruned(cols, predicate)`. Local files only.
_estimate_memory_usage()
_get_dtype_max(dt)
_get_dtype_min(dt)
_known_timezones()
_parquet_metadata_pruned_json(path, projection, predicate)
Decode a parquet footer, apply `FileMetadata::pruned(projection, predicate)`,
and return the result as a JSON string. Format-agnostic custom serde lets
the same wire DTOs emit JSON for inspection or bincode for dispatch.
Used by py-polars tests to assert structural prune behavior (only kept
columns survive, stats only on predicate columns). Local files only.
_register_extension_type(name, cls)
_unregister_extension_type(name)
all_horizontal(exprs)
any_horizontal(exprs)
arctan2(y, x)
arg_sort_by(by, descending, nulls_last, multithreaded, maintain_order)
arg_where(condition)
as_struct(exprs)
assert_dataframe_equal_py(left, right, *, check_row_order, check_column_order, check_dtypes, check_exact, rel_tol, abs_tol, categorical_as_str)
assert_schema_equal_py(left_schema, right_schema, check_dtypes, check_column_order)
assert_series_equal_py(left, right, *, check_dtypes, check_names, check_order, check_exact, rel_tol, abs_tol, categorical_as_str)
business_day_count(start, end, week_mask, holidays)
check_length(check)
coalesce(exprs)
col(name)
collect_all(lfs, engine, optflags)
collect_all_lazy(lfs, optflags)
collect_all_with_callback(...)
concat_arr(s)
concat_df(dfs)
concat_df_diagonal(dfs)
concat_df_horizontal(dfs, strict)
concat_expr(e, rechunk)
concat_lf(seq, rechunk, parallel, to_supertypes, maintain_order)
concat_lf_diagonal(lfs, rechunk, parallel, to_supertypes, maintain_order)
concat_lf_horizontal(lfs, parallel, strict)
concat_list(s)
concat_series(series)
concat_str(s, separator, ignore_nulls)
config_reload_env_var(var)
config_reload_env_vars()
cov(a, b, ddof)
cum_fold(...)
cum_reduce(...)
date_range(start, end, interval, closed)
date_ranges(start, end, interval, closed)
datetime(year, month, day, hour=None, minute=None, second=None, microsecond=None, time_unit=Ellipsis, time_zone=Ellipsis, ambiguous=Ellipsis)
datetime_range(start, end, interval, closed, time_unit, time_zone)
datetime_ranges(start, end, interval, closed, time_unit, time_zone)
dtype_str_repr(dtype)
duration(weeks, days, hours, minutes, seconds, milliseconds, microseconds, nanoseconds, time_unit)
eager_int_range(lower, upper, step, dtype)
Eager version of `int_range` to avoid overhead from the expression engine.
element()
escape_regex(s)
explain_all(lfs, optflags)
field(names)
fold(...)
gen_uuid_v7()
get_decimal_separator()
get_engine_affinity()
get_float_fmt()
get_float_precision()
get_index_type()
get_thousands_separator()
get_trim_decimal_zeros()
init_polars_schema_from_arrow_c_schema(polars_schema, schema_object)
int_range(start, end, step, dtype)
int_ranges(start, end, step, dtype)
len()
linear_space(start, end, num_samples, closed)
linear_spaces(start, end, num_samples, closed, as_array)
list(s)
lit(value, allow_object, is_scalar)
map_expr(...)
max_horizontal(exprs)
mean_horizontal(exprs, ignore_nulls)
min_horizontal(exprs)
pearson_corr(a, b)
polars_schema_field_from_arrow_c_schema(schema_object)
polars_schema_to_pycapsule(schema, compat_level)
prepare_cloud_plan(lf)
read_clipboard_string()
read_ipc_schema(py_f)
read_parquet_metadata(py_f, storage_options, credential_provider)
reduce(...)
register_plugin_function(plugin_path, function_name, args, kwargs, is_elementwise, input_wildcard_expansion, returns_scalar, cast_to_supertype, pass_name_to_apply, changes_length)
repeat(value, n, dtype=None)
rolling_corr(x, y, window_size, min_periods, ddof)
rolling_cov(x, y, window_size, min_periods, ddof)
set_decimal_separator(sep)
set_float_fmt(fmt)
set_float_precision(precision)
set_random_seed(seed)
set_thousands_separator(sep)
set_trim_decimal_zeros(trim)
spearman_rank_corr(a, b, propagate_nans)
sql_expr(sql)
sum_horizontal(exprs, ignore_nulls)
thread_pool_size()
time_range(start, end, every, closed)
time_ranges(start, end, every, closed)
when(condition)
write_clipboard_string(s)
DATA
RUNTIME_REPR = 'rt32'
__all__ = ['PySeries', 'PyDataFrame', 'PyLazyFrame', 'PyOptFlags', 'Py...
_allocator = <capsule object "polars.polars._allocator">
_debug = False
VERSION
1.43.1
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\_polars_runtime_32\_polars_runtime.pyd
polars.plugins
Python Library Documentation: module polars.plugins in polars
NAME
polars.plugins
FUNCTIONS
register_plugin_function(*, plugin_path: 'Path | str', function_name: 'str', args: 'IntoExpr | Iterable[IntoExpr]', kwargs: 'dict[str, Any] | None' = None, is_elementwise: 'bool' = False, changes_length: 'bool' = False, returns_scalar: 'bool' = False, cast_to_supertype: 'bool' = False, input_wildcard_expansion: 'bool' = False, pass_name_to_apply: 'bool' = False, use_abs_path: 'bool' = False) -> 'Expr'
Register a plugin function.
See the `user guide <https://docs.pola.rs/user-guide/plugins/expr_plugins>`_
for more information about plugins.
Parameters
----------
plugin_path
Path to the plugin package. Accepts either the file path to the dynamic library
file or the path to the directory containing it.
function_name
The name of the Rust function to register.
args
The arguments passed to this function. These get passed to the `input`
argument on the Rust side, and have to be expressions (or be convertible
to expressions).
kwargs
Non-expression arguments to the plugin function. These must be
JSON serializable.
is_elementwise
Indicate that the function operates on scalars only. This will potentially
trigger fast paths.
changes_length
Indicate that the function will change the length of the expression.
For example, a `unique` or `slice` operation.
returns_scalar
Automatically explode on unit length if the function ran as final aggregation.
This is the case for aggregations like `sum`, `min`, `covariance` etc.
cast_to_supertype
Cast the input expressions to their supertype.
input_wildcard_expansion
Expand wildcard expressions before executing the function.
pass_name_to_apply
If set to `True`, the `Series` passed to the function in a group-by operation
will ensure the name is set. This is an extra heap allocation per group.
use_abs_path
If set to `True`, the path will be resolved to an absolute path.
The path to the dynamic library is relative to the virtual environment by
default.
Returns
-------
Expr
Warnings
--------
This is highly unsafe as this will call the C function loaded by
`plugin::function_name`.
The parameters you set dictate how Polars will handle the function.
Make sure they are correct!
DATA
__all__ = ['register_plugin_function']
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\plugins.py
polars.quantile
Python Library Documentation: function quantile in module polars.functions.lazy
quantile(column: 'str', quantile: 'float | Expr', interpolation: 'QuantileMethod' = 'nearest') -> 'Expr'
Syntactic sugar for `pl.col("foo").quantile(..)`.
Parameters
----------
column
Column name.
quantile
Quantile between 0.0 and 1.0.
interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
Interpolation method.
polars.read_avro
Python Library Documentation: function read_avro in module polars.io.avro
read_avro(source: 'str | Path | IO[bytes] | bytes', *, columns: 'list[int] | list[str] | None' = None, n_rows: 'int | None' = None) -> 'DataFrame'
Read into a DataFrame from Apache Avro format.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
columns
Columns to select. Accepts a list of column indices (starting at zero) or a list
of column names.
n_rows
Stop reading from Apache Avro file after reading `n_rows`.
Returns
-------
DataFrame
polars.read_clipboard
Python Library Documentation: function read_clipboard in module polars.io.clipboard
read_clipboard(separator: 'str' = '\t', **kwargs: 'Any') -> 'DataFrame'
Read text from clipboard and pass to `read_csv`.
Useful for reading data copied from Excel or other similar spreadsheet software.
Parameters
----------
separator
Single byte character to use as separator parsing csv from clipboard.
kwargs
Additional arguments passed to `read_csv`.
See Also
--------
read_csv : Read a csv file into a DataFrame.
DataFrame.write_clipboard : Write a DataFrame to the clipboard.
polars.read_csv
Python Library Documentation: function read_csv in module polars.io.csv.functions
read_csv(source: 'str | Path | IO[str] | IO[bytes] | bytes', *, has_header: 'bool' = True, columns: 'Sequence[int] | Sequence[str] | None' = None, new_columns: 'Sequence[str] | None' = None, separator: 'str' = ',', comment_prefix: 'str | None' = None, quote_char: 'str | None' = '"', skip_rows: 'int' = 0, skip_lines: 'int' = 0, schema: 'SchemaDict | None' = None, schema_overrides: 'Mapping[str, PolarsDataType] | Sequence[PolarsDataType] | None' = None, null_values: 'str | Sequence[str] | dict[str, str] | None' = None, empty_string_is_null: 'bool' = True, ignore_errors: 'bool' = False, try_parse_dates: 'bool' = False, n_threads: 'int | None' = None, infer_schema: 'bool' = True, infer_schema_length: 'int | None' = 100, batch_size: 'int' = 8192, n_rows: 'int | None' = None, encoding: 'CsvEncoding | str' = 'utf8', low_memory: 'bool' = False, rechunk: 'bool' = False, use_pyarrow: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, skip_rows_after_header: 'int' = 0, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, sample_size: 'int' = 1024, eol_char: 'str' = '\n', raise_if_empty: 'bool' = True, truncate_ragged_lines: 'bool' = False, decimal_comma: 'bool' = False, glob: 'bool' = True) -> 'DataFrame'
Read a CSV file into a DataFrame.
Polars expects CSV data to strictly conform to RFC 4180, unless documented
otherwise. Malformed data, though common, may lead to undefined behavior.
.. versionchanged:: 0.20.31
The `dtypes` parameter was renamed `schema_overrides`.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). If `fsspec` is installed, it might be used
to open remote files. Compressed files are supported when reading from a path.
For file-like objects, the stream position may not be updated accordingly after
reading.
has_header
Indicate if the first row of the dataset is a header or not. If set to False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
columns
Columns to select. Accepts a list of column indices (starting
at zero) or a list of column names.
new_columns
Rename columns right after parsing the CSV file. If the given
list is shorter than the width of the DataFrame the remaining
columns will have their original name.
separator
Single byte character to use as separator in the file.
comment_prefix
A string used to indicate the start of a comment line. Comment lines are skipped
during parsing. Common examples of comment prefixes are `#` and `//`.
quote_char
Single byte character used for csv quoting, default = `"`.
Set to None to turn off special handling and escaping of quotes.
skip_rows
Start reading after ``skip_rows`` rows. The header will be parsed at this
offset. Note that we respect CSV escaping/comments when skipping rows.
If you want to skip by newline char only, use `skip_lines`.
skip_lines
Start reading after `skip_lines` lines. The header will be parsed at this
offset. Note that CSV escaping will not be respected when skipping lines.
If you want to skip valid CSV rows, use ``skip_rows``.
schema
Provide the schema. This means that polars doesn't do schema inference.
This argument expects the complete schema, whereas `schema_overrides` can be
used to partially overwrite a schema. Note that the order of the columns in
the provided `schema` must match the order of the columns in the CSV being read.
schema_overrides
Overwrite dtypes for specific or all columns during schema inference.
null_values
Values to interpret as null values. You can provide a:
- `str`: All values equal to this string will be null.
- `List[str]`: All values equal to any string in this list will be null.
- `Dict[str, str]`: A dictionary that maps column name to a
null value string.
empty_string_is_null
By default a missing string value is considered to be null. If
`empty_string_is_null` is set to False, missing string values are considered to
decoded as empty strings.
ignore_errors
Try to keep reading lines if some lines yield errors.
Before using this option, try to increase the number of lines used for schema
inference with e.g `infer_schema_length=10000` or override automatic dtype
inference for specific columns with the `schema_overrides` option or use
`infer_schema=False` to read all columns as `pl.String` to check which
values might cause an issue.
try_parse_dates
Try to automatically parse dates. Most ISO8601-like formats can
be inferred, as well as a handful of others. If this does not succeed,
the column remains of data type `pl.String`.
If `use_pyarrow=True`, dates will always be parsed.
n_threads
Number of threads to use in csv parsing.
Defaults to the number of physical cpu's of your system.
infer_schema
When `True`, the schema is inferred from the data using the first
`infer_schema_length` rows.
When `False`, the schema is not inferred and will be `pl.String` if not
specified in `schema` or `schema_overrides`.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data will be scanned into memory
**(this is slow)**.
Alternatively set `infer_schema=False` to read all columns as
`pl.String`.
batch_size
Number of lines to read into the buffer at once.
Modify this to change performance.
n_rows
Stop reading from CSV file after reading `n_rows`.
During multi-threaded parsing, an upper bound of `n_rows`
rows cannot be guaranteed.
encoding : {'utf8', 'utf8-lossy', 'windows-1252', 'windows-1252-lossy', ...}
Lossy means that invalid utf8 values are replaced with `�`
characters. When using other encodings than `utf8` or
`utf8-lossy`, the input is first decoded in memory with
python. Defaults to `utf8`.
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
use_pyarrow
Try to use pyarrow's native CSV parser. This will always
parse dates, even if `try_parse_dates=False`.
This is not always possible. The set of arguments given to
this function determines if it is possible to use pyarrow's
native parser. Note that pyarrow and polars may have a
different strategy regarding type inference.
storage_options
Extra options that make sense for `fsspec.open()` or a
particular storage connection.
e.g. host, port, username, password, etc.
skip_rows_after_header
Skip this number of rows when the header is parsed.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
sample_size
Set the sample size. This is used to sample statistics to estimate the
allocation needed.
.. deprecated:: 1.10.0
This parameter is now a no-op.
eol_char
Single byte end of line character (default: ``\n``). When encountering a file
with windows line endings (``\r\n``), one can go with the default ``\n``. The
extra ``\r`` will be removed when processed.
raise_if_empty
When there is no data in the source, `NoDataError` is raised. If this parameter
is set to False, an empty DataFrame (with no columns) is returned instead.
truncate_ragged_lines
Truncate lines that are longer than the schema.
decimal_comma
Parse floats using a comma as the decimal separator instead of a period.
glob
Expand path given via globbing rules.
Returns
-------
DataFrame
See Also
--------
scan_csv : Lazily read from a CSV file or multiple files via glob patterns.
Warnings
--------
Calling `read_csv().lazy()` is an antipattern as this forces Polars to materialize
a full csv file and therefore cannot push any optimizations into the reader.
Therefore always prefer `scan_csv` if you want to work with `LazyFrame` s.
Notes
-----
If the schema is inferred incorrectly (e.g. as `pl.Int64` instead of `pl.Float64`),
try to increase the number of lines used to infer the schema with
`infer_schema_length` or override the inferred dtype for those columns with
`schema_overrides`.
Examples
--------
>>> pl.read_csv("data.csv", separator="|") # doctest: +SKIP
Demonstrate use against a BytesIO object, parsing string dates.
>>> from io import BytesIO
>>> data = BytesIO(
... b"ID,Name,Birthday\n"
... b"1,Alice,1995-07-12\n"
... b"2,Bob,1990-09-20\n"
... b"3,Charlie,2002-03-08\n"
... )
>>> pl.read_csv(data, try_parse_dates=True)
shape: (3, 3)
┌─────┬─────────┬────────────┐
│ ID ┆ Name ┆ Birthday │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ date │
╞═════╪═════════╪════════════╡
│ 1 ┆ Alice ┆ 1995-07-12 │
│ 2 ┆ Bob ┆ 1990-09-20 │
│ 3 ┆ Charlie ┆ 2002-03-08 │
└─────┴─────────┴────────────┘
polars.read_csv_batched
Python Library Documentation: function read_csv_batched in module polars.io.csv.functions
read_csv_batched(source: 'str | Path', *, has_header: 'bool' = True, columns: 'Sequence[int] | Sequence[str] | None' = None, new_columns: 'Sequence[str] | None' = None, separator: 'str' = ',', comment_prefix: 'str | None' = None, quote_char: 'str | None' = '"', skip_rows: 'int' = 0, skip_lines: 'int' = 0, schema_overrides: 'Mapping[str, PolarsDataType] | Sequence[PolarsDataType] | None' = None, null_values: 'str | Sequence[str] | dict[str, str] | None' = None, empty_string_is_null: 'bool' = True, ignore_errors: 'bool' = False, try_parse_dates: 'bool' = False, n_threads: 'int | None' = None, infer_schema_length: 'int | None' = 100, batch_size: 'int' = 50000, n_rows: 'int | None' = None, encoding: 'CsvEncoding | str' = 'utf8', low_memory: 'bool' = False, rechunk: 'bool' = False, skip_rows_after_header: 'int' = 0, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, sample_size: 'int' = 1024, eol_char: 'str' = '\n', raise_if_empty: 'bool' = True, truncate_ragged_lines: 'bool' = False, decimal_comma: 'bool' = False) -> 'BatchedCsvReader'
Read a CSV file in batches.
Upon creation of the `BatchedCsvReader`, Polars will gather statistics and
determine the file chunks. After that, work will only be done if `next_batches`
is called, which will return a list of `n` frames of the given batch size.
.. versionchanged:: 0.20.31
The `dtypes` parameter was renamed `schema_overrides`.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
.. deprecated:: 1.37.0
Use `scan_csv().collect_batches()` instead.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). If `fsspec` is installed, it might be used
to open remote files. For file-like objects, the stream position may not be
updated accordingly after reading.
has_header
Indicate if the first row of the dataset is a header or not. If set to False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
columns
Columns to select. Accepts a list of column indices (starting
at zero) or a list of column names.
new_columns
Rename columns right after parsing the CSV file. If the given
list is shorter than the width of the DataFrame the remaining
columns will have their original name.
separator
Single byte character to use as separator in the file.
comment_prefix
A string used to indicate the start of a comment line. Comment lines are skipped
during parsing. Common examples of comment prefixes are `#` and `//`.
quote_char
Single byte character used for csv quoting, default = `"`.
Set to None to turn off special handling and escaping of quotes.
skip_rows
Start reading after ``skip_rows`` rows. The header will be parsed at this
offset. Note that we respect CSV escaping/comments when skipping rows.
If you want to skip by newline char only, use `skip_lines`.
skip_lines
Start reading after `skip_lines` lines. The header will be parsed at this
offset. Note that CSV escaping will not be respected when skipping lines.
If you want to skip valid CSV rows, use ``skip_rows``.
schema_overrides
Overwrite dtypes during inference.
null_values
Values to interpret as null values. You can provide a:
- `str`: All values equal to this string will be null.
- `List[str]`: All values equal to any string in this list will be null.
- `Dict[str, str]`: A dictionary that maps column name to a
null value string.
empty_string_is_null
By default a missing string value is considered to be null. If
`empty_string_is_null` is set to False, missing string values are considered to
decoded as empty strings.
ignore_errors
Try to keep reading lines if some lines yield errors.
First try `infer_schema_length=0` to read all columns as
`pl.String` to check which values might cause an issue.
try_parse_dates
Try to automatically parse dates. Most ISO8601-like formats can
be inferred, as well as a handful of others. If this does not succeed,
the column remains of data type `pl.String`.
n_threads
Number of threads to use in csv parsing.
Defaults to the number of physical cpu's of your system.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `0`, all columns will be read as `pl.String`.
If set to `None`, the full data may be scanned *(this is slow)*.
batch_size
Number of lines to read into the buffer at once.
Modify this to change performance.
n_rows
Stop reading from CSV file after reading `n_rows`.
During multi-threaded parsing, an upper bound of `n_rows`
rows cannot be guaranteed.
encoding : {'utf8', 'utf8-lossy', ...}
Lossy means that invalid utf8 values are replaced with `�`
characters. When using other encodings than `utf8` or
`utf8-lossy`, the input is first decoded in memory with
python. Defaults to `utf8`.
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
skip_rows_after_header
Skip this number of rows when the header is parsed.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
sample_size
Set the sample size. This is used to sample statistics to estimate the
allocation needed.
.. deprecated:: 1.10.0
Is a no-op.
eol_char
Single byte end of line character (default: ``\n``). When encountering a file
with windows line endings (``\r\n``), one can go with the default ``\n``. The
extra ``\r`` will be removed when processed.
raise_if_empty
When there is no data in the source,`NoDataError` is raised. If this parameter
is set to False, `None` will be returned from `next_batches(n)` instead.
truncate_ragged_lines
Truncate lines that are longer than the schema.
decimal_comma
Parse floats using a comma as the decimal separator instead of a period.
Returns
-------
BatchedCsvReader
See Also
--------
scan_csv : Lazily read from a CSV file or multiple files via glob patterns.
Examples
--------
>>> reader = pl.read_csv_batched(
... "./pdsh/tables_scale_100/lineitem.tbl",
... separator="|",
... try_parse_dates=True,
... ) # doctest: +SKIP
>>> batches = reader.next_batches(5) # doctest: +SKIP
>>> for df in batches: # doctest: +SKIP
... print(df)
Read big CSV file in batches and write a CSV file for each "group" of interest.
>>> seen_groups = set()
>>> reader = pl.read_csv_batched("big_file.csv") # doctest: +SKIP
>>> batches = reader.next_batches(100) # doctest: +SKIP
>>> while batches: # doctest: +SKIP
... df_current_batches = pl.concat(batches)
... partition_dfs = df_current_batches.partition_by("group", as_dict=True)
...
... for group, df in partition_dfs.items():
... if group in seen_groups:
... with open(f"./data/{group}.csv", "a") as fh:
... fh.write(df.write_csv(file=None, include_header=False))
... else:
... df.write_csv(file=f"./data/{group}.csv", include_header=True)
... seen_groups.add(group)
...
... batches = reader.next_batches(100)
polars.read_database
Python Library Documentation: function read_database in module polars.io.database.functions
read_database(query: 'str | TextClause | Selectable', connection: 'ConnectionOrCursor | str', *, iter_batches: 'bool' = False, batch_size: 'int | None' = None, schema_overrides: 'SchemaDict | None' = None, infer_schema_length: 'int | None' = 100, execute_options: 'dict[str, Any] | None' = None) -> 'DataFrame | Iterator[DataFrame]'
Read the results of a SQL query into a DataFrame, given a connection object.
Parameters
----------
query
SQL query to execute (if using a SQLAlchemy connection object this can
be a suitable "Selectable", otherwise it is expected to be a string).
connection
An instantiated connection (or cursor/client object) that the query can be
executed against. Can also pass a valid ODBC connection string (identified as
such if it contains the string "Driver={...}"), in which case the `arrow-odbc`
package will be used to establish the connection and return Arrow-native data
to Polars. Async driver connections are also supported, though this is currently
considered unstable. If using SQLAlchemy, you can configure the connection's
`execution_options` before passing to `read_database` to refine its behaviour
(see the `iter_batches` parameter for an example where this can be useful).
.. warning::
Use of asynchronous connections is currently considered **unstable**, and
unexpected issues may arise; if this happens, please report them.
iter_batches
Return an iterator of DataFrames, where each DataFrame represents a batch of
data returned by the query; this can be useful for processing large resultsets
in a more memory-efficient manner. If supported by the backend, this value is
passed to the underlying query execution method (note that lower values will
typically result in poor performance as they will cause many round-trips to
the database). If the backend does not support changing the batch size then
a single DataFrame is yielded from the iterator.
.. note::
If using SQLALchemy, you may also want to pass `stream_results=True` to the
connection's `execution_options` method when setting this parameter, which
will establish a server-side cursor; without this option some drivers (such
as "psycopg2") will still materialise the entire result set client-side
before batching the result locally.
batch_size
Indicate the size of each batch when `iter_batches` is True (note that you can
still set this when `iter_batches` is False, in which case the resulting
DataFrame is constructed internally using batched return before being returned
to you. Note that some backends (such as Snowflake) may support batch operation
but not allow for an explicit size to be set; in this case you will still
receive batches but their size is determined by the backend (in which case any
value set here will be ignored).
schema_overrides
A dictionary mapping column names to dtypes, used to override the schema
inferred from the query cursor or given by the incoming Arrow data (depending
on driver/backend). This can be useful if the given types can be more precisely
defined (for example, if you know that a given column can be declared as `u32`
instead of `i64`).
infer_schema_length
The maximum number of rows to scan for schema inference. If set to `None`, the
full data may be scanned *(this can be slow)*. This parameter only applies if
the data is read as a sequence of rows and the `schema_overrides` parameter
is not set for the given column; Arrow-aware drivers also ignore this value.
execute_options
These options will be passed through into the underlying query execution method
as kwargs. In the case of connections made using an ODBC string (which use
`arrow-odbc`) these options are passed to the `read_arrow_batches_from_odbc`
method.
Notes
-----
* This function supports a wide range of native database drivers (ranging from local
databases such as SQLite to large cloud databases such as Snowflake), as well as
generic libraries such as ADBC, SQLAlchemy and various flavours of ODBC. If the
backend supports returning Arrow data directly then this facility will be used to
efficiently instantiate the DataFrame; otherwise, the DataFrame is initialised
from row-wise data.
* Support for Arrow Flight SQL data is available via the `adbc-driver-flightsql`
package; see https://arrow.apache.org/adbc/current/driver/flight_sql.html for
more details about using this driver (notable databases implementing Flight SQL
include Dremio and InfluxDB).
* The `read_database_uri` function can be noticeably faster than `read_database`
if you are using a SQLAlchemy or DBAPI2 connection, as `connectorx` and `adbc`
optimise translation of the result set into Arrow format. Note that you can
determine a connection's URI from a SQLAlchemy engine object by calling
`conn.engine.url.render_as_string(hide_password=False)`.
* If Polars has to create a cursor from your connection in order to execute the
query then that cursor will be automatically closed when the query completes;
however, Polars will *never* close any other open connection or cursor.
* Polars is able to support more than just relational databases and SQL queries
through this function. For example, you can load local graph database results
from a `KùzuDB` connection in conjunction with a Cypher query, or use SurrealQL
with SurrealDB.
See Also
--------
read_database_uri : Create a DataFrame from a SQL query using a URI string.
Examples
--------
Instantiate a DataFrame from a SQL query against a user-supplied connection:
>>> df = pl.read_database(
... query="SELECT * FROM test_data",
... connection=user_conn,
... schema_overrides={"normalised_score": pl.UInt8},
... ) # doctest: +SKIP
Use a parameterised SQLAlchemy query, passing named values via `execute_options`:
>>> df = pl.read_database(
... query="SELECT * FROM test_data WHERE metric > :value",
... connection=alchemy_conn,
... execute_options={"parameters": {"value": 0}},
... ) # doctest: +SKIP
Use 'qmark' style parameterisation; values are still passed via `execute_options`,
but in this case the "parameters" value is a sequence of literals, not a dict:
>>> df = pl.read_database(
... query="SELECT * FROM test_data WHERE metric > ?",
... connection=alchemy_conn,
... execute_options={"parameters": [0]},
... ) # doctest: +SKIP
Batch the results of a large SQLAlchemy query into DataFrames, each containing
100,000 rows; explicitly establish a server-side cursor using the connection's
"execution_options" method to avoid loading the entire result locally before
batching (this is not required for all drivers, so check your driver's
documentation for more details):
>>> for df in pl.read_database(
... query="SELECT * FROM test_data",
... connection=alchemy_conn.execution_options(stream_results=True),
... iter_batches=True,
... batch_size=100_000,
... ):
... do_something(df) # doctest: +SKIP
Instantiate a DataFrame using an ODBC connection string (requires the `arrow-odbc`
package) setting upper limits on the buffer size of variadic text/binary columns:
>>> df = pl.read_database(
... query="SELECT * FROM test_data",
... connection="Driver={PostgreSQL};Server=localhost;Port=5432;Database=test;Uid=usr;Pwd=",
... execute_options={"max_text_size": 512, "max_binary_size": 1024},
... ) # doctest: +SKIP
Load data from an asynchronous SQLAlchemy driver/engine; note that asynchronous
connections and sessions are also supported here:
>>> from sqlalchemy.ext.asyncio import create_async_engine
>>> async_engine = create_async_engine("sqlite+aiosqlite:///test.db")
>>> df = pl.read_database(
... query="SELECT * FROM test_data",
... connection=async_engine,
... ) # doctest: +SKIP
Load data from an `AsyncSurrealDB` client connection object; note that both the "ws"
and "http" protocols are supported, as is the synchronous `SurrealDB` client. The
async loop can be run with standard `asyncio` or with `uvloop`:
>>> import asyncio # (or uvloop)
>>> async def surreal_query_to_frame(query: str, url: str):
... async with AsyncSurrealDB(url) as client:
... await client.use(namespace="test", database="test")
... return pl.read_database(query=query, connection=client)
>>> df = asyncio.run(
... surreal_query_to_frame(
... query="SELECT * FROM test",
... url="http://localhost:8000",
... )
... ) # doctest: +SKIP
polars.read_database_uri
Python Library Documentation: function read_database_uri in module polars.io.database.functions
read_database_uri(query: 'list[str] | str', uri: 'str', *, partition_on: 'str | None' = None, partition_range: 'tuple[int, int] | None' = None, partition_num: 'int | None' = None, protocol: 'str | None' = None, engine: 'DbReadEngine | None' = None, schema_overrides: 'SchemaDict | None' = None, execute_options: 'dict[str, Any] | None' = None, pre_execution_query: 'str | list[str] | None' = None) -> 'DataFrame'
Read the results of a SQL query into a DataFrame, given a URI.
Parameters
----------
query
Raw SQL query (or queries).
uri
A connectorx or ADBC connection URI string that starts with the backend's
driver name, for example:
* "postgresql://user:pass@server:port/database"
* "snowflake://user:pass@account/database/schema?warehouse=warehouse&role=role"
The caller is responsible for escaping any special characters in the string,
which will be passed "as-is" to the underlying engine (this is most often
required when coming across special characters in the password).
partition_on
The column on which to partition the result (connectorx).
partition_range
The value range of the partition column (connectorx).
partition_num
How many partitions to generate (connectorx).
protocol
Backend-specific transfer protocol directive (connectorx); see connectorx
documentation for more details.
engine : {'connectorx', 'adbc'}
Selects the engine used for reading the database (defaulting to connectorx):
* `'connectorx'`
Supports a range of databases, such as PostgreSQL, Redshift, MySQL, MariaDB,
Clickhouse, Oracle, BigQuery, SQL Server, and so on. For an up-to-date list
please see the connectorx docs:
https://github.com/sfu-db/connector-x#supported-sources--destinations
* `'adbc'`
Currently there is limited support for this engine, with a relatively small
number of drivers available, most of which are still in development. For
an up-to-date list of drivers please see the ADBC docs:
https://arrow.apache.org/adbc/
schema_overrides
A dictionary mapping column names to dtypes, used to override the schema
given in the data returned by the query.
execute_options
These options will be passed to the underlying query execution method as
kwargs. Note that connectorx does not support this parameter and ADBC currently
only supports positional 'qmark' style parameterization.
pre_execution_query
SQL query or list of SQL queries executed before main query (connectorx>=0.4.2).
Can be used to set runtime configurations using SET statements.
Only applicable for Postgres and MySQL source.
Only applicable with the connectorx engine.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Notes
-----
For `connectorx`, ensure that you have `connectorx>=0.3.2`. The documentation
is available `here <https://sfu-db.github.io/connector-x/intro.html>`_.
For `adbc` you will need to have installed the ADBC driver associated with the
backend you are connecting to, eg: `adbc-driver-postgresql`. For versions of
`adbc-driver-manager` < 1.7.0, `pyarrow` is also required.
If your password contains special characters, you will need to escape them.
This will usually require the use of a URL-escaping function, for example:
>>> from urllib.parse import quote, quote_plus
>>> quote_plus("pass word?")
'pass+word%3F'
>>> quote("pass word?")
'pass%20word%3F'
See Also
--------
read_database : Create a DataFrame from a SQL query using a connection object.
Examples
--------
Create a DataFrame from a SQL query using a single thread:
>>> uri = "postgresql://username:password@server:port/database"
>>> query = "SELECT * FROM lineitem"
>>> pl.read_database_uri(query, uri) # doctest: +SKIP
Create a DataFrame in parallel using 10 threads by automatically partitioning
the provided SQL on the partition column:
>>> uri = "postgresql://username:password@server:port/database"
>>> query = "SELECT * FROM lineitem"
>>> pl.read_database_uri(
... query,
... uri,
... partition_on="partition_col",
... partition_num=10,
... engine="connectorx",
... ) # doctest: +SKIP
Create a DataFrame in parallel using 2 threads by explicitly providing two
SQL queries:
>>> uri = "postgresql://username:password@server:port/database"
>>> queries = [
... "SELECT * FROM lineitem WHERE partition_col <= 10",
... "SELECT * FROM lineitem WHERE partition_col > 10",
... ]
>>> pl.read_database_uri(queries, uri, engine="connectorx") # doctest: +SKIP
Read data from Snowflake using the ADBC driver:
>>> df = pl.read_database_uri(
... "SELECT * FROM test_table",
... "snowflake://user:pass@company-org/testdb/public?warehouse=test&role=myrole",
... engine="adbc",
... ) # doctest: +SKIP
Pass a single parameter via `execute_options` into a query using the ADBC driver:
>>> df = pl.read_database_uri(
... "SELECT * FROM employees WHERE hourly_rate > ?",
... "sqlite:///:memory:",
... engine="adbc",
... execute_options={"parameters": (30,)},
... ) # doctest: +SKIP
Or pass multiple parameters:
>>> df = pl.read_database_uri(
... "SELECT * FROM employees WHERE hourly_rate BETWEEN ? AND ?",
... "sqlite:///:memory:",
... engine="adbc",
... execute_options={"parameters": (40, 20)},
... ) # doctest: +SKIP
polars.read_delta
Python Library Documentation: function read_delta in module polars.io.delta.functions
read_delta(source: 'str | Path | DeltaTable', *, version: 'int | str | datetime | None' = None, columns: 'list[str] | None' = None, rechunk: 'bool | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', delta_table_options: 'dict[str, Any] | None' = None, use_pyarrow: 'bool' = False, pyarrow_options: 'dict[str, Any] | None' = None) -> 'DataFrame'
Reads into a DataFrame from a Delta lake table.
Parameters
----------
source
DeltaTable or a Path or URI to the root of the Delta lake table.
Note: For Local filesystem, absolute and relative paths are supported but
for the supported object storages - GCS, Azure and S3 full URI must be provided.
version
Numerical version or timestamp version of the Delta lake table.
Note: If `version` is not provided, the latest version of delta lake
table is read.
columns
Columns to select. Accepts a list of column names.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
storage_options
Extra options for the storage backends supported by `deltalake`.
For cloud storages, this may include configurations for authentication etc.
More info is available `here
<https://delta-io.github.io/delta-rs/usage/loading-table/>`__.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
delta_table_options
Additional keyword arguments while reading a Delta lake Table.
use_pyarrow
Flag to enable pyarrow dataset reads.
pyarrow_options
Keyword arguments while converting a Delta lake Table to pyarrow table.
Returns
-------
DataFrame
Examples
--------
Reads a Delta table from local filesystem.
Note: Since version is not provided, the latest version of the delta table is read.
>>> table_path = "/path/to/delta-table/"
>>> pl.read_delta(table_path) # doctest: +SKIP
Reads a specific version of the Delta table from local filesystem.
Note: This will fail if the provided version of the delta table does not exist.
>>> pl.read_delta(table_path, version=1) # doctest: +SKIP
Time travel a delta table from local filesystem using a timestamp version.
>>> pl.read_delta(
... table_path, version=datetime(2020, 1, 1, tzinfo=timezone.utc)
... ) # doctest: +SKIP
Reads a Delta table from AWS S3.
See a list of supported storage options for S3 `here
<https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html#variants>`__.
>>> table_path = "s3://bucket/path/to/delta-table/"
>>> storage_options = {
... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
... }
>>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP
Reads a Delta table from Google Cloud storage (GCS).
See a list of supported storage options for GCS `here
<https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html#variants>`__.
>>> table_path = "gs://bucket/path/to/delta-table/"
>>> storage_options = {"SERVICE_ACCOUNT": "SERVICE_ACCOUNT_JSON_ABSOLUTE_PATH"}
>>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP
Reads a Delta table from Azure.
Following type of table paths are supported,
* az://<container>/<path>
* adl://<container>/<path>
* abfs://<container>/<path>
See a list of supported storage options for Azure `here
<https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants>`__.
>>> table_path = "az://container/path/to/delta-table/"
>>> storage_options = {
... "AZURE_STORAGE_ACCOUNT_NAME": "AZURE_STORAGE_ACCOUNT_NAME",
... "AZURE_STORAGE_ACCOUNT_KEY": "AZURE_STORAGE_ACCOUNT_KEY",
... }
>>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP
Reads a Delta table with additional delta specific options. In the below example,
`without_files` option is used which loads the table without file tracking
information.
>>> table_path = "/path/to/delta-table/"
>>> delta_table_options = {"without_files": True}
>>> pl.read_delta(
... table_path, delta_table_options=delta_table_options
... ) # doctest: +SKIP
polars.read_excel
Python Library Documentation: function read_excel in module polars.io.spreadsheet.functions
read_excel(source: 'FileSource | memoryview[int]', *, sheet_id: 'int | Sequence[int] | None' = None, sheet_name: 'str | list[str] | tuple[str, ...] | None' = None, table_name: 'str | None' = None, engine: 'ExcelSpreadsheetEngine' = 'calamine', engine_options: 'dict[str, Any] | None' = None, read_options: 'dict[str, Any] | None' = None, has_header: 'bool' = True, columns: 'Sequence[int] | Sequence[str] | str | None' = None, schema_overrides: 'SchemaDict | None' = None, infer_schema_length: 'int | None' = 100, include_file_paths: 'str | None' = None, drop_empty_rows: 'bool' = True, drop_empty_cols: 'bool' = True, raise_if_empty: 'bool' = True) -> 'pl.DataFrame | dict[str, pl.DataFrame]'
Read Excel spreadsheet data into a DataFrame.
.. versionadded:: 1.20
Support loading data from named table objects with `table_name` parameter.
.. versionadded:: 1.18
Support loading data from a list (or glob pattern) of multiple workbooks.
.. versionchanged:: 1.0
Default engine is now "calamine" (was "xlsx2csv").
.. versionchanged:: 0.20.7
The `read_csv_options` parameter was renamed `read_options`.
.. versionchanged:: 0.20.6
The `xlsx2csv_options` parameter was renamed `engine_options`.
Parameters
----------
source
Path(s) to a file or a file-like object (by "file-like object" we refer to
objects that have a `read()` method, such as a file handler like the builtin
`open` function, or a `BytesIO` instance). For file-like objects, the stream
position may not be updated after reading.
sheet_id
Sheet number(s) to convert (set `0` to load all sheets as DataFrames) and
return a `{sheetname:frame,}` dict. (Defaults to `1` if neither this nor
`sheet_name` are specified). Can also take a sequence of sheet numbers.
sheet_name
Sheet name(s) to convert; cannot be used in conjunction with `sheet_id`. If
more than one is given then a `{sheetname:frame,}` dict is returned.
table_name
Name of a specific table to read; note that table names are unique across
the workbook, so additionally specifying a sheet id or name is optional;
if one of those parameters *is* specified, an error will be raised if
the named table is not found in that particular sheet.
engine : {'calamine', 'openpyxl', 'xlsx2csv'}
Library used to parse the spreadsheet file; defaults to "calamine".
* "calamine": this engine can be used for reading all major types of Excel
Workbook (`.xlsx`, `.xlsb`, `.xls`) and is dramatically faster than the
other options, using the `fastexcel` module to bind the Rust-based Calamine
parser.
* "openpyxl": this engine is significantly slower than both `calamine` and
`xlsx2csv`, but can provide a useful fallback if you are otherwise unable
to read data from your workbook.
* "xlsx2csv": converts the data to an in-memory CSV before using the native
polars `read_csv` method to parse the result.
engine_options
Additional options passed to the underlying engine's primary parsing
constructor (given below), if supported:
* "calamine": n/a (can only provide `read_options`)
* "openpyxl": `load_workbook <https://openpyxl.readthedocs.io/en/stable/api/openpyxl.reader.excel.html#openpyxl.reader.excel.load_workbook>`_
* "xlsx2csv": `Xlsx2csv <https://github.com/dilshod/xlsx2csv/blob/f35734aa453d65102198a77e7b8cd04928e6b3a2/xlsx2csv.py#L157>`_
read_options
Options passed to the underlying engine method that reads the sheet data.
Where supported, this allows for additional control over parsing. The
specific read methods associated with each engine are:
* "calamine": `load_sheet_by_name <https://fastexcel.toucantoco.dev/fastexcel.html#ExcelReader.load_sheet_by_name>`_
(or `load_table <https://fastexcel.toucantoco.dev/fastexcel.html#ExcelReader.load_table>`_
if using the `table_name` parameter).
* "openpyxl": n/a (can only provide `engine_options`)
* "xlsx2csv": see :meth:`read_csv`
has_header
Indicate if the first row of the table data is a header or not. If False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
columns
Columns to read from the sheet; if not specified, all columns are read. Can
be given as a sequence of column names or indices, or a single column name.
schema_overrides
Support type specification or override of one or more columns.
infer_schema_length
The maximum number of rows to scan for schema inference. If set to `None`, the
entire dataset is scanned to determine the dtypes, which can slow parsing for
large workbooks. Note that only the "calamine" and "xlsx2csv" engines support
this parameter.
include_file_paths
Include the path of the source file(s) as a column with this name.
drop_empty_rows
Indicate whether to omit empty rows when reading data into the DataFrame.
drop_empty_cols
Indicate whether to omit empty columns (with no headers) when reading data into
the DataFrame (note that empty column identification may vary depending on the
underlying engine being used).
raise_if_empty
When there is no data in the sheet,`NoDataError` is raised. If this parameter
is set to False, an empty DataFrame (with no columns) is returned instead.
Returns
-------
DataFrame
If reading a single sheet.
dict
If reading multiple sheets, a "{sheetname: DataFrame, ...}" dict is returned.
See Also
--------
read_ods
Notes
-----
* Where possible, prefer the default "calamine" engine for reading Excel Workbooks,
as it is significantly faster than the other options.
* When using the `xlsx2csv` engine the target Excel sheet is first converted
to CSV using `xlsx2csv.Xlsx2csv(source).convert()` and then parsed with Polars'
:func:`read_csv` function. You can pass additional options to `read_options`
to influence this part of the parsing pipeline.
* If you want to read multiple sheets and set *different* options (`read_options`,
`schema_overrides`, etc), you should make separate calls as the options are set
globally, not on a per-sheet basis.
Examples
--------
Read the "data" worksheet from an Excel file into a DataFrame.
>>> pl.read_excel(
... source="test.xlsx",
... sheet_name="data",
... ) # doctest: +SKIP
If the correct dtypes can't be determined, use the `schema_overrides` parameter
to specify them, or increase the inference length with `infer_schema_length`.
>>> pl.read_excel(
... source="test.xlsx",
... schema_overrides={"dt": pl.Date},
... infer_schema_length=None,
... ) # doctest: +SKIP
Using the `xlsx2csv` engine, read table data from sheet 3 in an Excel workbook as a
DataFrame while skipping empty lines in the sheet. As sheet 3 does not have a header
row, you can pass the necessary additional settings for this to the `read_options`
parameter; these will be passed to :func:`read_csv`.
>>> pl.read_excel(
... source="test.xlsx",
... sheet_id=3,
... engine="xlsx2csv",
... engine_options={"skip_empty_lines": True},
... read_options={"has_header": False, "new_columns": ["a", "b", "c"]},
... ) # doctest: +SKIP
polars.read_ipc
Python Library Documentation: function read_ipc in module polars.io.ipc.functions
read_ipc(source: 'str | Path | IO[bytes] | bytes', *, columns: 'list[int] | list[str] | None' = None, n_rows: 'int | None' = None, use_pyarrow: 'bool' = False, memory_map: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, rechunk: 'bool' = True) -> 'DataFrame'
Read into a DataFrame from Arrow IPC (Feather v2) file.
See "File or Random Access format" on https://arrow.apache.org/docs/python/ipc.html.
Arrow IPC files are also known as Feather (v2) files.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). If `fsspec` is installed, it might be used
to open remote files. For file-like objects, the stream position may not be
updated accordingly after reading.
columns
Columns to select. Accepts a list of column indices (starting at zero) or a list
of column names.
n_rows
Stop reading from IPC file after reading `n_rows`.
Only valid when `use_pyarrow=False`.
use_pyarrow
Use pyarrow or the native Rust reader.
memory_map
Try to memory map the file. This can greatly improve performance on repeated
queries as the OS may cache pages.
Only uncompressed IPC files can be memory mapped.
storage_options
Extra options that make sense for `fsspec.open()` or a particular storage
connection, e.g. host, port, username, password, etc.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
rechunk
Make sure that all data is contiguous.
Returns
-------
DataFrame
See Also
--------
scan_ipc : Lazily read from an IPC file or multiple files via glob patterns.
Warnings
--------
Calling `read_ipc().lazy()` is an antipattern as this forces Polars to materialize
a full csv file and therefore cannot push any optimizations into the reader.
Therefore always prefer `scan_ipc` if you want to work with `LazyFrame` s.
If `memory_map` is set, the bytes on disk are mapped 1:1 to memory.
That means that:
- Arrow data in the file is not validated to be correct and invalid arrow
data is UB! Ensure this file is correct or set `memory_map=False`.
- You cannot write to the same filename.
E.g. `pl.read_ipc("my_file.arrow").write_ipc("my_file.arrow")`
will fail.
polars.read_ipc_schema
Python Library Documentation: function read_ipc_schema in module polars.io.ipc.functions
read_ipc_schema(source: 'str | Path | IO[bytes] | bytes') -> 'dict[str, DataType]'
Get the schema of an IPC file without reading data.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
Returns
-------
dict
Dictionary mapping column names to datatypes
polars.read_ipc_stream
Python Library Documentation: function read_ipc_stream in module polars.io.ipc.functions
read_ipc_stream(source: 'str | Path | IO[bytes] | bytes', *, columns: 'list[int] | list[str] | None' = None, n_rows: 'int | None' = None, use_pyarrow: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, rechunk: 'bool' = True) -> 'DataFrame'
Read into a DataFrame from Arrow IPC record batch stream.
See "Streaming format" on https://arrow.apache.org/docs/python/ipc.html.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). If `fsspec` is installed, it might be used
to open remote files. For file-like objects, the stream position may not be
updated accordingly after reading.
columns
Columns to select. Accepts a list of column indices (starting at zero) or a list
of column names.
n_rows
Stop reading from IPC stream after reading `n_rows`.
Only valid when `use_pyarrow=False`.
use_pyarrow
Use pyarrow or the native Rust reader.
storage_options
Extra options that make sense for `fsspec.open()` or a particular storage
connection, e.g. host, port, username, password, etc.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
rechunk
Make sure that all data is contiguous.
Returns
-------
DataFrame
polars.read_json
Python Library Documentation: function read_json in module polars.io.json.read
read_json(source: 'str | Path | IOBase | bytes', *, schema: 'SchemaDefinition | None' = None, schema_overrides: 'SchemaDefinition | None' = None, infer_schema_length: 'int | None' = 100) -> 'DataFrame'
Read into a DataFrame from a JSON file.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the schema param will be overridden.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
See Also
--------
read_ndjson
Examples
--------
>>> from io import StringIO
>>> json_str = '[{"foo":1,"bar":6},{"foo":2,"bar":7},{"foo":3,"bar":8}]'
>>> pl.read_json(StringIO(json_str))
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 6 │
│ 2 ┆ 7 │
│ 3 ┆ 8 │
└─────┴─────┘
With the schema defined.
>>> pl.read_json(StringIO(json_str), schema={"foo": pl.Int64, "bar": pl.Float64})
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 1 ┆ 6.0 │
│ 2 ┆ 7.0 │
│ 3 ┆ 8.0 │
└─────┴─────┘
polars.read_lines
Python Library Documentation: function read_lines in module polars.io.lines
read_lines(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]]', *, name: 'str' = 'line', n_rows: 'int | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, glob: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', include_file_paths: 'str | None' = None) -> 'DataFrame'
Read lines into a string column from a file.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
name
Name to use for the output column.
n_rows
Stop reading from parquet file after reading `n_rows`.
row_index_name
If not None, this will insert a row index column with the given name into the
DataFrame
row_index_offset
Offset to start the row index column (only used if the name is set)
glob
Expand path given via globbing rules.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \
`{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
include_file_paths
Include the path of the source file(s) as a column with this name.
See Also
--------
scan_lines
Examples
--------
>>> pl.read_lines(b"Hello\nworld")
shape: (2, 1)
┌───────┐
│ line │
│ --- │
│ str │
╞═══════╡
│ Hello │
│ world │
└───────┘
polars.read_ndjson
Python Library Documentation: function read_ndjson in module polars.io.ndjson
read_ndjson(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]]', *, schema: 'SchemaDefinition | None' = None, schema_overrides: 'SchemaDefinition | None' = None, infer_schema_length: 'int | None' = 100, batch_size: 'int | None' = 1024, n_rows: 'int | None' = None, low_memory: 'bool' = False, rechunk: 'bool' = False, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, ignore_errors: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, file_cache_ttl: 'int | None' = None, include_file_paths: 'str | None' = None) -> 'DataFrame'
Read into a DataFrame from a newline delimited JSON file.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the schema param will be overridden.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
batch_size
Number of rows to read in each batch.
n_rows
Stop reading from JSON file after reading `n_rows`.
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Reallocate to contiguous memory when all chunks/ files are parsed.
row_index_name
If not None, this will insert a row index column with give name into the
DataFrame
row_index_offset
Offset to start the row index column (only use if the name is set)
ignore_errors
Return `Null` if parsing fails because of schema mismatches.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \
`{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
file_cache_ttl
Amount of time to keep downloaded cloud files since their last access time,
in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable
(which defaults to 1 hour) if not given.
.. deprecated:: 1.39.0
File cache is no longer supported.
include_file_paths
Include the path of the source file(s) as a column with this name.
See Also
--------
scan_ndjson : Lazily read from an NDJSON file or multiple files via glob patterns.
Warnings
--------
Calling `read_ndjson().lazy()` is an antipattern as this forces Polars to
materialize a full ndjson file and therefore cannot push any optimizations into
the reader. Therefore always prefer `scan_ndjson` if you want to work with
`LazyFrame` s.
Examples
--------
>>> from io import StringIO
>>> json_str = '{"foo":1,"bar":6}\n{"foo":2,"bar":7}\n{"foo":3,"bar":8}\n'
>>> pl.read_ndjson(StringIO(json_str))
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 6 │
│ 2 ┆ 7 │
│ 3 ┆ 8 │
└─────┴─────┘
polars.read_ods
Python Library Documentation: function read_ods in module polars.io.spreadsheet.functions
read_ods(source: 'FileSource', *, sheet_id: 'int | Sequence[int] | None' = None, sheet_name: 'str | list[str] | tuple[str, ...] | None' = None, has_header: 'bool' = True, columns: 'Sequence[int] | Sequence[str] | None' = None, schema_overrides: 'SchemaDict | None' = None, infer_schema_length: 'int | None' = 100, include_file_paths: 'str | None' = None, drop_empty_rows: 'bool' = True, drop_empty_cols: 'bool' = True, raise_if_empty: 'bool' = True) -> 'pl.DataFrame | dict[str, pl.DataFrame]'
Read OpenOffice (ODS) spreadsheet data into a DataFrame.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
sheet_id
Sheet number(s) to convert, starting from 1 (set `0` to load *all* worksheets
as DataFrames) and return a `{sheetname:frame,}` dict. (Defaults to `1` if
neither this nor `sheet_name` are specified). Can also take a sequence of sheet
numbers.
sheet_name
Sheet name(s) to convert; cannot be used in conjunction with `sheet_id`. If
more than one is given then a `{sheetname:frame,}` dict is returned.
has_header
Indicate if the first row of the table data is a header or not. If False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
columns
Columns to read from the sheet; if not specified, all columns are read. Can
be given as a sequence of column names or indices.
schema_overrides
Support type specification or override of one or more columns.
infer_schema_length
The maximum number of rows to scan for schema inference. If set to `None`, the
entire dataset is scanned to determine the dtypes, which can slow parsing for
large workbooks.
include_file_paths
Include the path of the source file(s) as a column with this name.
drop_empty_rows
Indicate whether to omit empty rows when reading data into the DataFrame.
drop_empty_cols
Indicate whether to omit empty columns (with no headers) when reading data into
the DataFrame (note that empty column identification may vary depending on the
underlying engine being used).
raise_if_empty
When there is no data in the sheet,`NoDataError` is raised. If this parameter
is set to False, an empty DataFrame (with no columns) is returned instead.
Returns
-------
DataFrame, or a `{sheetname: DataFrame, ...}` dict if reading multiple sheets.
See Also
--------
read_excel
Examples
--------
Read the "data" worksheet from an OpenOffice spreadsheet file into a DataFrame.
>>> pl.read_ods(
... source="test.ods",
... sheet_name="data",
... ) # doctest: +SKIP
If the correct dtypes can't be determined, use the `schema_overrides` parameter
to specify them, or increase the inference length with `infer_schema_length`.
>>> pl.read_ods(
... source="test.ods",
... sheet_id=3,
... schema_overrides={"dt": pl.Date},
... raise_if_empty=False,
... ) # doctest: +SKIP
polars.read_parquet
Python Library Documentation: function read_parquet in module polars.io.parquet.functions
read_parquet(source: 'FileSource', *, columns: 'list[int] | list[str] | None' = None, n_rows: 'int | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, parallel: 'ParallelStrategy' = 'auto', use_statistics: 'bool' = True, hive_partitioning: 'bool | None' = None, glob: 'bool' = True, schema: 'SchemaDict | None' = None, hive_schema: 'SchemaDict | None' = None, try_parse_hive_dates: 'bool' = True, rechunk: 'bool' = False, low_memory: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, use_pyarrow: 'bool' = False, pyarrow_options: 'dict[str, Any] | None' = None, memory_map: 'bool' = True, include_file_paths: 'str | None' = None, missing_columns: "Literal['insert', 'raise']" = 'raise', allow_missing_columns: 'bool | None' = None) -> 'DataFrame'
Read into a DataFrame from a parquet file.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
File-like objects are supported (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
columns
Columns to select. Accepts a list of column indices (starting at zero) or a list
of column names.
n_rows
Stop reading from parquet file after reading `n_rows`.
Only valid when `use_pyarrow=False`.
row_index_name
Insert a row index column with the given name into the DataFrame as the first
column. If set to `None` (default), no row index column is created.
row_index_offset
Start the row index at this offset. Cannot be negative.
Only used if `row_index_name` is set.
parallel : {'auto', 'columns', 'row_groups', 'none'}
This determines the direction of parallelism. 'auto' will try to determine the
optimal direction.
use_statistics
Use statistics in the parquet to determine if pages
can be skipped from reading.
hive_partitioning
Infer statistics and schema from Hive partitioned URL and use them
to prune reads. This is unset by default (i.e. `None`), meaning it is
automatically enabled when a single directory is passed, and otherwise
disabled.
glob
Expand path given via globbing rules.
schema
Specify the datatypes of the columns. The datatypes must match the
datatypes in the file(s). If there are extra columns that are not in the
file(s), consider also passing `missing_columns='insert'`.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
hive_schema
The column names and data types of the columns by which the data is partitioned.
If set to `None` (default), the schema of the Hive partitions is inferred.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
try_parse_hive_dates
Whether to try parsing hive values as date/datetime types.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
low_memory
Reduce memory pressure at the expense of performance.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
use_pyarrow
Use PyArrow instead of the Rust-native Parquet reader. The PyArrow reader is
more stable.
pyarrow_options
Keyword arguments for `pyarrow.parquet.read_table
<https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_table.html>`_.
memory_map
Memory map underlying file. This will likely increase performance.
Only used when `use_pyarrow=True`.
include_file_paths
Include the path of the source file(s) as a column with this name.
Only valid when `use_pyarrow=False`.
missing_columns
Configuration for behavior when columns defined in the schema
are missing from the data:
* `insert`: Inserts the missing columns using NULLs as the row values.
* `raise`: Raises an error.
allow_missing_columns
When reading a list of parquet files, if a column existing in the first
file cannot be found in subsequent files, the default behavior is to
raise an error. However, if `allow_missing_columns` is set to
`True`, a full-NULL column is returned instead of erroring for the files
that do not contain the column.
.. deprecated:: 1.30.0
Use the parameter `missing_columns` instead and pass one of
`('insert', 'raise')`.
Returns
-------
DataFrame
See Also
--------
scan_parquet: Lazily read from a parquet file or multiple files via glob patterns.
scan_pyarrow_dataset
Warnings
--------
Calling `read_parquet().lazy()` is an antipattern as this forces Polars to
materialize a full parquet file and therefore cannot push any optimizations
into the reader. Therefore always prefer `scan_parquet` if you want to work
with `LazyFrame` s.
polars.read_parquet_metadata
Python Library Documentation: function read_parquet_metadata in module polars.io.parquet.functions
read_parquet_metadata(source: 'str | Path | IO[bytes] | bytes', storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None) -> 'dict[str, str]'
Get file-level custom metadata of a Parquet file without reading data.
.. warning::
This functionality is considered **experimental**. It may be removed or
changed at any point without it being considered a breaking change.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
Returns
-------
dict
Dictionary with the metadata. Empty if no custom metadata is available.
polars.read_parquet_schema
Python Library Documentation: function read_parquet_schema in module polars.io.parquet.functions
read_parquet_schema(source: 'str | Path | IO[bytes] | bytes') -> 'dict[str, DataType]'
Get the schema of a Parquet file without reading data.
If you would like to read the schema of a cloud file with authentication
configuration, it is recommended use `scan_parquet` - e.g.
`scan_parquet(..., storage_options=...).collect_schema()`.
Parameters
----------
source
Path to a file or a file-like object (by "file-like object" we refer to objects
that have a `read()` method, such as a file handler like the builtin `open`
function, or a `BytesIO` instance). For file-like objects, the stream position
may not be updated accordingly after reading.
Returns
-------
dict
Dictionary mapping column names to datatypes
See Also
--------
scan_parquet
polars.reduce
Python Library Documentation: function reduce in module polars.functions.lazy
reduce(function: 'Callable[[Series, Series], Series]', exprs: 'Sequence[Expr | str] | Expr', *, returns_scalar: 'bool' = False, return_dtype: 'pl.DataTypeExpr | PolarsDataType | None' = None) -> 'Expr'
Accumulate over multiple columns horizontally/ row wise with a left fold.
Parameters
----------
function
Function to apply over the accumulator and the value.
Fn(acc, value) -> new_value
exprs
Expressions to aggregate over. May also be a wildcard expression.
returns_scalar
Whether or not `function` applied returns a scalar. This must be set correctly
by the user.
return_dtype
Output datatype.
If not set, the dtype will be inferred based on the dtype of the input
expressions.
Notes
-----
See `fold` for the version with an explicit accumulator.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 2, 3],
... "b": [0, 1, 2],
... }
... )
>>> df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 0 │
│ 2 ┆ 1 │
│ 3 ┆ 2 │
└─────┴─────┘
Horizontally sum over all columns.
>>> df.select(
... pl.reduce(function=lambda acc, x: acc + x, exprs=pl.col("*")).alias("sum")
... )
shape: (3, 1)
┌─────┐
│ sum │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 3 │
│ 5 │
└─────┘
polars.register_extension_type
Python Library Documentation: function register_extension_type in module polars.datatypes.extension
register_extension_type(ext_name: 'str', ext_class: 'type[dt.BaseExtension] | None' = None, *, as_storage: 'bool' = False) -> 'None'
Register the extension type for the given extension name.
.. warning::
This functionality is currently considered **unstable**. It may be
changed at any point without it being considered a breaking change.
polars.repeat
Python Library Documentation: function repeat in module polars.functions.repeat
repeat(value: 'IntoExpr | None', n: 'int | Expr', *, dtype: 'PolarsDataType | None' = None, eager: 'bool' = False) -> 'Expr | Series'
Construct a column of length `n` filled with the given value.
Parameters
----------
value
Value to repeat.
n
Length of the resulting column.
dtype
Data type of the resulting column. If set to `None` (default), data type is
inferred from the given value. Defaults to Int32 for integer values, unless
Int64 is required to fit the given value. Defaults to Float64 for float values.
eager
Evaluate immediately and return a `Series`. If set to `False` (default),
return an expression instead.
Notes
-----
If you want to construct a column in lazy mode and do not need a pre-determined
length, use :func:`lit` instead.
See Also
--------
lit
Examples
--------
Construct a column with a repeated value in a lazy context.
>>> pl.select(pl.repeat("z", n=3)).to_series()
shape: (3,)
Series: 'repeat' [str]
[
"z"
"z"
"z"
]
Generate a Series directly by setting `eager=True`.
>>> pl.repeat(3, n=3, dtype=pl.Int8, eager=True)
shape: (3,)
Series: 'repeat' [i8]
[
3
3
3
]
polars.rolling_corr
Python Library Documentation: function rolling_corr in module polars.functions.lazy
rolling_corr(a: 'str | Expr', b: 'str | Expr', *, window_size: 'int', min_samples: 'int | None' = None, ddof: 'int' = 1) -> 'Expr'
Compute the rolling correlation between two columns/ expressions.
The window at a given row includes the row itself and the
`window_size - 1` elements before it.
.. versionchanged:: 1.21.0
The `min_periods` parameter was renamed `min_samples`.
Parameters
----------
a
Column name or Expression.
b
Column name or Expression.
window_size
The length of the window.
min_samples
The number of values in the window that should be non-null before computing
a result. If None, it will be set equal to window size.
ddof
Has no effect, do not use.
.. deprecated:: 1.40.0
polars.rolling_cov
Python Library Documentation: function rolling_cov in module polars.functions.lazy
rolling_cov(a: 'str | Expr', b: 'str | Expr', *, window_size: 'int', min_samples: 'int | None' = None, ddof: 'int' = 1) -> 'Expr'
Compute the rolling covariance between two columns/ expressions.
The window at a given row includes the row itself and the
`window_size - 1` elements before it.
.. versionchanged:: 1.21.0
The `min_periods` parameter was renamed `min_samples`.
Parameters
----------
a
Column name or Expression.
b
Column name or Expression.
window_size
The length of the window.
min_samples
The number of values in the window that should be non-null before computing
a result. If None, it will be set equal to window size.
ddof
Delta degrees of freedom. The divisor used in calculations
is `N - ddof`, where `N` represents the number of elements.
polars.row_index
Python Library Documentation: function row_index in module polars.functions.lazy
row_index(name: 'str' = 'index') -> 'pl.Expr'
Generates a sequence of integers.
The length of the returned sequence will match the context length, and the
datatype will match the one returned by `get_index_dtype()`.
.. versionadded:: 1.32.0
If you would like to generate sequences with custom offsets / length /
step size / datatypes, it is recommended to use `int_range` instead.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
name
Name of the returned column.
Returns
-------
Expr
Column of integers.
See Also
--------
int_range : Generate a range of integers.
Examples
--------
>>> df = pl.DataFrame({"x": ["A", "A", "B", "B", "B"]})
>>> df.with_columns(pl.row_index(), pl.row_index("another_index"))
shape: (5, 3)
┌─────┬───────┬───────────────┐
│ x ┆ index ┆ another_index │
│ --- ┆ --- ┆ --- │
│ str ┆ u32 ┆ u32 │
╞═════╪═══════╪═══════════════╡
│ A ┆ 0 ┆ 0 │
│ A ┆ 1 ┆ 1 │
│ B ┆ 2 ┆ 2 │
│ B ┆ 3 ┆ 3 │
│ B ┆ 4 ┆ 4 │
└─────┴───────┴───────────────┘
>>> df.group_by("x").agg(pl.row_index()).sort("x")
shape: (2, 2)
┌─────┬───────────┐
│ x ┆ index │
│ --- ┆ --- │
│ str ┆ list[u32] │
╞═════╪═══════════╡
│ A ┆ [0, 1] │
│ B ┆ [0, 1, 2] │
└─────┴───────────┘
>>> df.select(pl.row_index())
shape: (5, 1)
┌───────┐
│ index │
│ --- │
│ u32 │
╞═══════╡
│ 0 │
│ 1 │
│ 2 │
│ 3 │
│ 4 │
└───────┘
polars.scan_arrow_c_stream
Python Library Documentation: function scan_arrow_c_stream in module polars.io.arrow_c_stream
scan_arrow_c_stream(source: 'Any') -> 'LazyFrame'
Scan a source that implements the Arrow PyCapsule Interface.
This creates a lazy scan over an object that exposes a `__arrow_c_stream__`
method (e.g. a `pyarrow.RecordBatchReader`, or a `nanoarrow` stream). The
source is consumed, batch by batch, during query execution.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
source
Any object implementing the Arrow PyCapsule Interface
(i.e. it has a `__arrow_c_stream__` method).
Returns
-------
LazyFrame
Notes
-----
- The source must produce Arrow RecordBatches with a struct-typed schema.
See: https://arrow.apache.org/docs/format/CStreamInterface.html.
- Once the stream is consumed, the same source cannot be scanned again.
Examples
--------
Scan from a PyArrow RecordBatchReader:
>>> import pyarrow as pa
>>> schema = pa.schema([("a", pa.int64()), ("b", pa.string())])
>>> batches = [
... pa.record_batch([[1, 2, 3], ["x", "y", "z"]], schema=schema),
... pa.record_batch([[4, 5], ["a", "b"]], schema=schema),
... ]
>>> reader = pa.RecordBatchReader.from_batches(schema, batches)
>>> pl.scan_arrow_c_stream(reader).collect()
shape: (5, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1 ┆ x │
│ 2 ┆ y │
│ 3 ┆ z │
│ 4 ┆ a │
│ 5 ┆ b │
└─────┴─────┘
polars.scan_csv
Python Library Documentation: function scan_csv in module polars.io.csv.functions
scan_csv(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]] | list[bytes]', *, has_header: 'bool' = True, separator: 'str' = ',', comment_prefix: 'str | None' = None, quote_char: 'str | None' = '"', skip_rows: 'int' = 0, skip_lines: 'int' = 0, schema: 'SchemaDict | None' = None, schema_overrides: 'SchemaDict | Sequence[PolarsDataType] | None' = None, null_values: 'str | Sequence[str] | dict[str, str] | None' = None, empty_string_is_null: 'bool' = True, ignore_errors: 'bool' = False, cache: 'bool | None' = None, with_column_names: 'Callable[[list[str]], list[str]] | None' = None, infer_schema: 'bool' = True, infer_schema_length: 'int | None' = 100, n_rows: 'int | None' = None, encoding: 'CsvEncoding' = 'utf8', low_memory: 'bool' = False, rechunk: 'bool' = False, skip_rows_after_header: 'int' = 0, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, try_parse_dates: 'bool' = False, eol_char: 'str' = '\n', new_columns: 'Sequence[str] | None' = None, raise_if_empty: 'bool' = True, truncate_ragged_lines: 'bool' = False, decimal_comma: 'bool' = False, glob: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, file_cache_ttl: 'int | None' = None, include_file_paths: 'str | None' = None, missing_columns: "Literal['insert', 'raise'] | None" = None) -> 'LazyFrame'
Lazily read from a CSV file or multiple files via glob patterns.
This allows the query optimizer to push down predicates and
projections to the scan level, thereby potentially reducing
memory overhead.
.. versionchanged:: 0.20.31
The `dtypes` parameter was renamed `schema_overrides`.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
has_header
Indicate if the first row of the dataset is a header or not. If set to False,
column names will be autogenerated in the following format: `column_x`, with
`x` being an enumeration over every column in the dataset, starting at 1.
separator
Single byte character to use as separator in the file.
comment_prefix
A string used to indicate the start of a comment line. Comment lines are skipped
during parsing. Common examples of comment prefixes are `#` and `//`.
quote_char
Single byte character used for csv quoting, default = `"`.
Set to None to turn off special handling and escaping of quotes.
skip_rows
Start reading after ``skip_rows`` rows. The header will be parsed at this
offset. Note that we respect CSV escaping/comments when skipping rows.
If you want to skip by newline char only, use `skip_lines`.
skip_lines
Start reading after `skip_lines` lines. The header will be parsed at this
offset. Note that CSV escaping will not be respected when skipping lines.
If you want to skip valid CSV rows, use ``skip_rows``.
schema
Provide the schema. This means that polars doesn't do schema inference.
This argument expects the complete schema, whereas `schema_overrides` can be
used to partially overwrite a schema. Note that the order of the columns in
the provided `schema` must match the order of the columns in the CSV being read.
schema_overrides
Overwrite dtypes during inference; should be a {colname:dtype,} dict or,
if providing a list of strings to `new_columns`, a list of dtypes of
the same length.
null_values
Values to interpret as null values. You can provide a:
- `str`: All values equal to this string will be null.
- `List[str]`: All values equal to any string in this list will be null.
- `Dict[str, str]`: A dictionary that maps column name to a
null value string.
empty_string_is_null
By default a missing string value is considered to be null. If
`empty_string_is_null` is set to False, missing string values are considered to
decoded as empty strings.
ignore_errors
Try to keep reading lines if some lines yield errors.
First try `infer_schema=False` to read all columns as
`pl.String` to check which values might cause an issue.
cache
Cache the result after reading.
.. deprecated:: 1.39.0
File cache is no longer supported.
with_column_names
Apply a function over the column names just in time (when they are determined);
this function will receive (and should return) a list of column names.
infer_schema
When `True`, the schema is inferred from the data using the first
`infer_schema_length` rows.
When `False`, the schema is not inferred and will be `pl.String` if not
specified in `schema` or `schema_overrides`.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data will be scanned into memory
**(this is slow)**.
Alternatively set `infer_schema=False` to read all columns as
`pl.String`.
n_rows
Stop reading from CSV file after reading `n_rows`.
encoding : {'utf8', 'utf8-lossy'}
Lossy means that invalid utf8 values are replaced with `�`
characters. Defaults to "utf8".
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Reallocate to contiguous memory when all chunks/ files are parsed.
skip_rows_after_header
Skip this number of rows when the header is parsed.
row_index_name
If not None, this will insert a row index column with the given name into
the DataFrame.
row_index_offset
Offset to start the row index column (only used if the name is set).
try_parse_dates
Try to automatically parse dates. Most ISO8601-like formats
can be inferred, as well as a handful of others. If this does not succeed,
the column remains of data type `pl.String`.
eol_char
Single byte end of line character (default: ``\n``). When encountering a file
with windows line endings (``\r\n``), one can go with the default ``\n``. The
extra ``\r`` will be removed when processed.
new_columns
Provide an explicit list of string column names to use (for example, when
scanning a headerless CSV file). If the given list is shorter than the width of
the DataFrame the remaining columns will have their original name.
raise_if_empty
When there is no data in the source, `NoDataError` is raised. If this parameter
is set to False, an empty LazyFrame (with no columns) is returned instead.
truncate_ragged_lines
Truncate lines that are longer than the schema.
decimal_comma
Parse floats using a comma as the decimal separator instead of a period.
glob
Expand path given via globbing rules.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \
`{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
file_cache_ttl
Amount of time to keep downloaded cloud files since their last access time,
in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable
(which defaults to 1 hour) if not given.
.. deprecated:: 1.39.0
File cache is no longer supported.
include_file_paths
Include the path of the source file(s) as a column with this name.
missing_columns
Configuration for behavior when columns defined in the schema are
missing from the data:
* ``"insert"``: Insert the missing columns with NULL values.
* ``"raise"``: Raise an error.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Returns
-------
LazyFrame
See Also
--------
read_csv : Read a CSV file into a DataFrame.
Examples
--------
>>> import pathlib
>>>
>>> (
... pl.scan_csv("my_long_file.csv") # lazy, doesn't do a thing
... .select(
... ["a", "c"]
... ) # select only 2 columns (other columns will not be read)
... .filter(
... pl.col("a") > 10
... ) # the filter is pushed down the scan, so less data is read into memory
... .head(100) # constrain number of returned results to 100
... ) # doctest: +SKIP
We can use `with_column_names` to modify the header before scanning:
>>> df = pl.DataFrame(
... {"BrEeZaH": [1, 2, 3, 4], "LaNgUaGe": ["is", "hard", "to", "read"]}
... )
>>> path: pathlib.Path = dirpath / "mydf.csv"
>>> df.write_csv(path)
>>> pl.scan_csv(
... path, with_column_names=lambda cols: [col.lower() for col in cols]
... ).collect()
shape: (4, 2)
┌─────────┬──────────┐
│ breezah ┆ language │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════════╪══════════╡
│ 1 ┆ is │
│ 2 ┆ hard │
│ 3 ┆ to │
│ 4 ┆ read │
└─────────┴──────────┘
You can also simply replace column names (or provide them if the file has none)
by passing a list of new column names to the `new_columns` parameter:
>>> df.write_csv(path)
>>> pl.scan_csv(
... path,
... new_columns=["idx", "txt"],
... schema_overrides=[pl.UInt16, pl.String],
... ).collect()
shape: (4, 2)
┌─────┬──────┐
│ idx ┆ txt │
│ --- ┆ --- │
│ u16 ┆ str │
╞═════╪══════╡
│ 1 ┆ is │
│ 2 ┆ hard │
│ 3 ┆ to │
│ 4 ┆ read │
└─────┴──────┘
polars.scan_delta
Python Library Documentation: function scan_delta in module polars.io.delta.functions
scan_delta(source: 'str | Path | DeltaTable', *, version: 'int | str | datetime | None' = None, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', delta_table_options: 'dict[str, Any] | None' = None, use_pyarrow: 'bool' = False, pyarrow_options: 'dict[str, Any] | None' = None, rechunk: 'bool | None' = None) -> 'LazyFrame'
Lazily read from a Delta lake table.
Parameters
----------
source
DeltaTable or a Path or URI to the root of the Delta lake table.
Note: For Local filesystem, absolute and relative paths are supported but
for the supported object storages - GCS, Azure and S3 full URI must be provided.
version
Numerical version or timestamp version of the Delta lake table.
Note: If `version` is not provided, the latest version of delta lake
table is read.
storage_options
Extra options for the storage backends supported by `deltalake`.
For cloud storages, this may include configurations for authentication etc.
More info is available `here
<https://delta-io.github.io/delta-rs/usage/loading-table/>`__.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
delta_table_options
Additional keyword arguments while reading a Delta lake Table.
use_pyarrow
Flag to enable pyarrow dataset reads.
pyarrow_options
Keyword arguments while converting a Delta lake Table to pyarrow table.
Use this parameter when filtering on partitioned columns or to read
from a 'fsspec' supported filesystem.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
Returns
-------
LazyFrame
Examples
--------
Creates a scan for a Delta table from local filesystem.
Note: Since version is not provided, the latest version of the delta table is read.
>>> table_path = "/path/to/delta-table/"
>>> pl.scan_delta(table_path).collect() # doctest: +SKIP
Creates a scan for a specific version of the Delta table from local filesystem.
Note: This will fail if the provided version of the delta table does not exist.
>>> pl.scan_delta(table_path, version=1).collect() # doctest: +SKIP
Time travel a delta table from local filesystem using a timestamp version.
>>> pl.scan_delta(
... table_path, version=datetime(2020, 1, 1, tzinfo=timezone.utc)
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table from AWS S3.
See a list of supported storage options for S3 `here
<https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html#variants>`__.
>>> table_path = "s3://bucket/path/to/delta-table/"
>>> storage_options = {
... "AWS_REGION": "eu-central-1",
... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
... }
>>> pl.scan_delta(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table from Google Cloud storage (GCS).
See a list of supported storage options for GCS `here
<https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html#variants>`__.
>>> table_path = "gs://bucket/path/to/delta-table/"
>>> storage_options = {"SERVICE_ACCOUNT": "SERVICE_ACCOUNT_JSON_ABSOLUTE_PATH"}
>>> pl.scan_delta(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table from Azure.
Supported options for Azure are available `here
<https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html#variants>`__.
Following type of table paths are supported,
* az://<container>/<path>
* adl://<container>/<path>
* abfs[s]://<container>/<path>
>>> table_path = "az://container/path/to/delta-table/"
>>> storage_options = {
... "AZURE_STORAGE_ACCOUNT_NAME": "AZURE_STORAGE_ACCOUNT_NAME",
... "AZURE_STORAGE_ACCOUNT_KEY": "AZURE_STORAGE_ACCOUNT_KEY",
... }
>>> pl.scan_delta(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table with additional delta specific options.
In the below example, `without_files` option is used which loads the table without
file tracking information.
>>> table_path = "/path/to/delta-table/"
>>> delta_table_options = {"without_files": True}
>>> pl.scan_delta(
... table_path, delta_table_options=delta_table_options
... ).collect() # doctest: +SKIP
polars.scan_iceberg
Python Library Documentation: function scan_iceberg in module polars.io.iceberg.functions
scan_iceberg(source: 'str | pyiceberg.table.Table', *, snapshot_id: 'int | None' = None, storage_options: 'StorageOptionsDict | None' = None, catalog: 'pyiceberg.catalog.Catalog | polars.io.iceberg.IcebergCatalogConfig | None' = None, reader_override: "Literal['native', 'pyiceberg'] | None" = None, use_metadata_statistics: 'bool' = True, fast_deletion_count: 'bool | None' = None, use_pyiceberg_filter: 'bool' = True) -> 'LazyFrame'
Lazily read from an Apache Iceberg table.
Parameters
----------
source
A PyIceberg table, or a 'namespace.table_name' identifier string,
or an absolute path to the metadata.
snapshot_id
The snapshot ID to scan from.
storage_options
Extra options for the storage backends supported by `pyiceberg`.
For cloud storages, this may include configurations for authentication etc.
More info is available `here <https://py.iceberg.apache.org/configuration/>`__.
catalog
PyIceberg catalog to load the table from if the provided `target`
was a table name.
reader_override
Overrides the reader used to read the data.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Note that this parameter should not be necessary outside of testing, as
polars will by default automatically select the best reader.
Available options:
* native: Uses polars native reader. This allows for more optimizations to
improve performance.
* pyiceberg: Uses PyIceberg, which may support more features.
use_metadata_statistics
Whether to allow using statistics from Iceberg metadata files.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
When a filter is present, this allows using min/max statistics present
in the Iceberg metadata files can be used to allow the reader to skip
scanning of metadata from data files that are guaranteed to not match
the filter.
If a row-count is requested (i.e. `scan_iceberg().select(pl.len())`), this
allows returning a count directly from Iceberg metadata. Note however that
for datasets containing position delete files, `fast_deletion_count` must
also be enabled for this to work.
fast_deletion_count
Allows returning a row count calculated directly from Iceberg metadata
for datasets that contain position delete files. This will give incorrect
results if position delete files contain duplicated entries.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
use_pyiceberg_filter
Convert and push the filter to PyIceberg where possible.
Returns
-------
LazyFrame
Examples
--------
Creates a scan for an Iceberg table from local filesystem, or object store.
>>> table_path = "file:/path/to/iceberg-table/metadata.json"
>>> pl.scan_iceberg(table_path).collect() # doctest: +SKIP
Creates a scan for an Iceberg table from S3.
See a list of supported storage options for S3 `here
<https://py.iceberg.apache.org/configuration/#fileio>`__.
>>> table_path = "s3://bucket/path/to/iceberg-table/metadata.json"
>>> storage_options = {
... "s3.region": "eu-central-1",
... "s3.access-key-id": "THE_AWS_ACCESS_KEY_ID",
... "s3.secret-access-key": "THE_AWS_SECRET_ACCESS_KEY",
... }
>>> pl.scan_iceberg(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for an Iceberg table from Azure.
Supported options for Azure are available `here
<https://py.iceberg.apache.org/configuration/#azure-data-lake>`__.
Following type of table paths are supported:
* az://<container>/<path>/metadata.json
* adl://<container>/<path>/metadata.json
* abfs[s]://<container>/<path>/metadata.json
>>> table_path = "az://container/path/to/iceberg-table/metadata.json"
>>> storage_options = {
... "adlfs.account-name": "AZURE_STORAGE_ACCOUNT_NAME",
... "adlfs.account-key": "AZURE_STORAGE_ACCOUNT_KEY",
... }
>>> pl.scan_iceberg(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for an Iceberg table from Google Cloud Storage.
Supported options for GCS are available `here
<https://py.iceberg.apache.org/configuration/#google-cloud-storage>`__.
>>> table_path = "s3://bucket/path/to/iceberg-table/metadata.json"
>>> storage_options = {
... "gcs.project-id": "my-gcp-project",
... "gcs.oauth.token": "ya29.dr.AfM...",
... }
>>> pl.scan_iceberg(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for an Iceberg table with additional options.
In the below example, `without_files` option is used which loads the table without
file tracking information.
>>> table_path = "/path/to/iceberg-table/metadata.json"
>>> storage_options = {"py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO"}
>>> pl.scan_iceberg(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for an Iceberg table using a specific snapshot ID.
>>> table_path = "/path/to/iceberg-table/metadata.json"
>>> snapshot_id = 7051579356916758811
>>> pl.scan_iceberg(table_path, snapshot_id=snapshot_id).collect() # doctest: +SKIP
polars.scan_ipc
Python Library Documentation: function scan_ipc in module polars.io.ipc.functions
scan_ipc(source: 'str | Path | IO[bytes] | bytes | list[str] | list[Path] | list[IO[bytes]] | list[bytes]', *, n_rows: 'int | None' = None, cache: 'bool | None' = None, rechunk: 'bool' = False, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, glob: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', memory_map: 'bool' = True, retries: 'int | None' = None, file_cache_ttl: 'int | None' = None, hive_partitioning: 'bool | None' = None, hive_schema: 'SchemaDict | None' = None, try_parse_hive_dates: 'bool' = True, include_file_paths: 'str | None' = None, _record_batch_statistics: 'bool' = False) -> 'LazyFrame'
Lazily read from an Arrow IPC (Feather v2) file or multiple files via glob patterns.
This allows the query optimizer to push down predicates and projections to the scan
level, thereby potentially reducing memory overhead.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
n_rows
Stop reading from IPC file after reading `n_rows`.
cache
Cache the result after reading.
.. deprecated:: 1.40.0
File cache is no longer supported.
rechunk
Reallocate to contiguous memory when all chunks/ files are parsed.
row_index_name
If not None, this will insert a row index column with give name into the
DataFrame
row_index_offset
Offset to start the row index column (only use if the name is set)
glob
Expand path given via globbing rules.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
memory_map
Try to memory map the file. This can greatly improve performance on repeated
queries as the OS may cache pages.
Only uncompressed IPC files can be memory mapped.
.. deprecated:: 1.40.0
Controlling memory map behavior is no longer supported.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
file_cache_ttl
Amount of time to keep downloaded cloud files since their last access time,
in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable
(which defaults to 1 hour) if not given.
.. deprecated:: 1.40.0
File cache is no longer supported.
hive_partitioning
Infer statistics and schema from Hive partitioned URL and use them
to prune reads. This is unset by default (i.e. `None`), meaning it is
automatically enabled when a single directory is passed, and otherwise
disabled.
hive_schema
The column names and data types of the columns by which the data is partitioned.
If set to `None` (default), the schema of the Hive partitions is inferred.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
try_parse_hive_dates
Whether to try parsing hive values as date/datetime types.
include_file_paths
Include the path of the source file(s) as a column with this name.
polars.scan_lines
Python Library Documentation: function scan_lines in module polars.io.lines
scan_lines(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]]', *, name: 'str' = 'line', n_rows: 'int | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, glob: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', include_file_paths: 'str | None' = None) -> 'LazyFrame'
Construct a LazyFrame which scans lines into a string column from a file.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
name
Name to use for the output column.
n_rows
Stop reading from parquet file after reading `n_rows`.
row_index_name
If not None, this will insert a row index column with the given name into the
DataFrame
row_index_offset
Offset to start the row index column (only used if the name is set)
glob
Expand path given via globbing rules.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: \
`{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
include_file_paths
Include the path of the source file(s) as a column with this name.
See Also
--------
read_lines
Examples
--------
>>> pl.scan_lines(b"Hello\nworld").collect()
shape: (2, 1)
┌───────┐
│ line │
│ --- │
│ str │
╞═══════╡
│ Hello │
│ world │
└───────┘
polars.scan_ndjson
Python Library Documentation: function scan_ndjson in module polars.io.ndjson
scan_ndjson(source: 'str | Path | IO[str] | IO[bytes] | bytes | list[str] | list[Path] | list[IO[str]] | list[IO[bytes]]', *, schema: 'SchemaDefinition | None' = None, schema_overrides: 'SchemaDefinition | None' = None, infer_schema_length: 'int | None' = 100, batch_size: 'int | None' = 1024, n_rows: 'int | None' = None, low_memory: 'bool' = False, rechunk: 'bool' = False, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, ignore_errors: 'bool' = False, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, file_cache_ttl: 'int | None' = None, include_file_paths: 'str | None' = None) -> 'LazyFrame'
Lazily read from a newline delimited JSON file or multiple files via glob patterns.
This allows the query optimizer to push down predicates and projections to the scan
level, thereby potentially reducing memory overhead.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
Parameters
----------
source
Path to a file.
schema : Sequence of str, (str,DataType) pairs, or a {str:DataType,} dict
The DataFrame schema may be declared in several ways:
* As a dict of {name:type} pairs; if type is None, it will be auto-inferred.
* As a list of column names; in this case types are automatically inferred.
* As a list of (name,type) pairs; this is equivalent to the dictionary form.
If you supply a list of column names that does not match the names in the
underlying data, the names given here will overwrite them. The number
of names given in the schema should match the underlying data dimensions.
schema_overrides : dict, default None
Support type specification or override of one or more columns; note that
any dtypes inferred from the schema param will be overridden.
infer_schema_length
The maximum number of rows to scan for schema inference.
If set to `None`, the full data may be scanned *(this is slow)*.
batch_size
Number of rows to read in each batch.
n_rows
Stop reading from JSON file after reading `n_rows`.
low_memory
Reduce memory pressure at the expense of performance.
rechunk
Reallocate to contiguous memory when all chunks/ files are parsed.
row_index_name
If not None, this will insert a row index column with give name into the
DataFrame
row_index_offset
Offset to start the row index column (only use if the name is set)
ignore_errors
Return `Null` if parsing fails because of schema mismatches.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
file_cache_ttl
Amount of time to keep downloaded cloud files since their last access time,
in seconds. Uses the `POLARS_FILE_CACHE_TTL` environment variable
(which defaults to 1 hour) if not given.
.. deprecated:: 1.39.0
File cache is no longer supported.
include_file_paths
Include the path of the source file(s) as a column with this name.
polars.scan_parquet
Python Library Documentation: function scan_parquet in module polars.io.parquet.functions
scan_parquet(source: 'FileSource', *, n_rows: 'int | None' = None, row_index_name: 'str | None' = None, row_index_offset: 'int' = 0, parallel: 'ParallelStrategy' = 'auto', use_statistics: 'bool' = True, hive_partitioning: 'bool | None' = None, glob: 'bool' = True, hidden_file_prefix: 'str | Sequence[str] | None' = None, schema: 'SchemaDict | None' = None, hive_schema: 'SchemaDict | None' = None, try_parse_hive_dates: 'bool' = True, rechunk: 'bool' = False, low_memory: 'bool' = False, cache: 'bool' = True, storage_options: 'StorageOptionsDict | None' = None, credential_provider: "CredentialProviderFunction | Literal['auto'] | None" = 'auto', retries: 'int | None' = None, include_file_paths: 'str | None' = None, missing_columns: "Literal['insert', 'raise']" = 'raise', allow_missing_columns: 'bool | None' = None, extra_columns: "Literal['ignore', 'raise']" = 'raise', cast_options: 'ScanCastOptions | None' = None, _column_mapping: 'ColumnMapping | None' = None, _default_values: 'DefaultFieldValues | None' = None, _deletion_files: 'DeletionFiles | None' = None, _table_statistics: 'DataFrame | None' = None, _row_count: 'tuple[int, int] | None' = None) -> 'LazyFrame'
Lazily read from a local or cloud-hosted parquet file (or files).
This function allows the query optimizer to push down predicates and projections to
the scan level, typically increasing performance and reducing memory overhead.
.. versionchanged:: 0.20.4
* The `row_count_name` parameter was renamed `row_index_name`.
* The `row_count_offset` parameter was renamed `row_index_offset`.
.. versionchanged:: 1.30.0
* The `allow_missing_columns` is deprecated in favor of `missing_columns`.
Parameters
----------
source
Path(s) to a file or directory
When needing to authenticate for scanning cloud locations, see the
`storage_options` parameter.
n_rows
Stop reading from parquet file after reading `n_rows`.
row_index_name
If not None, this will insert a row index column with the given name into the
DataFrame
row_index_offset
Offset to start the row index column (only used if the name is set)
parallel : {'auto', 'columns', 'row_groups', 'prefiltered', 'none'}
This determines the direction and strategy of parallelism. 'auto' will
try to determine the optimal direction.
The `prefiltered` strategy first evaluates the pushed-down predicates in
parallel and determines a mask of which rows to read. Then, it
parallelizes over both the columns and the row groups while filtering
out rows that do not need to be read. This can provide significant
speedups for large files (i.e. many row-groups) with a predicate that
filters clustered rows or filters heavily. In other cases,
`prefiltered` may slow down the scan compared other strategies.
The `prefiltered` settings falls back to `auto` if no predicate is
given.
.. warning::
The `prefiltered` strategy is considered **unstable**. It may be
changed at any point without it being considered a breaking change.
use_statistics
Use statistics in the parquet to determine if pages
can be skipped from reading.
hive_partitioning
Infer statistics and schema from hive partitioned URL and use them
to prune reads.
glob
Expand path given via globbing rules.
hidden_file_prefix
Skip reading files whose names begin with the specified prefixes.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
schema
Specify the datatypes of the columns. The datatypes must match the
datatypes in the file(s). If there are extra columns that are not in the
file(s), consider also passing `missing_columns='insert'`.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
hive_schema
The column names and data types of the columns by which the data is partitioned.
If set to `None` (default), the schema of the Hive partitions is inferred.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
try_parse_hive_dates
Whether to try parsing hive values as date/datetime types.
rechunk
In case of reading multiple files via a glob pattern rechunk the final DataFrame
into contiguous memory chunks.
low_memory
Reduce memory pressure at the expense of performance.
cache
Cache the result after reading.
storage_options
Options that indicate how to connect to a cloud provider.
The cloud providers currently supported are AWS, GCP, and Azure.
See supported keys here:
* `aws <https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html>`_
* `gcp <https://docs.rs/object_store/latest/object_store/gcp/enum.GoogleConfigKey.html>`_
* `azure <https://docs.rs/object_store/latest/object_store/azure/enum.AzureConfigKey.html>`_
* Hugging Face (`hf://`): Accepts an API key under the `token` parameter: `{'token': '...'}`, or by setting the `HF_TOKEN` environment variable.
If `storage_options` is not provided, Polars will try to infer the information
from environment variables.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
retries
Number of retries if accessing a cloud instance fails.
.. deprecated:: 1.37.1
Pass {"max_retries": n} via `storage_options` instead.
include_file_paths
Include the path of the source file(s) as a column with this name.
missing_columns
Configuration for behavior when columns defined in the schema
are missing from the data:
* `insert`: Inserts the missing columns using NULLs as the row values.
* `raise`: Raises an error.
allow_missing_columns
When reading a list of parquet files, if a column existing in the first
file cannot be found in subsequent files, the default behavior is to
raise an error. However, if `allow_missing_columns` is set to
`True`, a full-NULL column is returned instead of erroring for the files
that do not contain the column.
.. deprecated:: 1.30.0
Use the parameter `missing_columns` instead and pass one of
`('insert', 'raise')`.
extra_columns
Configuration for behavior when extra columns outside of the
defined schema are encountered in the data:
* `ignore`: Silently ignores.
* `raise`: Raises an error.
cast_options
Configuration for column type-casting during scans. Useful for datasets
containing files that have differing schemas.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
See Also
--------
read_parquet
scan_pyarrow_dataset
Examples
--------
Scan a local Parquet file.
>>> pl.scan_parquet("path/to/file.parquet") # doctest: +SKIP
Scan a file on AWS S3.
>>> source = "s3://bucket/*.parquet"
>>> pl.scan_parquet(source) # doctest: +SKIP
>>> storage_options = {
... "aws_access_key_id": "<secret>",
... "aws_secret_access_key": "<secret>",
... "aws_region": "us-east-1",
... }
>>> pl.scan_parquet(source, storage_options=storage_options) # doctest: +SKIP
polars.scan_pyarrow_dataset
Python Library Documentation: function scan_pyarrow_dataset in module polars.io.pyarrow_dataset.functions
scan_pyarrow_dataset(source: 'pa.dataset.Dataset', *, allow_pyarrow_filter: 'bool' = True, batch_size: 'int | None' = None) -> 'LazyFrame'
Scan a pyarrow dataset.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
This can be useful to connect to cloud or partitioned datasets.
Parameters
----------
source
Pyarrow dataset to scan.
allow_pyarrow_filter
Allow predicates to be pushed down to pyarrow. This can lead to different
results if comparisons are done with null values as pyarrow handles this
different than polars does.
batch_size
The maximum row count for scanned pyarrow record batches.
Warnings
--------
Don't use this if you accept untrusted user inputs. Predicates will be evaluated
with python 'eval'. There is sanitation in place, but it is a possible attack
vector.
This method can only can push down predicates that are allowed by PyArrow
(e.g. not the full Polars API).
If :func:`scan_parquet` works for your source, you should use that instead.
Notes
-----
When using partitioning, the appropriate `partitioning` option must be set on
`pyarrow.dataset.dataset` before passing to Polars or the partitioned-on column(s)
may not get passed to Polars.
Examples
--------
>>> import pyarrow.dataset as ds
>>> dset = ds.dataset("s3://my-partitioned-folder/", format="ipc") # doctest: +SKIP
>>> (
... pl.scan_pyarrow_dataset(dset)
... .filter("bools")
... .select("bools", "floats", "date")
... .collect()
... ) # doctest: +SKIP
shape: (1, 3)
┌───────┬────────┬────────────┐
│ bools ┆ floats ┆ date │
│ --- ┆ --- ┆ --- │
│ bool ┆ f64 ┆ date │
╞═══════╪════════╪════════════╡
│ true ┆ 2.0 ┆ 1970-05-04 │
└───────┴────────┴────────────┘
polars.schema
Python Library Documentation: module polars.schema in polars
NAME
polars.schema
CLASSES
collections.OrderedDict(builtins.dict)
Schema
class Schema(collections.OrderedDict)
| Schema(schema: 'Mapping[str, SchemaInitDataType] | Iterable[tuple[str, SchemaInitDataType] | ArrowSchemaExportable] | ArrowSchemaExportable | None' = None, *, check_dtypes: 'bool' = True) -> 'None'
|
| Ordered mapping of column names to their data type.
|
| Parameters
| ----------
| schema
| The schema definition given by column names and their associated
| Polars data type. Accepts a mapping, or an iterable of tuples, or any
| object implementing the `__arrow_c_schema__` PyCapsule interface
| (e.g. pyarrow schemas).
|
| Examples
| --------
| Define a schema by passing instantiated data types.
|
| >>> schema = pl.Schema(
| ... {
| ... "foo": pl.String(),
| ... "bar": pl.Duration("us"),
| ... "baz": pl.Array(pl.Int8, 4),
| ... }
| ... )
| >>> schema
| Schema({'foo': String, 'bar': Duration(time_unit='us'), 'baz': Array(Int8, shape=(4,))})
|
| Access the data type associated with a specific column name.
|
| >>> schema["baz"]
| Array(Int8, shape=(4,))
|
| Access various schema properties using the `names`, `dtypes`, and `len` methods.
|
| >>> schema.names()
| ['foo', 'bar', 'baz']
| >>> schema.dtypes()
| [String, Duration(time_unit='us'), Array(Int8, shape=(4,))]
| >>> schema.len()
| 3
|
| Import a pyarrow schema.
|
| >>> import pyarrow as pa
| >>> pl.Schema(pa.schema([pa.field("x", pa.int32())]))
| Schema({'x': Int32})
|
| Export a schema to pyarrow.
|
| >>> pa.schema(pl.Schema({"x": pl.Int32}))
| x: int32
|
| Method resolution order:
| Schema
| collections.OrderedDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __arrow_c_schema__(self) -> 'object'
| Export a Schema via the Arrow PyCapsule Interface.
|
| https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html
|
| __eq__(self, other: 'object') -> 'bool'
| Return self==value.
|
| __init__(self, schema: 'Mapping[str, SchemaInitDataType] | Iterable[tuple[str, SchemaInitDataType] | ArrowSchemaExportable] | ArrowSchemaExportable | None' = None, *, check_dtypes: 'bool' = True) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __ne__(self, other: 'object') -> 'bool'
| Return self!=value.
|
| __setitem__(self, name: 'str', dtype: 'DataType | DataTypeClass | PythonDataType') -> 'None'
| Set self[key] to value.
|
| contains_dtype(self, dtype: 'DataType', *, recursive: 'bool') -> 'bool'
| Check if the schema contains the given data type.
|
| Parameters
| ----------
| dtype
| The data type to search for.
| recursive
| If False, only check top-level column dtypes.
| If True, also search within nested types (List, Array, Struct).
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.Int64(), "y": pl.List(pl.Float64)})
| >>> s.contains_dtype(pl.Int64, recursive=False)
| True
| >>> s.contains_dtype(pl.Float64, recursive=False)
| False
| >>> s.contains_dtype(pl.Float64, recursive=True)
| True
|
| dtypes(self) -> 'list[DataType]'
| Get the data types of the schema.
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.UInt8(), "y": pl.List(pl.UInt8)})
| >>> s.dtypes()
| [UInt8, List(UInt8)]
|
| len(self) -> 'int'
| Get the number of schema entries.
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.Int32(), "y": pl.List(pl.String)})
| >>> s.len()
| 2
| >>> len(s)
| 2
|
| names(self) -> 'list[str]'
| Get the column names of the schema.
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.Float64(), "y": pl.Datetime(time_zone="UTC")})
| >>> s.names()
| ['x', 'y']
|
| to_arrow(self, *, compat_level: 'CompatLevel | None' = None) -> 'pa.Schema'
| Convert the schema to a pyarrow schema.
|
| Parameters
| ----------
| compat_level
| Use a specific compatibility level
| when exporting Polars' internal data types.
|
| Examples
| --------
| >>> pl.Schema({"x": pl.String}).to_arrow()
| x: string_view
|
| to_frame(self, *, eager: 'bool' = True) -> 'DataFrame | LazyFrame'
| Create an empty DataFrame (or LazyFrame) from this Schema.
|
| Parameters
| ----------
| eager
| If True, create a DataFrame; otherwise, create a LazyFrame.
|
| Examples
| --------
| >>> s = pl.Schema({"x": pl.Int32(), "y": pl.String()})
| >>> s.to_frame()
| shape: (0, 2)
| ┌─────┬─────┐
| │ x ┆ y │
| │ --- ┆ --- │
| │ i32 ┆ str │
| ╞═════╪═════╡
| └─────┴─────┘
| >>> s.to_frame(eager=False) # doctest: +IGNORE_RESULT
| <LazyFrame at 0x11BC0AD80>
|
| to_python(self) -> 'dict[str, type]'
| Return a dictionary of column names and Python types.
|
| Examples
| --------
| >>> s = pl.Schema(
| ... {
| ... "x": pl.Int8(),
| ... "y": pl.String(),
| ... "z": pl.Duration("us"),
| ... }
| ... )
| >>> s.to_python()
| {'x': <class 'int'>, 'y': <class 'str'>, 'z': <class 'datetime.timedelta'>}
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __hash__ = None
|
| __orig_bases__ = (collections.OrderedDict[str, polars.datatypes.classe...
|
| ----------------------------------------------------------------------
| Methods inherited from collections.OrderedDict:
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __ge__(self, value, /)
| Return self>=value.
|
| __gt__(self, value, /)
| Return self>value.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __lt__(self, value, /)
| Return self<value.
|
| __or__(self, value, /)
| Return self|value.
|
| __reduce__(...)
| Return state information for pickling
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(...)
| od.__reversed__() <==> reversed(od)
|
| __ror__(self, value, /)
| Return value|self.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| od.clear() -> None. Remove all items from od.
|
| copy(...)
| od.copy() -> a shallow copy of od
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| move_to_end(self, /, key, last=True)
| Move an existing element to the end (or beginning if last is false).
|
| Raise KeyError if the element does not exist.
|
| pop(...)
| od.pop(key[,default]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /, last=True)
| Remove and return a (key, value) pair from the dictionary.
|
| Pairs are returned in LIFO order if last is true or FIFO order if false.
|
| setdefault(self, /, key, default=None)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.OrderedDict:
|
| fromkeys(iterable, value=None)
| Create a new ordered dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from collections.OrderedDict:
|
| __dict__
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __len__(self, /)
| Return len(self).
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
DATA
__all__ = ['Schema']
__annotations__ = {'SchemaInitDataType': 'TypeAlias'}
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\schema.py
polars.select
Python Library Documentation: function select in module polars.functions.lazy
select(*exprs: 'IntoExpr | Iterable[IntoExpr]', eager: 'bool' = True, **named_exprs: 'IntoExpr') -> 'DataFrame | LazyFrame'
Run polars expressions without a context.
This is syntactic sugar for running `df.select` on an empty DataFrame
(or LazyFrame if eager=False).
Parameters
----------
*exprs
Column(s) to select, specified as positional arguments.
Accepts expression input. Strings are parsed as column names,
other non-expression inputs are parsed as literals.
eager
Evaluate immediately and return a `DataFrame` (default); if set to `False`,
return a `LazyFrame` instead.
**named_exprs
Additional columns to select, specified as keyword arguments.
The columns will be renamed to the keyword used.
Returns
-------
DataFrame or LazyFrame
Examples
--------
>>> foo = pl.Series("foo", [1, 2, 3])
>>> bar = pl.Series("bar", [3, 2, 1])
>>> pl.select(min=pl.min_horizontal(foo, bar))
shape: (3, 1)
┌─────┐
│ min │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
│ 1 │
└─────┘
>>> pl.select(pl.int_range(0, 100_000, 2).alias("n"), eager=False).filter(
... pl.col("n") % 22_500 == 0
... ).collect()
shape: (5, 1)
┌───────┐
│ n │
│ --- │
│ i64 │
╞═══════╡
│ 0 │
│ 22500 │
│ 45000 │
│ 67500 │
│ 90000 │
└───────┘
polars.selectors
Python Library Documentation: module polars.selectors in polars
NAME
polars.selectors
CLASSES
polars.expr.expr.Expr(builtins.object)
Selector
class Selector(polars.expr.expr.Expr)
| Base column selector expression/proxy.
|
| Method resolution order:
| Selector
| polars.expr.expr.Expr
| builtins.object
|
| Methods defined here:
|
| __add__(self, other: 'Any') -> 'Expr'
|
| __and__(self, other: 'Any') -> 'Selector | Expr'
|
| __getstate__(self) -> 'bytes'
| Helper for pickle.
|
| __hash__(self) -> 'int'
| Return hash(self).
|
| __invert__(cls) -> 'Selector'
| Invert the selector.
|
| __or__(self, other: 'Any') -> 'Selector | Expr'
| Return self|value.
|
| __radd__(self, other: 'Any') -> 'Expr'
|
| __rand__(self, other: 'Any') -> 'Expr'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __ror__(self, other: 'Any') -> 'Expr'
| Return value|self.
|
| __rsub__(self, other: 'Any') -> 'NoReturn'
|
| __rxor__(self, other: 'Any') -> 'Expr'
|
| __setstate__(self, state: 'bytes') -> 'None'
|
| __sub__(self, other: 'Any') -> 'Selector | Expr'
|
| __xor__(self, other: 'Any') -> 'Selector | Expr'
|
| as_expr(self) -> 'Expr'
| Materialize the `selector` as a normal expression.
|
| This ensures that the operators `|`, `&`, `~` and `-`
| are applied on the data and not on the selector sets.
|
| Examples
| --------
| >>> import polars.selectors as cs
| >>> df = pl.DataFrame(
| ... {
| ... "colx": ["aa", "bb", "cc"],
| ... "coly": [True, False, True],
| ... "colz": [1, 2, 3],
| ... }
| ... )
|
| Inverting the boolean selector will choose the non-boolean columns:
|
| >>> df.select(~cs.boolean())
| shape: (3, 2)
| ┌──────┬──────┐
| │ colx ┆ colz │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞══════╪══════╡
| │ aa ┆ 1 │
| │ bb ┆ 2 │
| │ cc ┆ 3 │
| └──────┴──────┘
|
| To invert the *values* in the selected boolean columns, we need to
| materialize the selector as a standard expression instead:
|
| >>> df.select(~cs.boolean().as_expr())
| shape: (3, 1)
| ┌───────┐
| │ coly │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ true │
| │ false │
| └───────┘
|
| exclude(self, columns: 'str | PolarsDataType | Collection[str | PolarsDataType]', *more_columns: 'str | PolarsDataType') -> 'Selector'
| Exclude columns from a multi-column expression.
|
| Only works after a wildcard or regex column selection, and you cannot provide
| both string column names *and* dtypes (you may prefer to use selectors instead).
|
| Parameters
| ----------
| columns
| The name or datatype of the column(s) to exclude. Accepts regular expression
| input. Regular expressions should start with `^` and end with `$`.
| *more_columns
| Additional names or datatypes of columns to exclude, specified as positional
| arguments.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_pyselector': 'PySelector'}
|
| ----------------------------------------------------------------------
| Methods inherited from polars.expr.expr.Expr:
|
| __abs__(self) -> 'Expr'
|
| __array_ufunc__(self, ufunc: 'Callable[..., Any]', method: 'str_', *inputs: 'Any', **kwargs: 'Any') -> 'Expr'
| Numpy universal functions.
|
| __bool__(self) -> 'NoReturn'
|
| __eq__(self, other: 'IntoExpr') -> 'Expr'
| Return self==value.
|
| __floordiv__(self, other: 'IntoExpr') -> 'Expr'
|
| __ge__(self, other: 'IntoExpr') -> 'Expr'
| Return self>=value.
|
| __gt__(self, other: 'IntoExpr') -> 'Expr'
| Return self>value.
|
| __le__(self, other: 'IntoExpr') -> 'Expr'
| Return self<=value.
|
| __lt__(self, other: 'IntoExpr') -> 'Expr'
| Return self<value.
|
| __mod__(self, other: 'IntoExpr') -> 'Expr'
|
| __mul__(self, other: 'IntoExpr') -> 'Expr'
|
| __ne__(self, other: 'IntoExpr') -> 'Expr'
| Return self!=value.
|
| __neg__(self) -> 'Expr'
|
| __pos__(self) -> 'Expr'
|
| __pow__(self, exponent: 'IntoExprColumn | int | float') -> 'Expr'
|
| __rfloordiv__(self, other: 'IntoExpr') -> 'Expr'
|
| __rmod__(self, other: 'IntoExpr') -> 'Expr'
|
| __rmul__(self, other: 'IntoExpr') -> 'Expr'
|
| __rpow__(self, base: 'IntoExprColumn | int | float') -> 'Expr'
|
| __rtruediv__(self, other: 'IntoExpr') -> 'Expr'
|
| __str__(self) -> 'str_'
| Return str(self).
|
| __truediv__(self, other: 'IntoExpr') -> 'Expr'
|
| abs(self) -> 'Expr'
| Compute absolute values.
|
| Same as `abs(expr)`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [-1.0, 0.0, 1.0, 2.0],
| ... }
| ... )
| >>> df.select(pl.col("A").abs())
| shape: (4, 1)
| ┌─────┐
| │ A │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| │ 0.0 │
| │ 1.0 │
| │ 2.0 │
| └─────┘
|
| add(self, other: 'Any') -> 'Expr'
| Method equivalent of addition operator `expr + other`.
|
| Parameters
| ----------
| other
| numeric or string value; accepts expression input.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 3, 4, 5]})
| >>> df.with_columns(
| ... pl.col("x").add(2).alias("x+int"),
| ... pl.col("x").add(pl.col("x").cum_prod()).alias("x+expr"),
| ... )
| shape: (5, 3)
| ┌─────┬───────┬────────┐
| │ x ┆ x+int ┆ x+expr │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═══════╪════════╡
| │ 1 ┆ 3 ┆ 2 │
| │ 2 ┆ 4 ┆ 4 │
| │ 3 ┆ 5 ┆ 9 │
| │ 4 ┆ 6 ┆ 28 │
| │ 5 ┆ 7 ┆ 125 │
| └─────┴───────┴────────┘
|
| >>> df = pl.DataFrame(
| ... {"x": ["a", "d", "g"], "y": ["b", "e", "h"], "z": ["c", "f", "i"]}
| ... )
| >>> df.with_columns(pl.col("x").add(pl.col("y")).add(pl.col("z")).alias("xyz"))
| shape: (3, 4)
| ┌─────┬─────┬─────┬─────┐
| │ x ┆ y ┆ z ┆ xyz │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ str ┆ str ┆ str │
| ╞═════╪═════╪═════╪═════╡
| │ a ┆ b ┆ c ┆ abc │
| │ d ┆ e ┆ f ┆ def │
| │ g ┆ h ┆ i ┆ ghi │
| └─────┴─────┴─────┴─────┘
|
| agg_groups(self) -> 'Expr'
| Get the group indexes of the group by operation.
|
| .. deprecated:: 1.35
| use `df.with_row_index().group_by(...).agg(pl.col('index'))` instead.
| This method will be removed in Polars 2.0.
|
| Should be used in aggregation context only.
|
| Examples
| --------
| >>> import warnings
| >>> warnings.filterwarnings("ignore", category=DeprecationWarning)
| >>> df = pl.DataFrame(
| ... {
| ... "group": [
| ... "one",
| ... "one",
| ... "one",
| ... "two",
| ... "two",
| ... "two",
| ... ],
| ... "value": [94, 95, 96, 97, 97, 99],
| ... }
| ... )
| >>> df.group_by("group", maintain_order=True).agg(pl.col("value").agg_groups())
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ list[u32] │
| ╞═══════╪═══════════╡
| │ one ┆ [0, 1, 2] │
| │ two ┆ [3, 4, 5] │
| └───────┴───────────┘
|
| New recommended approach:
| >>> (
| ... df.with_row_index()
| ... .group_by("group", maintain_order=True)
| ... .agg(pl.col("index"))
| ... )
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ index │
| │ --- ┆ --- │
| │ str ┆ list[u32] │
| ╞═══════╪═══════════╡
| │ one ┆ [0, 1, 2] │
| │ two ┆ [3, 4, 5] │
| └───────┴───────────┘
|
| alias(self, name: 'str_') -> 'Expr'
| Rename the expression.
|
| Parameters
| ----------
| name
| The new name.
|
| See Also
| --------
| name.map
| name.prefix
| name.suffix
|
| Examples
| --------
| Rename an expression to avoid overwriting an existing column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["x", "y", "z"],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("a") + 10,
| ... pl.col("b").str.to_uppercase().alias("c"),
| ... )
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ str │
| ╞═════╪═════╪═════╡
| │ 11 ┆ x ┆ X │
| │ 12 ┆ y ┆ Y │
| │ 13 ┆ z ┆ Z │
| └─────┴─────┴─────┘
|
| Overwrite the default name of literal columns to prevent errors due to duplicate
| column names.
|
| >>> df.with_columns(
| ... pl.lit(True).alias("c"),
| ... pl.lit(4.0).alias("d"),
| ... )
| shape: (3, 4)
| ┌─────┬─────┬──────┬─────┐
| │ a ┆ b ┆ c ┆ d │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ bool ┆ f64 │
| ╞═════╪═════╪══════╪═════╡
| │ 1 ┆ x ┆ true ┆ 4.0 │
| │ 2 ┆ y ┆ true ┆ 4.0 │
| │ 3 ┆ z ┆ true ┆ 4.0 │
| └─────┴─────┴──────┴─────┘
|
| all(self, *, ignore_nulls: 'bool' = True) -> 'Expr'
| Return whether all values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| .. note::
| This method is not to be confused with the function :func:`polars.all`,
| which can be used to select all columns.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `True`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `False` values,
| the output is null.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [True, True],
| ... "b": [False, True],
| ... "c": [None, True],
| ... }
| ... )
| >>> df.select(pl.col("*").all())
| shape: (1, 3)
| ┌──────┬───────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪══════╡
| │ true ┆ false ┆ true │
| └──────┴───────┴──────┘
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> df.select(pl.col("*").all(ignore_nulls=False))
| shape: (1, 3)
| ┌──────┬───────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪══════╡
| │ true ┆ false ┆ null │
| └──────┴───────┴──────┘
|
| and_(self, *others: 'Any') -> 'Expr'
| Method equivalent of bitwise "and" operator `expr & other & ...`.
|
| This has the effect of combining logical boolean expressions,
| but operates bitwise on integers.
|
| Parameters
| ----------
| *others
| One or more integer or boolean expressions to evaluate/combine.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5, 6, 7, 4, 8],
| ... "y": [1.5, 2.5, 1.0, 4.0, -5.75],
| ... "z": [-9, 2, -1, 4, 8],
| ... }
| ... )
|
| Combine logical "and" conditions:
|
| >>> df.select(
| ... (pl.col("x") >= pl.col("z"))
| ... .and_(
| ... pl.col("y") >= pl.col("z"),
| ... pl.col("y") == pl.col("y"),
| ... pl.col("z") <= pl.col("x"),
| ... pl.col("y") != pl.col("x"),
| ... )
| ... .alias("all")
| ... )
| shape: (5, 1)
| ┌───────┐
| │ all │
| │ --- │
| │ bool │
| ╞═══════╡
| │ true │
| │ true │
| │ true │
| │ false │
| │ false │
| └───────┘
|
| Bitwise "and" operation on integer columns:
|
| >>> df.select("x", "z", x_and_z=pl.col("x").and_(pl.col("z")))
| shape: (5, 3)
| ┌─────┬─────┬─────────┐
| │ x ┆ z ┆ x_and_z │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════════╡
| │ 5 ┆ -9 ┆ 5 │
| │ 6 ┆ 2 ┆ 2 │
| │ 7 ┆ -1 ┆ 7 │
| │ 4 ┆ 4 ┆ 4 │
| │ 8 ┆ 8 ┆ 8 │
| └─────┴─────┴─────────┘
|
| any(self, *, ignore_nulls: 'bool' = True) -> 'Expr'
| Return whether any of the values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `False`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `True` values,
| the output is null.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [True, False],
| ... "b": [False, False],
| ... "c": [None, False],
| ... }
| ... )
| >>> df.select(pl.col("*").any())
| shape: (1, 3)
| ┌──────┬───────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪═══════╡
| │ true ┆ false ┆ false │
| └──────┴───────┴───────┘
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> df.select(pl.col("*").any(ignore_nulls=False))
| shape: (1, 3)
| ┌──────┬───────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪═══════╪══════╡
| │ true ┆ false ┆ null │
| └──────┴───────┴──────┘
|
| append(self, other: 'IntoExpr', *, upcast: 'bool' = True) -> 'Expr'
| Append expressions.
|
| This is done by adding the chunks of `other` to this `Series`.
|
| Parameters
| ----------
| other
| Expression to append.
| upcast
| Cast both `Series` to the same supertype.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [8, 9, 10],
| ... "b": [None, 4, 4],
| ... }
| ... )
| >>> df.select(pl.all().head(1).append(pl.all().tail(1)))
| shape: (2, 2)
| ┌─────┬──────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════╡
| │ 8 ┆ null │
| │ 10 ┆ 4 │
| └─────┴──────┘
|
| approx_n_unique(self) -> 'Expr'
| Approximate count of unique values.
|
| This is done using the HyperLogLog++ algorithm for cardinality estimation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [1, 1, 2]})
| >>> df.select(pl.col("n").approx_n_unique())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 2 │
| └─────┘
| >>> df = pl.DataFrame({"n": range(1000)})
| >>> df.select(
| ... exact=pl.col("n").n_unique(),
| ... approx=pl.col("n").approx_n_unique(),
| ... ) # doctest: +SKIP
| shape: (1, 2)
| ┌───────┬────────┐
| │ exact ┆ approx │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═══════╪════════╡
| │ 1000 ┆ 1005 │
| └───────┴────────┘
|
| arccos(self) -> 'Expr'
| Compute the element-wise value for the inverse cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`, measuring an angle in radians.
|
| Notes
| -----
| To convert the result from radians to degrees,
| call :meth:`.degrees() <polars.Expr.degrees>`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, 0.5, 0]})
| >>> df.select(pl.col("a").arccos())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.0 │
| │ 1.047198 │
| │ 1.570796 │
| └──────────┘
| >>> df.select(pl.col("a").arccos().degrees())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 0.0 │
| │ 60.0 │
| │ 90.0 │
| └──────┘
|
| arccosh(self) -> 'Expr'
| Compute the element-wise value for the inverse hyperbolic cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").arccosh())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| └─────┘
|
| arcsin(self) -> 'Expr'
| Compute the element-wise value for the inverse sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`, measuring an angle in radians.
|
| Notes
| -----
| To convert the result from radians to degrees,
| call :meth:`.degrees() <polars.Expr.degrees>`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, 0.5, 0]})
| >>> df.select(pl.col("a").arcsin())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.570796 │
| │ 0.523599 │
| │ 0.0 │
| └──────────┘
| >>> df.select(pl.col("a").arcsin().degrees())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 90.0 │
| │ 30.0 │
| │ 0.0 │
| └──────┘
|
| arcsinh(self) -> 'Expr'
| Compute the element-wise value for the inverse hyperbolic sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").arcsinh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.881374 │
| └──────────┘
|
| arctan(self) -> 'Expr'
| Compute the element-wise value for the inverse tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`, measuring an angle in radians.
|
| Notes
| -----
| To convert the result from radians to degrees,
| call :meth:`.degrees() <polars.Expr.degrees>`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [float("Inf"), 1, 0]})
| >>> df.select(pl.col("a").arctan())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.570796 │
| │ 0.785398 │
| │ 0.0 │
| └──────────┘
| >>> df.select(pl.col("a").arctan().degrees())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 90.0 │
| │ 45.0 │
| │ 0.0 │
| └──────┘
|
| arctanh(self) -> 'Expr'
| Compute the element-wise value for the inverse hyperbolic tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").arctanh())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ inf │
| └─────┘
|
| arg_max(self) -> 'Expr'
| Get the index of the maximal value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [20, 10, 30],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_max())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 2 │
| └─────┘
|
| arg_min(self) -> 'Expr'
| Get the index of the minimal value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [20, 10, 30],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_min())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| └─────┘
|
| arg_sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Expr'
| Get the index values that would sort this column.
|
| Parameters
| ----------
| descending
| Sort in descending (descending) order.
| nulls_last
| Place null values last instead of first.
|
| Returns
| -------
| Expr
| Expression of data type :class:`UInt32`.
|
| See Also
| --------
| Expr.gather: Take values by index.
| Expr.rank : Get the rank of each row.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [20, 10, 30],
| ... "b": [1, 2, 3],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_sort())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| │ 0 │
| │ 2 │
| └─────┘
|
| Use gather to apply the arg sort to other columns.
|
| >>> df.select(pl.col("b").gather(pl.col("a").arg_sort()))
| shape: (3, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 1 │
| │ 3 │
| └─────┘
|
| arg_true(self) -> 'Expr'
| Return indices where expression evaluates `True`.
|
| .. warning::
| Modifies number of rows returned, so will fail in combination with other
| expressions. Use as only expression in `select` / `with_columns`.
|
| See Also
| --------
| Series.arg_true : Return indices where Series is True
| polars.arg_where
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 1]})
| >>> df.select((pl.col("a") == 1).arg_true())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 0 │
| │ 1 │
| │ 3 │
| └─────┘
|
| arg_unique(self) -> 'Expr'
| Get index of first unique value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [8, 9, 10],
| ... "b": [None, 4, 4],
| ... }
| ... )
| >>> df.select(pl.col("a").arg_unique())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 0 │
| │ 1 │
| │ 2 │
| └─────┘
| >>> df.select(pl.col("b").arg_unique())
| shape: (2, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 0 │
| │ 1 │
| └─────┘
|
| backward_fill(self, limit: 'int | None' = None) -> 'Expr'
| Fill missing values with the next non-null value.
|
| This is an alias of `.fill_null(strategy="backward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to backward fill.
|
| See Also
| --------
| fill_null
| forward_fill
| shift
|
| bitwise_and(self) -> 'Expr'
| Perform an aggregation of bitwise ANDs.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-1, 0, 1]})
| >>> df.select(pl.col("n").bitwise_and())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 0 │
| └─────┘
| >>> df = pl.DataFrame(
| ... {"grouper": ["a", "a", "a", "b", "b"], "n": [-1, 0, 1, -1, 1]}
| ... )
| >>> df.group_by("grouper", maintain_order=True).agg(pl.col("n").bitwise_and())
| shape: (2, 2)
| ┌─────────┬─────┐
| │ grouper ┆ n │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════╡
| │ a ┆ 0 │
| │ b ┆ 1 │
| └─────────┴─────┘
|
| bitwise_count_ones(self) -> 'Expr'
| Evaluate the number of set bits.
|
| bitwise_count_zeros(self) -> 'Expr'
| Evaluate the number of unset bits.
|
| bitwise_leading_ones(self) -> 'Expr'
| Evaluate the number most-significant set bits before seeing an unset bit.
|
| bitwise_leading_zeros(self) -> 'Expr'
| Evaluate the number most-significant unset bits before seeing a set bit.
|
| bitwise_or(self) -> 'Expr'
| Perform an aggregation of bitwise ORs.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-1, 0, 1]})
| >>> df.select(pl.col("n").bitwise_or())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ i64 │
| ╞═════╡
| │ -1 │
| └─────┘
| >>> df = pl.DataFrame(
| ... {"grouper": ["a", "a", "a", "b", "b"], "n": [-1, 0, 1, -1, 1]}
| ... )
| >>> df.group_by("grouper", maintain_order=True).agg(pl.col("n").bitwise_or())
| shape: (2, 2)
| ┌─────────┬─────┐
| │ grouper ┆ n │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════╡
| │ a ┆ -1 │
| │ b ┆ -1 │
| └─────────┴─────┘
|
| bitwise_trailing_ones(self) -> 'Expr'
| Evaluate the number least-significant set bits before seeing an unset bit.
|
| bitwise_trailing_zeros(self) -> 'Expr'
| Evaluate the number least-significant unset bits before seeing a set bit.
|
| bitwise_xor(self) -> 'Expr'
| Perform an aggregation of bitwise XORs.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-1, 0, 1]})
| >>> df.select(pl.col("n").bitwise_xor())
| shape: (1, 1)
| ┌─────┐
| │ n │
| │ --- │
| │ i64 │
| ╞═════╡
| │ -2 │
| └─────┘
| >>> df = pl.DataFrame(
| ... {"grouper": ["a", "a", "a", "b", "b"], "n": [-1, 0, 1, -1, 1]}
| ... )
| >>> df.group_by("grouper", maintain_order=True).agg(pl.col("n").bitwise_xor())
| shape: (2, 2)
| ┌─────────┬─────┐
| │ grouper ┆ n │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════════╪═════╡
| │ a ┆ -2 │
| │ b ┆ -2 │
| └─────────┴─────┘
|
| bottom_k(self, k: 'int | IntoExprColumn' = 5) -> 'Expr'
| Return the `k` smallest elements.
|
| Non-null elements are always preferred over null elements. The output is
| not guaranteed to be in any particular order, call :func:`sort` after
| this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k_by
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.select(
| ... pl.col("value").top_k().alias("top_k"),
| ... pl.col("value").bottom_k().alias("bottom_k"),
| ... )
| shape: (5, 2)
| ┌───────┬──────────┐
| │ top_k ┆ bottom_k │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪══════════╡
| │ 4 ┆ 1 │
| │ 98 ┆ 98 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| │ 99 ┆ 4 │
| └───────┴──────────┘
|
| bottom_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int | IntoExprColumn' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Expr'
| Return the elements corresponding to the `k` smallest elements of the `by` column(s).
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed `reverse`.
|
| Parameters
| ----------
| by
| Column(s) used to determine the smallest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` largest elements of the `by` column(s) (instead of the `k`
| smallest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [6, 5, 4, 3, 2, 1],
| ... "c": ["Apple", "Orange", "Apple", "Apple", "Banana", "Banana"],
| ... }
| ... )
| >>> df
| shape: (6, 3)
| ┌─────┬─────┬────────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪════════╡
| │ 1 ┆ 6 ┆ Apple │
| │ 2 ┆ 5 ┆ Orange │
| │ 3 ┆ 4 ┆ Apple │
| │ 4 ┆ 3 ┆ Apple │
| │ 5 ┆ 2 ┆ Banana │
| │ 6 ┆ 1 ┆ Banana │
| └─────┴─────┴────────┘
|
| Get the bottom 2 rows by column `a` or `b`.
|
| >>> df.select(
| ... pl.all().bottom_k_by("a", 2).name.suffix("_btm_by_a"),
| ... pl.all().bottom_k_by("b", 2).name.suffix("_btm_by_b"),
| ... )
| shape: (2, 6)
| ┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
| │ a_btm_by_a ┆ b_btm_by_a ┆ c_btm_by_a ┆ a_btm_by_b ┆ b_btm_by_b ┆ c_btm_by_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞════════════╪════════════╪════════════╪════════════╪════════════╪════════════╡
| │ 1 ┆ 6 ┆ Apple ┆ 6 ┆ 1 ┆ Banana │
| │ 2 ┆ 5 ┆ Orange ┆ 5 ┆ 2 ┆ Banana │
| └────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
|
| Get the bottom 2 rows by multiple columns with given order.
|
| >>> df.select(
| ... pl.all()
| ... .bottom_k_by(["c", "a"], 2, reverse=[False, True])
| ... .name.suffix("_by_ca"),
| ... pl.all()
| ... .bottom_k_by(["c", "b"], 2, reverse=[False, True])
| ... .name.suffix("_by_cb"),
| ... )
| shape: (2, 6)
| ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
| │ a_by_ca ┆ b_by_ca ┆ c_by_ca ┆ a_by_cb ┆ b_by_cb ┆ c_by_cb │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
| │ 4 ┆ 3 ┆ Apple ┆ 1 ┆ 6 ┆ Apple │
| │ 3 ┆ 4 ┆ Apple ┆ 3 ┆ 4 ┆ Apple │
| └─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
|
| Get the bottom 2 rows by column `a` in each group.
|
| >>> (
| ... df.group_by("c", maintain_order=True)
| ... .agg(pl.all().bottom_k_by("a", 2))
| ... .explode(pl.all().exclude("c"))
| ... )
| shape: (5, 3)
| ┌────────┬─────┬─────┐
| │ c ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ Apple ┆ 1 ┆ 6 │
| │ Apple ┆ 3 ┆ 4 │
| │ Orange ┆ 2 ┆ 5 │
| │ Banana ┆ 5 ┆ 2 │
| │ Banana ┆ 6 ┆ 1 │
| └────────┴─────┴─────┘
|
| cast(self, dtype: 'PolarsDataType | pl.DataTypeExpr | type[Any]', *, strict: 'bool' = True, wrap_numerical: 'bool' = False) -> 'Expr'
| Cast between data types.
|
| Parameters
| ----------
| dtype
| DataType to cast to.
| strict
| Raise if cast is invalid on rows after predicates are pushed down.
| If `False`, invalid casts will produce null values.
| wrap_numerical
| If True numeric casts wrap overflowing values instead of
| marking the cast as invalid.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": ["4", "5", "6"],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("a").cast(pl.Float64),
| ... pl.col("b").cast(pl.Int32),
| ... )
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ i32 │
| ╞═════╪═════╡
| │ 1.0 ┆ 4 │
| │ 2.0 ┆ 5 │
| │ 3.0 ┆ 6 │
| └─────┴─────┘
|
| cbrt(self) -> 'Expr'
| Compute the cube root of the elements.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").cbrt())
| shape: (3, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 1.259921 │
| │ 1.587401 │
| └──────────┘
|
| ceil(self) -> 'Expr'
| Rounds up to the nearest integer value.
|
| Only works on floating point Series.
|
| See Also
| --------
| floor : Round down to the nearest integer.
| round : Round to the nearest integer.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.3, 0.5, 1.0, 1.1]})
| >>> df.select(pl.col("a").ceil())
| shape: (4, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| │ 1.0 │
| │ 1.0 │
| │ 2.0 │
| └─────┘
|
| clip(self, lower_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None, upper_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None) -> 'Expr'
| Set values outside the given boundaries to the boundary value.
|
| Parameters
| ----------
| lower_bound
| Lower bound. Accepts expression input. Non-expression inputs are
| parsed as literals. Strings are parsed as column names.
| upper_bound
| Upper bound. Accepts expression input. Non-expression inputs are
| parsed as literals. Strings are parsed as column names.
|
| See Also
| --------
| when
|
| Notes
| -----
| This method only works for numeric and temporal columns. To clip other data
| types, consider writing a `when-then-otherwise` expression. See :func:`when`.
|
| Examples
| --------
| Specifying both a lower and upper bound:
|
| >>> df = pl.DataFrame({"a": [-50, 5, 50, None]})
| >>> df.with_columns(clip=pl.col("a").clip(1, 10))
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ clip │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ -50 ┆ 1 │
| │ 5 ┆ 5 │
| │ 50 ┆ 10 │
| │ null ┆ null │
| └──────┴──────┘
|
| Specifying only a single bound:
|
| >>> df.with_columns(clip=pl.col("a").clip(upper_bound=10))
| shape: (4, 2)
| ┌──────┬──────┐
| │ a ┆ clip │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪══════╡
| │ -50 ┆ -50 │
| │ 5 ┆ 5 │
| │ 50 ┆ 10 │
| │ null ┆ null │
| └──────┴──────┘
|
| Using columns as bounds:
|
| >>> df = pl.DataFrame(
| ... {"a": [-50, 5, 50, None], "low": [10, 1, 0, 0], "up": [20, 4, 3, 2]}
| ... )
| >>> df.with_columns(clip=pl.col("a").clip("low", "up"))
| shape: (4, 4)
| ┌──────┬─────┬─────┬──────┐
| │ a ┆ low ┆ up ┆ clip │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞══════╪═════╪═════╪══════╡
| │ -50 ┆ 10 ┆ 20 ┆ 10 │
| │ 5 ┆ 1 ┆ 4 ┆ 4 │
| │ 50 ┆ 0 ┆ 3 ┆ 3 │
| │ null ┆ 0 ┆ 2 ┆ null │
| └──────┴─────┴─────┴──────┘
|
| cos(self) -> 'Expr'
| Compute the element-wise value for the cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 2]})
| >>> df.select(pl.col("a").cos())
| shape: (2, 1)
| ┌────────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════════╡
| │ 1.0 │
| │ 6.1232e-17 │
| └────────────┘
| >>> df = pl.DataFrame({"a": [0.0, 90]})
| >>> df.select(pl.col("a").radians().cos())
| shape: (2, 1)
| ┌────────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════════╡
| │ 1.0 │
| │ 6.1232e-17 │
| └────────────┘
|
| cosh(self) -> 'Expr'
| Compute the element-wise value for the hyperbolic cosine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").cosh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.543081 │
| └──────────┘
|
| cot(self) -> 'Expr'
| Compute the element-wise value for the cotangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 4]})
| >>> df.select(pl.col("a").cot())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ inf │
| │ 1.0 │
| └─────┘
| >>> df = pl.DataFrame({"a": [0.0, 45]})
| >>> df.select(pl.col("a").radians().cot())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ inf │
| │ 1.0 │
| └─────┘
|
| count(self) -> 'Expr'
| Return the number of non-null elements in the column.
|
| Returns
| -------
| Expr
| Expression of data type :class:`UInt32`.
|
| See Also
| --------
| len
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [None, 4, 4]})
| >>> df.select(pl.all().count())
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 3 ┆ 2 │
| └─────┴─────┘
|
| cum_count(self, *, reverse: 'bool' = False) -> 'Expr'
| Return the cumulative count of the non-null values in the column.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": ["x", "k", None, "d"]})
| >>> df.with_columns(
| ... pl.col("a").cum_count().alias("cum_count"),
| ... pl.col("a").cum_count(reverse=True).alias("cum_count_reverse"),
| ... )
| shape: (4, 3)
| ┌──────┬───────────┬───────────────────┐
| │ a ┆ cum_count ┆ cum_count_reverse │
| │ --- ┆ --- ┆ --- │
| │ str ┆ u32 ┆ u32 │
| ╞══════╪═══════════╪═══════════════════╡
| │ x ┆ 1 ┆ 3 │
| │ k ┆ 2 ┆ 2 │
| │ null ┆ 2 ┆ 1 │
| │ d ┆ 3 ┆ 1 │
| └──────┴───────────┴───────────────────┘
|
| cum_max(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative max computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 2]})
| >>> df.with_columns(
| ... pl.col("a").cum_max().alias("cum_max"),
| ... pl.col("a").cum_max(reverse=True).alias("cum_max_reverse"),
| ... )
| shape: (3, 3)
| ┌─────┬─────────┬─────────────────┐
| │ a ┆ cum_max ┆ cum_max_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════════╪═════════════════╡
| │ 1 ┆ 1 ┆ 3 │
| │ 3 ┆ 3 ┆ 3 │
| │ 2 ┆ 3 ┆ 2 │
| └─────┴─────────┴─────────────────┘
|
|
| Null values are excluded, but can also be filled by calling
| `fill_null(strategy="forward")`.
|
| >>> df = pl.DataFrame({"values": [None, 10, None, 8, 9, None, 16, None]})
| >>> df.with_columns(
| ... pl.col("values").cum_max().alias("cum_max"),
| ... pl.col("values")
| ... .cum_max()
| ... .fill_null(strategy="forward")
| ... .alias("cum_max_all_filled"),
| ... )
| shape: (8, 3)
| ┌────────┬─────────┬────────────────────┐
| │ values ┆ cum_max ┆ cum_max_all_filled │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞════════╪═════════╪════════════════════╡
| │ null ┆ null ┆ null │
| │ 10 ┆ 10 ┆ 10 │
| │ null ┆ null ┆ 10 │
| │ 8 ┆ 10 ┆ 10 │
| │ 9 ┆ 10 ┆ 10 │
| │ null ┆ null ┆ 10 │
| │ 16 ┆ 16 ┆ 16 │
| │ null ┆ null ┆ 16 │
| └────────┴─────────┴────────────────────┘
|
| cum_min(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative min computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [3, 1, 2]})
| >>> df.with_columns(
| ... pl.col("a").cum_min().alias("cum_min"),
| ... pl.col("a").cum_min(reverse=True).alias("cum_min_reverse"),
| ... )
| shape: (3, 3)
| ┌─────┬─────────┬─────────────────┐
| │ a ┆ cum_min ┆ cum_min_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════════╪═════════════════╡
| │ 3 ┆ 3 ┆ 1 │
| │ 1 ┆ 1 ┆ 1 │
| │ 2 ┆ 1 ┆ 2 │
| └─────┴─────────┴─────────────────┘
|
| cum_prod(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative product computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.with_columns(
| ... pl.col("a").cum_prod().alias("cum_prod"),
| ... pl.col("a").cum_prod(reverse=True).alias("cum_prod_reverse"),
| ... )
| shape: (4, 3)
| ┌─────┬──────────┬──────────────────┐
| │ a ┆ cum_prod ┆ cum_prod_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪══════════╪══════════════════╡
| │ 1 ┆ 1 ┆ 24 │
| │ 2 ┆ 2 ┆ 24 │
| │ 3 ┆ 6 ┆ 12 │
| │ 4 ┆ 24 ┆ 4 │
| └─────┴──────────┴──────────────────┘
|
| cum_sum(self, *, reverse: 'bool' = False) -> 'Expr'
| Get an array with the cumulative sum computed at every element.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.with_columns(
| ... pl.col("a").cum_sum().alias("cum_sum"),
| ... pl.col("a").cum_sum(reverse=True).alias("cum_sum_reverse"),
| ... )
| shape: (4, 3)
| ┌─────┬─────────┬─────────────────┐
| │ a ┆ cum_sum ┆ cum_sum_reverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════════╪═════════════════╡
| │ 1 ┆ 1 ┆ 10 │
| │ 2 ┆ 3 ┆ 9 │
| │ 3 ┆ 6 ┆ 7 │
| │ 4 ┆ 10 ┆ 4 │
| └─────┴─────────┴─────────────────┘
|
| Null values are excluded, but can also be filled by calling
| `fill_null(strategy="forward")`.
|
| >>> df = pl.DataFrame({"values": [None, 10, None, 8, 9, None, 16, None]})
| >>> df.with_columns(
| ... pl.col("values").cum_sum().alias("value_cum_sum"),
| ... pl.col("values")
| ... .cum_sum()
| ... .fill_null(strategy="forward")
| ... .alias("value_cum_sum_all_filled"),
| ... )
| shape: (8, 3)
| ┌────────┬───────────────┬──────────────────────────┐
| │ values ┆ value_cum_sum ┆ value_cum_sum_all_filled │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞════════╪═══════════════╪══════════════════════════╡
| │ null ┆ null ┆ null │
| │ 10 ┆ 10 ┆ 10 │
| │ null ┆ null ┆ 10 │
| │ 8 ┆ 18 ┆ 18 │
| │ 9 ┆ 27 ┆ 27 │
| │ null ┆ null ┆ 27 │
| │ 16 ┆ 43 ┆ 43 │
| │ null ┆ null ┆ 43 │
| └────────┴───────────────┴──────────────────────────┘
|
| cumulative_eval(self, expr: 'Expr', *, min_samples: 'int' = 1) -> 'Expr'
| Run an expression over a sliding window that increases `1` slot every iteration.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| expr
| Expression to evaluate
| min_samples
| Number of valid values there should be in the window before the expression
| is evaluated. valid values = `length - null_count`
|
| Warnings
| --------
| This can be really slow as it can have `O(n^2)` complexity. Don't use this
| for operations that visit all elements.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1, 2, 3, 4, 5]})
| >>> df.select(
| ... [
| ... pl.col("values").cumulative_eval(
| ... pl.element().first() - pl.element().last() ** 2
| ... )
| ... ]
| ... )
| shape: (5, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 0 │
| │ -3 │
| │ -8 │
| │ -15 │
| │ -24 │
| └────────┘
|
| cut(self, breaks: 'Sequence[float]', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, include_breaks: 'bool' = False) -> 'Expr'
| Bin continuous values into discrete categories.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| breaks
| List of unique cut points.
| labels
| Names of the categories. The number of labels must be equal to the number
| of cut points plus one.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from an
| :class:`Enum` to a :class:`Struct`.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Enum` if `include_breaks` is set to
| `False` (default), otherwise an expression of data type :class:`Struct`.
|
| See Also
| --------
| qcut
|
| Examples
| --------
| Divide a column into three categories.
|
| >>> df = pl.DataFrame({"foo": [-2, -1, 0, 1, 2]})
| >>> df.with_columns(
| ... pl.col("foo").cut([-1, 1], labels=["a", "b", "c"]).alias("cut")
| ... )
| shape: (5, 2)
| ┌─────┬──────┐
| │ foo ┆ cut │
| │ --- ┆ --- │
| │ i64 ┆ enum │
| ╞═════╪══════╡
| │ -2 ┆ a │
| │ -1 ┆ a │
| │ 0 ┆ b │
| │ 1 ┆ b │
| │ 2 ┆ c │
| └─────┴──────┘
|
| Add both the category and the breakpoint.
|
| >>> df.with_columns(
| ... pl.col("foo").cut([-1, 1], include_breaks=True).alias("cut")
| ... ).unnest("cut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ enum │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| degrees(self) -> 'Expr'
| Convert from radians to degrees.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> import math
| >>> df = pl.DataFrame({"a": [x * math.pi for x in range(-4, 5)]})
| >>> df.select(pl.col("a").degrees())
| shape: (9, 1)
| ┌────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════╡
| │ -720.0 │
| │ -540.0 │
| │ -360.0 │
| │ -180.0 │
| │ 0.0 │
| │ 180.0 │
| │ 360.0 │
| │ 540.0 │
| │ 720.0 │
| └────────┘
|
| diff(self, n: 'int | IntoExpr' = 1, null_behavior: 'NullBehavior' = 'ignore') -> 'Expr'
| Calculate the first discrete difference between shifted items.
|
| Parameters
| ----------
| n
| Number of slots to shift.
| null_behavior : {'ignore', 'drop'}
| How to handle null values.
|
| Examples
| --------
| >>> df = pl.DataFrame({"int": [20, 10, 30, 25, 35]})
| >>> df.with_columns(change=pl.col("int").diff())
| shape: (5, 2)
| ┌─────┬────────┐
| │ int ┆ change │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪════════╡
| │ 20 ┆ null │
| │ 10 ┆ -10 │
| │ 30 ┆ 20 │
| │ 25 ┆ -5 │
| │ 35 ┆ 10 │
| └─────┴────────┘
|
| >>> df.with_columns(change=pl.col("int").diff(n=2))
| shape: (5, 2)
| ┌─────┬────────┐
| │ int ┆ change │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪════════╡
| │ 20 ┆ null │
| │ 10 ┆ null │
| │ 30 ┆ 10 │
| │ 25 ┆ 15 │
| │ 35 ┆ 5 │
| └─────┴────────┘
|
| >>> df.select(pl.col("int").diff(n=2, null_behavior="drop").alias("diff"))
| shape: (3, 1)
| ┌──────┐
| │ diff │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 10 │
| │ 15 │
| │ 5 │
| └──────┘
|
| dot(self, other: 'Expr | str_') -> 'Expr'
| Compute the dot/inner product between two Expressions.
|
| Parameters
| ----------
| other
| Expression to compute dot product with.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 5], "b": [2, 4, 6]})
| >>> df.select(pl.col("a").dot(pl.col("b")))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 44 │
| └─────┘
|
| drop_nans(self) -> 'Expr'
| Drop all floating point NaN values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nulls
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To drop null values, use :func:`drop_nulls`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, None, 3.0, float("nan")]})
| >>> df.select(pl.col("a").drop_nans())
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 1.0 │
| │ null │
| │ 3.0 │
| └──────┘
|
| drop_nulls(self) -> 'Expr'
| Drop all null values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nans
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To drop NaN values, use :func:`drop_nans`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0, None, 3.0, float("nan")]})
| >>> df.select(pl.col("a").drop_nulls())
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| │ 3.0 │
| │ NaN │
| └─────┘
|
| entropy(self, base: 'float' = 2.718281828459045, *, normalize: 'bool' = True) -> 'Expr'
| Computes the entropy.
|
| Uses the formula `-sum(pk * log(pk))` where `pk` are discrete probabilities.
|
| Parameters
| ----------
| base
| Given base, defaults to `e`
| normalize
| Normalize pk if it doesn't sum to 1.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").entropy(base=2))
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.459148 │
| └──────────┘
| >>> df.select(pl.col("a").entropy(base=2, normalize=False))
| shape: (1, 1)
| ┌───────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═══════════╡
| │ -6.754888 │
| └───────────┘
|
| eq(self, other: 'Any') -> 'Expr'
| Method equivalent of equality operator `expr == other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0],
| ... "y": [2.0, 2.0, float("nan"), 4.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").eq(pl.col("y")).alias("x == y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x == y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 1.0 ┆ 2.0 ┆ false │
| │ 2.0 ┆ 2.0 ┆ true │
| │ NaN ┆ NaN ┆ true │
| │ 4.0 ┆ 4.0 ┆ true │
| └─────┴─────┴────────┘
|
| eq_missing(self, other: 'Any') -> 'Expr'
| Method equivalent of equality operator `expr == other` where `None == None`.
|
| This differs from default `eq` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0, None, None],
| ... "y": [2.0, 2.0, float("nan"), 4.0, 5.0, None],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").eq(pl.col("y")).alias("x eq y"),
| ... pl.col("x").eq_missing(pl.col("y")).alias("x eq_missing y"),
| ... )
| shape: (6, 4)
| ┌──────┬──────┬────────┬────────────────┐
| │ x ┆ y ┆ x eq y ┆ x eq_missing y │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪══════╪════════╪════════════════╡
| │ 1.0 ┆ 2.0 ┆ false ┆ false │
| │ 2.0 ┆ 2.0 ┆ true ┆ true │
| │ NaN ┆ NaN ┆ true ┆ true │
| │ 4.0 ┆ 4.0 ┆ true ┆ true │
| │ null ┆ 5.0 ┆ null ┆ false │
| │ null ┆ null ┆ null ┆ true │
| └──────┴──────┴────────┴────────────────┘
|
| ewm_mean(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving average.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_mean(com=1, ignore_nulls=False))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 1.666667 │
| │ 2.428571 │
| └──────────┘
|
| ewm_mean_by(self, by: 'str_ | IntoExpr', *, half_life: 'str_ | timedelta') -> 'Expr'
| Compute time-based exponentially weighted moving average.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMA is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \alpha_i &= 1 - \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= \alpha_i x_i + (1 - \alpha_i) y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Times to calculate average by. Should be ``DateTime``, ``Date``, ``UInt64``,
| ``UInt32``, ``Int64``, or ``Int32`` data type.
| half_life
| Unit over which observation decays to half its value.
|
| Can be created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 day)
| - 1w (1 week)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| Note that `half_life` is treated as a constant duration - calendar
| durations such as months (or even days in the time-zone-aware case)
| are not supported, please express your duration in an approximately
| equivalent number of hours (e.g. '370h' instead of '1mo').
|
| Returns
| -------
| Expr
| :class:`.Float16` if input is `Float16`, class:`.Float32` if input is
| `Float32`, otherwise class:`.Float64`.
|
| Examples
| --------
| >>> from datetime import date, timedelta
| >>> df = pl.DataFrame(
| ... {
| ... "values": [0, 1, 2, None, 4],
| ... "times": [
| ... date(2020, 1, 1),
| ... date(2020, 1, 3),
| ... date(2020, 1, 10),
| ... date(2020, 1, 15),
| ... date(2020, 1, 17),
| ... ],
| ... }
| ... ).sort("times")
| >>> df.with_columns(
| ... result=pl.col("values").ewm_mean_by("times", half_life="4d"),
| ... )
| shape: (5, 3)
| ┌────────┬────────────┬──────────┐
| │ values ┆ times ┆ result │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ date ┆ f64 │
| ╞════════╪════════════╪══════════╡
| │ 0 ┆ 2020-01-01 ┆ 0.0 │
| │ 1 ┆ 2020-01-03 ┆ 0.292893 │
| │ 2 ┆ 2020-01-10 ┆ 1.492474 │
| │ null ┆ 2020-01-15 ┆ null │
| │ 4 ┆ 2020-01-17 ┆ 3.254508 │
| └────────┴────────────┴──────────┘
|
| ewm_std(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving standard deviation.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_std(com=1, ignore_nulls=False))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ null │
| │ 0.707107 │
| │ 0.963624 │
| └──────────┘
|
| ewm_sum(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_sum(alpha=0.5))
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ 1.0 │
| │ 2.5 │
| │ 4.25 │
| └──────┘
|
| ewm_sum_by(self, by: 'str_ | IntoExpr', *, half_life: 'str_ | timedelta') -> 'Expr'
| Compute time-based exponentially-weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMS is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \lambda_i &= \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= x_i + \lambda_i y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Column to use as reference for the time decay.
| half_life
| Half-life of the exponential decay.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "values": [1, 2, 3, 4, 5],
| ... "times": [0, 1, 2, 5, 6],
| ... }
| ... )
| >>> df.select(
| ... pl.col("values").ewm_sum_by("times", half_life="1i"),
| ... )
| shape: (5, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 2.5 │
| │ 4.25 │
| │ 4.53125 │
| │ 7.265625 │
| └──────────┘
|
| ewm_var(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Expr'
| Compute exponentially-weighted moving variance.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").ewm_var(com=1, ignore_nulls=False))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ null │
| │ 0.5 │
| │ 0.928571 │
| └──────────┘
|
| exp(self) -> 'Expr'
| Compute the exponential, element-wise.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").exp())
| shape: (3, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 2.718282 │
| │ 7.389056 │
| │ 54.59815 │
| └──────────┘
|
| explode(self, *, empty_as_null: 'bool' = <object object at 0x00000280D5528F20>, keep_nulls: 'bool' = True) -> 'Expr'
| Explode a list expression.
|
| This means that every item is expanded to a new row.
|
| Parameters
| ----------
| empty_as_null
| Explode an empty list/array into a `null`.
| keep_nulls
| Explode a `null` list/array into a `null`.
|
| Returns
| -------
| Expr
| Expression with the data type of the list elements.
|
| See Also
| --------
| Expr.list.explode : Explode a list column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["a", "b"],
| ... "values": [
| ... [1, 2],
| ... [3, 4],
| ... ],
| ... }
| ... )
| >>> df.select(pl.col("values").explode(empty_as_null=False))
| shape: (4, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 1 │
| │ 2 │
| │ 3 │
| │ 4 │
| └────────┘
|
| extend_constant(self, value: 'IntoExpr', n: 'int | IntoExprColumn') -> 'Expr'
| Extremely fast method for extending the Series with 'n' copies of a value.
|
| Parameters
| ----------
| value
| A constant literal value or a unit expression with which to extend the
| expression result Series; can pass None to extend with nulls.
| n
| The number of additional values that will be added.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1, 2, 3]})
| >>> df.select((pl.col("values") - 1).extend_constant(99, n=2))
| shape: (5, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 0 │
| │ 1 │
| │ 2 │
| │ 99 │
| │ 99 │
| └────────┘
|
| fill_nan(self, value: 'int | float | Expr | None') -> 'Expr'
| Fill floating point NaN value with a fill value.
|
| Parameters
| ----------
| value
| Value used to fill NaN values.
|
| See Also
| --------
| fill_null
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To fill null values, use :func:`fill_null`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1.0, None, float("nan")],
| ... "b": [4.0, float("nan"), 6],
| ... }
| ... )
| >>> df.with_columns(pl.col("b").fill_nan(0))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪═════╡
| │ 1.0 ┆ 4.0 │
| │ null ┆ 0.0 │
| │ NaN ┆ 6.0 │
| └──────┴─────┘
|
| fill_null(self, value: 'Any | Expr | None' = None, strategy: 'FillNullStrategy | None' = None, limit: 'int | None' = None) -> 'Expr'
| Fill null values using the specified value or strategy.
|
| To interpolate over null values see interpolate.
| See the examples below to fill nulls with an expression.
|
| Parameters
| ----------
| value
| Value used to fill null values.
| strategy : {None, 'forward', 'backward', 'min', 'max', 'mean', 'zero', 'one'}
| Strategy used to fill null values.
| limit
| Number of consecutive null values to fill when using the 'forward' or
| 'backward' strategy.
|
| See Also
| --------
| backward_fill
| fill_nan
| forward_fill
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To fill NaN values, use :func:`fill_nan`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": [4, None, 6],
| ... }
| ... )
| >>> df.with_columns(pl.col("b").fill_null(strategy="zero"))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 0 │
| │ null ┆ 6 │
| └──────┴─────┘
| >>> df.with_columns(pl.col("b").fill_null(99))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 99 │
| │ null ┆ 6 │
| └──────┴─────┘
| >>> df.with_columns(pl.col("b").fill_null(strategy="forward"))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪═════╡
| │ 1 ┆ 4 │
| │ 2 ┆ 4 │
| │ null ┆ 6 │
| └──────┴─────┘
| >>> df.with_columns(pl.col("b").fill_null(pl.col("b").median()))
| shape: (3, 2)
| ┌──────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞══════╪═════╡
| │ 1 ┆ 4.0 │
| │ 2 ┆ 5.0 │
| │ null ┆ 6.0 │
| └──────┴─────┘
| >>> df.with_columns(pl.all().fill_null(pl.all().median()))
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 1.0 ┆ 4.0 │
| │ 2.0 ┆ 5.0 │
| │ 1.5 ┆ 6.0 │
| └─────┴─────┘
|
| filter(self, *predicates: 'IntoExprColumn | Iterable[IntoExprColumn]', **constraints: 'Any') -> 'Expr'
| Filter the expression based on one or more predicate expressions.
|
| The original order of the remaining elements is preserved.
|
| Elements where the filter does not evaluate to True are discarded, including
| nulls.
|
| Mostly useful in an aggregation context. If you want to filter on a DataFrame
| level, use `LazyFrame.filter`.
|
| Parameters
| ----------
| predicates
| Expression(s) that evaluates to a boolean Series.
| constraints
| Column filters; use `name = value` to filter columns by the supplied value.
| Each constraint will behave the same as `pl.col(name).eq(value)`, and
| be implicitly joined with the other filter conditions using `&`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group_col": ["g1", "g1", "g2"],
| ... "b": [1, 2, 3],
| ... }
| ... )
| >>> df.group_by("group_col").agg(
| ... lt=pl.col("b").filter(pl.col("b") < 2).sum(),
| ... gte=pl.col("b").filter(pl.col("b") >= 2).sum(),
| ... ).sort("group_col")
| shape: (2, 3)
| ┌───────────┬─────┬─────┐
| │ group_col ┆ lt ┆ gte │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════════╪═════╪═════╡
| │ g1 ┆ 1 ┆ 2 │
| │ g2 ┆ 0 ┆ 3 │
| └───────────┴─────┴─────┘
|
| Filter expressions can also take constraints as keyword arguments.
|
| >>> df = pl.DataFrame(
| ... {
| ... "key": ["a", "a", "a", "a", "b", "b", "b", "b", "b"],
| ... "n": [1, 2, 2, 3, 1, 3, 3, 2, 3],
| ... },
| ... )
| >>> df.group_by("key").agg(
| ... n_1=pl.col("n").filter(n=1).sum(),
| ... n_2=pl.col("n").filter(n=2).sum(),
| ... n_3=pl.col("n").filter(n=3).sum(),
| ... ).sort(by="key")
| shape: (2, 4)
| ┌─────┬─────┬─────┬─────┐
| │ key ┆ n_1 ┆ n_2 ┆ n_3 │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═════╡
| │ a ┆ 1 ┆ 4 ┆ 3 │
| │ b ┆ 1 ┆ 2 ┆ 9 │
| └─────┴─────┴─────┴─────┘
|
| first(self, *, ignore_nulls: 'bool' = False) -> 'Expr'
| Get the first value.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the first non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [None, 1, 2]})
| >>> df.select(pl.col("a").first())
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| └──────┘
| >>> df.select(pl.col("a").first(ignore_nulls=True))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| └─────┘
|
| flatten(self) -> 'Expr'
| Flatten a list or string column.
|
| Alias for :func:`Expr.list.explode`.
|
| .. deprecated:: 1.38
| `Expr.flatten()` is deprecated and will be removed in version 2.0.
| Use `Expr.list.explode(keep_nulls=False, empty_as_null=False)` instead,
| which provides the behavior you likely expect.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["a", "b", "b"],
| ... "values": [[1, 2], [2, 3], [4]],
| ... }
| ... )
| >>> df.group_by("group").agg(pl.col("values").flatten()) # doctest: +SKIP
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ values │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪═══════════╡
| │ a ┆ [1, 2] │
| │ b ┆ [2, 3, 4] │
| └───────┴───────────┘
|
| floor(self) -> 'Expr'
| Rounds down to the nearest integer value.
|
| Only works on floating point Series.
|
| See Also
| --------
| ceil : Round up to the nearest integer value.
| round : Round to the nearest integer.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.3, 0.5, 1.0, 1.1]})
| >>> df.select(pl.col("a").floor())
| shape: (4, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 0.0 │
| │ 1.0 │
| │ 1.0 │
| └─────┘
|
| floordiv(self, other: 'Any') -> 'Expr'
| Method equivalent of integer division operator `expr // other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| See Also
| --------
| truediv
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 3, 4, 5]})
| >>> df.with_columns(
| ... pl.col("x").truediv(2).alias("x/2"),
| ... pl.col("x").floordiv(2).alias("x//2"),
| ... )
| shape: (5, 3)
| ┌─────┬─────┬──────┐
| │ x ┆ x/2 ┆ x//2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ i64 │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 0.5 ┆ 0 │
| │ 2 ┆ 1.0 ┆ 1 │
| │ 3 ┆ 1.5 ┆ 1 │
| │ 4 ┆ 2.0 ┆ 2 │
| │ 5 ┆ 2.5 ┆ 2 │
| └─────┴─────┴──────┘
|
| Note that Polars' `floordiv` is subtly different from Python's floor division.
| For example, consider 6.0 floor-divided by 0.1.
| Python gives:
|
| >>> 6.0 // 0.1
| 59.0
|
| because `0.1` is not represented internally as that exact value,
| but a slightly larger value.
| So the result of the division is slightly less than 60,
| meaning the flooring operation returns 59.0.
|
| Polars instead first does the floating-point division,
| resulting in a floating-point value of 60.0,
| and then performs the flooring operation using :any:`floor`:
|
| >>> df = pl.DataFrame({"x": [6.0, 6.03]})
| >>> df.with_columns(
| ... pl.col("x").truediv(0.1).alias("x/0.1"),
| ... ).with_columns(
| ... pl.col("x/0.1").floor().alias("x/0.1 floor"),
| ... )
| shape: (2, 3)
| ┌──────┬───────┬─────────────┐
| │ x ┆ x/0.1 ┆ x/0.1 floor │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 │
| ╞══════╪═══════╪═════════════╡
| │ 6.0 ┆ 60.0 ┆ 60.0 │
| │ 6.03 ┆ 60.3 ┆ 60.0 │
| └──────┴───────┴─────────────┘
|
| yielding the more intuitive result 60.0.
| The row with x = 6.03 is included to demonstrate
| the effect of the flooring operation.
|
| `floordiv` combines those two steps
| to give the same result with one expression:
|
| >>> df.with_columns(
| ... pl.col("x").floordiv(0.1).alias("x//0.1"),
| ... )
| shape: (2, 2)
| ┌──────┬────────┐
| │ x ┆ x//0.1 │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞══════╪════════╡
| │ 6.0 ┆ 60.0 │
| │ 6.03 ┆ 60.0 │
| └──────┴────────┘
|
| forward_fill(self, limit: 'int | None' = None) -> 'Expr'
| Fill missing values with the last non-null value.
|
| This is an alias of `.fill_null(strategy="forward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to forward fill.
|
| See Also
| --------
| backward_fill
| fill_null
| shift
|
| gather(self, indices: 'int | Sequence[int] | IntoExpr | Series | np.ndarray[Any, Any]', *, null_on_oob: 'bool' = False) -> 'Expr'
| Take values by index.
|
| Parameters
| ----------
| indices
| An expression that leads to a UInt32 dtyped Series.
| null_on_oob
| Behavior if an index is out of bounds:
|
| - True -> set the result to null
| - False -> raise an error
|
| Returns
| -------
| Expr
| Expression of the same data type.
|
| See Also
| --------
| Expr.get : Take a single value
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": [
| ... "one",
| ... "one",
| ... "one",
| ... "two",
| ... "two",
| ... "two",
| ... ],
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.group_by("group", maintain_order=True).agg(
| ... pl.col("value").gather([2, 1])
| ... )
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪═══════════╡
| │ one ┆ [2, 98] │
| │ two ┆ [4, 99] │
| └───────┴───────────┘
|
| Use `null_on_oob=True` to return null for out-of-bounds indices.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").gather([0, 1, 10], null_on_oob=True))
| shape: (3, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 1 │
| │ 2 │
| │ null │
| └──────┘
|
| gather_every(self, n: 'int', offset: 'int' = 0) -> 'Expr'
| Take every nth value in the Series and return as a new Series.
|
| Parameters
| ----------
| n
| Gather every *n*-th row.
| offset
| Starting index.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7, 8, 9]})
| >>> df.select(pl.col("foo").gather_every(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 4 │
| │ 7 │
| └─────┘
|
| >>> df.select(pl.col("foo").gather_every(3, offset=1))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 5 │
| │ 8 │
| └─────┘
|
| ge(self, other: 'Any') -> 'Expr'
| Method equivalent of "greater than or equal" operator `expr >= other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5.0, 4.0, float("nan"), 2.0],
| ... "y": [5.0, 3.0, float("nan"), 1.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").ge(pl.col("y")).alias("x >= y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x >= y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 5.0 ┆ 5.0 ┆ true │
| │ 4.0 ┆ 3.0 ┆ true │
| │ NaN ┆ NaN ┆ true │
| │ 2.0 ┆ 1.0 ┆ true │
| └─────┴─────┴────────┘
|
| get(self, index: 'int | Expr', *, null_on_oob: 'bool' = False) -> 'Expr'
| Return a single value by index.
|
| Parameters
| ----------
| index
| An expression that evaluates to an integer.
| Negative indexing is supported.
|
| null_on_oob
| Behavior if an index is out of bounds:
|
| - True -> set the result to null
| - False -> raise an error
|
| Returns
| -------
| Expr
| Expression of the same data type.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group": [
| ... "one",
| ... "one",
| ... "one",
| ... "two",
| ... "two",
| ... "two",
| ... ],
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.group_by("group", maintain_order=True).agg(pl.col("value").get(1))
| shape: (2, 2)
| ┌───────┬───────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═══════╪═══════╡
| │ one ┆ 98 │
| │ two ┆ 99 │
| └───────┴───────┘
|
| gt(self, other: 'Any') -> 'Expr'
| Method equivalent of "greater than" operator `expr > other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5.0, 4.0, float("nan"), 2.0],
| ... "y": [5.0, 3.0, float("nan"), 1.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").gt(pl.col("y")).alias("x > y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────┐
| │ x ┆ y ┆ x > y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪═══════╡
| │ 5.0 ┆ 5.0 ┆ false │
| │ 4.0 ┆ 3.0 ┆ true │
| │ NaN ┆ NaN ┆ false │
| │ 2.0 ┆ 1.0 ┆ true │
| └─────┴─────┴───────┘
|
| has_nulls(self) -> 'Expr'
| Check whether the expression contains one or more null values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 1, None],
| ... "b": [10, None, 300],
| ... "c": [350, 650, 850],
| ... }
| ... )
| >>> df.select(pl.all().has_nulls())
| shape: (1, 3)
| ┌──────┬──────┬───────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞══════╪══════╪═══════╡
| │ true ┆ true ┆ false │
| └──────┴──────┴───────┘
|
| hash(self, seed: 'int' = 0, seed_1: 'int | None' = None, seed_2: 'int | None' = None, seed_3: 'int | None' = None) -> 'Expr'
| Hash the elements in the selection.
|
| The hash value is of type `UInt64`.
|
| Parameters
| ----------
| seed
| Random seed parameter. Defaults to 0.
| seed_1
| Random seed parameter. Defaults to `seed` if not set.
| seed_2
| Random seed parameter. Defaults to `seed` if not set.
| seed_3
| Random seed parameter. Defaults to `seed` if not set.
|
| Notes
| -----
| This implementation of `hash` does not guarantee stable results
| across different Polars versions. Its stability is only guaranteed within a
| single version.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None],
| ... "b": ["x", None, "z"],
| ... }
| ... )
| >>> df.with_columns(pl.all().hash(10, 20, 30, 40)) # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌──────────────────────┬──────────────────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u64 ┆ u64 │
| ╞══════════════════════╪══════════════════════╡
| │ 9774092659964970114 ┆ 13614470193936745724 │
| │ 1101441246220388612 ┆ 11638928888656214026 │
| │ 11638928888656214026 ┆ 13382926553367784577 │
| └──────────────────────┴──────────────────────┘
|
| head(self, n: 'int | Expr' = 10) -> 'Expr'
| Get the first `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7]})
| >>> df.select(pl.col("foo").head(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| hist(self, bins: 'IntoExpr | None' = None, *, bin_count: 'int | None' = None, include_category: 'bool' = False, include_breakpoint: 'bool' = False) -> 'Expr'
| Bin values into buckets and count their occurrences.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| bins
| Bin edges. If None given, we determine the edges based on the data.
| bin_count
| If `bins` is not provided, `bin_count` uniform bins are created that fully
| encompass the data.
| include_breakpoint
| Include a column that indicates the upper breakpoint.
| include_category
| Include a column that shows the intervals as categories.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 8, 8, 2, 1, 3]})
| >>> df.select(pl.col("a").hist(bins=[1, 2, 3]))
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 3 │
| │ 2 │
| └─────┘
| >>> df.select(
| ... pl.col("a").hist(
| ... bins=[1, 2, 3], include_breakpoint=True, include_category=True
| ... )
| ... )
| shape: (2, 1)
| ┌──────────────────────┐
| │ a │
| │ --- │
| │ struct[3] │
| ╞══════════════════════╡
| │ {2.0,"[1.0, 2.0]",3} │
| │ {3.0,"(2.0, 3.0]",2} │
| └──────────────────────┘
|
| implode(self, *, maintain_order: 'bool' = True) -> 'Expr'
| Aggregate values into a list.
|
| The returned list itself is a scalar value of `list` dtype.
|
| Parameters
| ----------
| maintain_order
| Whether to preserve the order of elements in the list. Setting this
| to `False` can improve performance, especially within `group_by`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": [4, 5, 6],
| ... }
| ... )
| >>> df.select(pl.all().implode())
| shape: (1, 2)
| ┌───────────┬───────────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ list[i64] ┆ list[i64] │
| ╞═══════════╪═══════════╡
| │ [1, 2, 3] ┆ [4, 5, 6] │
| └───────────┴───────────┘
|
| index_of(self, element: 'IntoExpr') -> 'Expr'
| Get the index of the first occurrence of a value, or ``None`` if it's not found.
|
| Parameters
| ----------
| element
| Value to find.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, None, 17]})
| >>> df.select(
| ... [
| ... pl.col("a").index_of(17).alias("seventeen"),
| ... pl.col("a").index_of(None).alias("null"),
| ... pl.col("a").index_of(55).alias("fiftyfive"),
| ... ]
| ... )
| shape: (1, 3)
| ┌───────────┬──────┬───────────┐
| │ seventeen ┆ null ┆ fiftyfive │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═══════════╪══════╪═══════════╡
| │ 2 ┆ 1 ┆ null │
| └───────────┴──────┴───────────┘
|
| inspect(self, fmt: 'str_' = '{}') -> 'Expr'
| Print the value that this expression evaluates to and pass on the value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 1, 2]})
| >>> df.select(pl.col("foo").cum_sum().inspect("value is: {}").alias("bar"))
| value is: shape: (3,)
| Series: 'foo' [i64]
| [
| 1
| 2
| 4
| ]
| shape: (3, 1)
| ┌─────┐
| │ bar │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 4 │
| └─────┘
|
| interpolate(self, method: 'InterpolationMethod' = 'linear') -> 'Expr'
| Interpolate intermediate values.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| method : {'linear', 'nearest'}
| Interpolation method.
|
| Examples
| --------
| Fill null values using linear interpolation.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, None, 3],
| ... "b": [1.0, float("nan"), 3.0],
| ... }
| ... )
| >>> df.select(pl.all().interpolate())
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════╡
| │ 1.0 ┆ 1.0 │
| │ 2.0 ┆ NaN │
| │ 3.0 ┆ 3.0 │
| └─────┴─────┘
|
| Fill null values using nearest interpolation.
|
| >>> df.select(pl.all().interpolate("nearest"))
| shape: (3, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════╪═════╡
| │ 1 ┆ 1.0 │
| │ 3 ┆ NaN │
| │ 3 ┆ 3.0 │
| └─────┴─────┘
|
| Regrid data to a new grid.
|
| >>> df_original_grid = pl.DataFrame(
| ... {
| ... "grid_points": [1, 3, 10],
| ... "values": [2.0, 6.0, 20.0],
| ... }
| ... ) # Interpolate from this to the new grid
| >>> df_new_grid = pl.DataFrame({"grid_points": range(1, 11)})
| >>> df_new_grid.join(
| ... df_original_grid, on="grid_points", how="left", coalesce=True
| ... ).with_columns(pl.col("values").interpolate())
| shape: (10, 2)
| ┌─────────────┬────────┐
| │ grid_points ┆ values │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞═════════════╪════════╡
| │ 1 ┆ 2.0 │
| │ 2 ┆ 4.0 │
| │ 3 ┆ 6.0 │
| │ 4 ┆ 8.0 │
| │ 5 ┆ 10.0 │
| │ 6 ┆ 12.0 │
| │ 7 ┆ 14.0 │
| │ 8 ┆ 16.0 │
| │ 9 ┆ 18.0 │
| │ 10 ┆ 20.0 │
| └─────────────┴────────┘
|
| interpolate_by(self, by: 'IntoExpr') -> 'Expr'
| Fill null values using interpolation based on another column.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| by
| Column to interpolate values based on.
|
| Examples
| --------
| Fill null values using linear interpolation.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, None, None, 3],
| ... "b": [1, 2, 7, 8],
| ... }
| ... )
| >>> df.with_columns(a_interpolated=pl.col("a").interpolate_by("b"))
| shape: (4, 3)
| ┌──────┬─────┬────────────────┐
| │ a ┆ b ┆ a_interpolated │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞══════╪═════╪════════════════╡
| │ 1 ┆ 1 ┆ 1.0 │
| │ null ┆ 2 ┆ 1.285714 │
| │ null ┆ 7 ┆ 2.714286 │
| │ 3 ┆ 8 ┆ 3.0 │
| └──────┴─────┴────────────────┘
|
| is_between(self, lower_bound: 'IntoExpr', upper_bound: 'IntoExpr', closed: 'ClosedInterval' = 'both') -> 'Expr'
| Check if this expression is between the given lower and upper bounds.
|
| Parameters
| ----------
| lower_bound
| Lower bound value. Accepts expression input. Strings are parsed as column
| names, other non-expression inputs are parsed as literals.
| upper_bound
| Upper bound value. Accepts expression input. Strings are parsed as column
| names, other non-expression inputs are parsed as literals.
| closed : {'both', 'left', 'right', 'none'}
| Define which sides of the interval are closed (inclusive).
|
| Notes
| -----
| If the value of the `lower_bound` is greater than that of the `upper_bound`
| then the result will be False, as no value can satisfy the condition.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"num": [1, 2, 3, 4, 5]})
| >>> df.with_columns(pl.col("num").is_between(2, 4).alias("is_between"))
| shape: (5, 2)
| ┌─────┬────────────┐
| │ num ┆ is_between │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪════════════╡
| │ 1 ┆ false │
| │ 2 ┆ true │
| │ 3 ┆ true │
| │ 4 ┆ true │
| │ 5 ┆ false │
| └─────┴────────────┘
|
| Use the `closed` argument to include or exclude the values at the bounds:
|
| >>> df.with_columns(
| ... pl.col("num").is_between(2, 4, closed="left").alias("is_between")
| ... )
| shape: (5, 2)
| ┌─────┬────────────┐
| │ num ┆ is_between │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪════════════╡
| │ 1 ┆ false │
| │ 2 ┆ true │
| │ 3 ┆ true │
| │ 4 ┆ false │
| │ 5 ┆ false │
| └─────┴────────────┘
|
| You can also use strings as well as numeric/temporal values (note: ensure that
| string literals are wrapped with `lit` so as not to conflate them with
| column names):
|
| >>> df = pl.DataFrame({"a": ["a", "b", "c", "d", "e"]})
| >>> df.with_columns(
| ... pl.col("a")
| ... .is_between(pl.lit("a"), pl.lit("c"), closed="both")
| ... .alias("is_between")
| ... )
| shape: (5, 2)
| ┌─────┬────────────┐
| │ a ┆ is_between │
| │ --- ┆ --- │
| │ str ┆ bool │
| ╞═════╪════════════╡
| │ a ┆ true │
| │ b ┆ true │
| │ c ┆ true │
| │ d ┆ false │
| │ e ┆ false │
| └─────┴────────────┘
|
| Use column expressions as lower/upper bounds, comparing to a literal value:
|
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5], "b": [5, 4, 3, 2, 1]})
| >>> df.with_columns(
| ... pl.lit(3).is_between(pl.col("a"), pl.col("b")).alias("between_ab")
| ... )
| shape: (5, 3)
| ┌─────┬─────┬────────────┐
| │ a ┆ b ┆ between_ab │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ bool │
| ╞═════╪═════╪════════════╡
| │ 1 ┆ 5 ┆ true │
| │ 2 ┆ 4 ┆ true │
| │ 3 ┆ 3 ┆ true │
| │ 4 ┆ 2 ┆ false │
| │ 5 ┆ 1 ┆ false │
| └─────┴─────┴────────────┘
|
| is_close(self, other: 'IntoExpr', *, abs_tol: 'float' = 0.0, rel_tol: 'float' = 1e-09, nans_equal: 'bool' = False) -> 'Expr'
| Check if this expression is close, i.e. almost equal, to the other expression.
|
| Two values `a` and `b` are considered close if the following condition holds:
|
| .. math::
| |a-b| \le max \{ \text{rel_tol} \cdot max \{ |a|, |b| \}, \text{abs_tol} \}
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
| abs_tol
| Absolute tolerance. This is the maximum allowed absolute difference between
| two values. Must be non-negative.
| rel_tol
| Relative tolerance. This is the maximum allowed difference between two
| values, relative to the larger absolute value. Must be non-negative.
| nans_equal
| Whether NaN values should be considered equal.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Notes
| -----
| The implementation of this method is symmetric and mirrors the behavior of
| :meth:`math.isclose`. Specifically note that this behavior is different to
| :meth:`numpy.isclose`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.5, 2.0, 2.5], "b": [1.55, 2.2, 3.0]})
| >>> df.with_columns(pl.col("a").is_close("b", abs_tol=0.1).alias("is_close"))
| shape: (3, 3)
| ┌─────┬──────┬──────────┐
| │ a ┆ b ┆ is_close │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪══════╪══════════╡
| │ 1.5 ┆ 1.55 ┆ true │
| │ 2.0 ┆ 2.2 ┆ false │
| │ 2.5 ┆ 3.0 ┆ false │
| └─────┴──────┴──────────┘
|
| is_duplicated(self) -> 'Expr'
| Return a boolean mask indicating duplicated values.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
| >>> df.select(pl.col("a").is_duplicated())
| shape: (3, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ true │
| │ true │
| │ false │
| └───────┘
|
| is_empty(self, *, ignore_nulls: 'bool' = False) -> 'Expr'
| Return whether the column is empty.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| ignore_nulls
| If true a column containing only nulls will also be considered empty.
| The default is false.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [None, None]})
| >>> df.select(
| ... a=pl.col.x.is_empty(),
| ... b=pl.col.x.drop_nulls().is_empty(),
| ... c=pl.col.x.is_empty(ignore_nulls=True),
| ... )
| shape: (1, 3)
| ┌───────┬──────┬──────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞═══════╪══════╪══════╡
| │ false ┆ true ┆ true │
| └───────┴──────┴──────┘
|
| is_finite(self) -> 'Expr'
| Returns a boolean Series indicating which values are finite.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1.0, 2],
| ... "B": [3.0, float("inf")],
| ... }
| ... )
| >>> df.select(pl.all().is_finite())
| shape: (2, 2)
| ┌──────┬───────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ bool ┆ bool │
| ╞══════╪═══════╡
| │ true ┆ true │
| │ true ┆ false │
| └──────┴───────┘
|
| is_first_distinct(self) -> 'Expr'
| Return a boolean mask indicating the first occurrence of each distinct value.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 3, 2]})
| >>> df.with_columns(pl.col("a").is_first_distinct().alias("first"))
| shape: (5, 2)
| ┌─────┬───────┐
| │ a ┆ first │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪═══════╡
| │ 1 ┆ true │
| │ 1 ┆ false │
| │ 2 ┆ true │
| │ 3 ┆ true │
| │ 2 ┆ false │
| └─────┴───────┘
|
| is_in(self, other: 'Expr | Collection[Any] | Series', *, nulls_equal: 'bool' = False) -> 'Expr'
| Check if elements of this expression are present in the other Series.
|
| Parameters
| ----------
| other
| Series or sequence of primitive type.
| nulls_equal : bool, default False
| If True, treat null as a distinct value. Null values will not propagate.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"sets": [[1, 2, 3], [1, 2], [9, 10]], "optional_members": [1, 2, 3]}
| ... )
| >>> df.with_columns(contains=pl.col("optional_members").is_in("sets"))
| shape: (3, 3)
| ┌───────────┬──────────────────┬──────────┐
| │ sets ┆ optional_members ┆ contains │
| │ --- ┆ --- ┆ --- │
| │ list[i64] ┆ i64 ┆ bool │
| ╞═══════════╪══════════════════╪══════════╡
| │ [1, 2, 3] ┆ 1 ┆ true │
| │ [1, 2] ┆ 2 ┆ true │
| │ [9, 10] ┆ 3 ┆ false │
| └───────────┴──────────────────┴──────────┘
|
| is_infinite(self) -> 'Expr'
| Returns a boolean Series indicating which values are infinite.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1.0, 2],
| ... "B": [3.0, float("inf")],
| ... }
| ... )
| >>> df.select(pl.all().is_infinite())
| shape: (2, 2)
| ┌───────┬───────┐
| │ A ┆ B │
| │ --- ┆ --- │
| │ bool ┆ bool │
| ╞═══════╪═══════╡
| │ false ┆ false │
| │ false ┆ true │
| └───────┴───────┘
|
| is_last_distinct(self) -> 'Expr'
| Return a boolean mask indicating the last occurrence of each distinct value.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 3, 2]})
| >>> df.with_columns(pl.col("a").is_last_distinct().alias("last"))
| shape: (5, 2)
| ┌─────┬───────┐
| │ a ┆ last │
| │ --- ┆ --- │
| │ i64 ┆ bool │
| ╞═════╪═══════╡
| │ 1 ┆ false │
| │ 1 ┆ true │
| │ 2 ┆ false │
| │ 3 ┆ true │
| │ 2 ┆ true │
| └─────┴───────┘
|
| is_nan(self) -> 'Expr'
| Returns a boolean Series indicating which values are NaN.
|
| Notes
| -----
| Floating point `NaN` (Not A Number) should not be confused
| with missing data represented as `Null/None`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(pl.col(pl.Float64).is_nan().name.suffix("_isnan"))
| shape: (5, 3)
| ┌──────┬─────┬─────────┐
| │ a ┆ b ┆ b_isnan │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪═════╪═════════╡
| │ 1 ┆ 1.0 ┆ false │
| │ 2 ┆ 2.0 ┆ false │
| │ null ┆ NaN ┆ true │
| │ 1 ┆ 1.0 ┆ false │
| │ 5 ┆ 5.0 ┆ false │
| └──────┴─────┴─────────┘
|
| is_not_nan(self) -> 'Expr'
| Returns a boolean Series indicating which values are not NaN.
|
| Notes
| -----
| Floating point `NaN` (Not A Number) should not be confused
| with missing data represented as `Null/None`.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(pl.col(pl.Float64).is_not_nan().name.suffix("_is_not_nan"))
| shape: (5, 3)
| ┌──────┬─────┬──────────────┐
| │ a ┆ b ┆ b_is_not_nan │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool │
| ╞══════╪═════╪══════════════╡
| │ 1 ┆ 1.0 ┆ true │
| │ 2 ┆ 2.0 ┆ true │
| │ null ┆ NaN ┆ false │
| │ 1 ┆ 1.0 ┆ true │
| │ 5 ┆ 5.0 ┆ true │
| └──────┴─────┴──────────────┘
|
| is_not_null(self) -> 'Expr'
| Returns a boolean Series indicating which values are not null.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.all().is_not_null().name.suffix("_not_null") # nan != null
| ... )
| shape: (5, 4)
| ┌──────┬─────┬────────────┬────────────┐
| │ a ┆ b ┆ a_not_null ┆ b_not_null │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪═════╪════════════╪════════════╡
| │ 1 ┆ 1.0 ┆ true ┆ true │
| │ 2 ┆ 2.0 ┆ true ┆ true │
| │ null ┆ NaN ┆ false ┆ true │
| │ 1 ┆ 1.0 ┆ true ┆ true │
| │ 5 ┆ 5.0 ┆ true ┆ true │
| └──────┴─────┴────────────┴────────────┘
|
| is_null(self) -> 'Expr'
| Returns a boolean Series indicating which values are null.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, None, 1, 5],
| ... "b": [1.0, 2.0, float("nan"), 1.0, 5.0],
| ... }
| ... )
| >>> df.with_columns(pl.all().is_null().name.suffix("_isnull")) # nan != null
| shape: (5, 4)
| ┌──────┬─────┬──────────┬──────────┐
| │ a ┆ b ┆ a_isnull ┆ b_isnull │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪═════╪══════════╪══════════╡
| │ 1 ┆ 1.0 ┆ false ┆ false │
| │ 2 ┆ 2.0 ┆ false ┆ false │
| │ null ┆ NaN ┆ true ┆ false │
| │ 1 ┆ 1.0 ┆ false ┆ false │
| │ 5 ┆ 5.0 ┆ false ┆ false │
| └──────┴─────┴──────────┴──────────┘
|
| is_sorted(self, *, descending: 'bool | None' = False, nulls_last: 'bool | None' = False) -> 'Expr'
| Checks if an expression is sorted.
|
| If `descending` and/or `nulls_last` are None, it will check `True` and `False`
| for the unspecified option(s), and return `True` if the expression is sorted
| under any combination of those settings.
|
| Parameters
| ----------
| descending
| Checks if the expression is sorted in descending order.
| Defaults to False.
| nulls_last
| Consider null values as being ordered last when checking sortedness.
| Defaults to False.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Boolean`.
|
| Examples
| --------
| Check if a column is sorted in ascending order.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.select(pl.col("a").is_sorted())
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ bool │
| ╞══════╡
| │ true │
| └──────┘
|
| Check if a column is sorted in descending order.
|
| >>> df = pl.DataFrame({"a": [4, 3, 2, 1]})
| >>> df.select(pl.col("a").is_sorted(descending=True))
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ bool │
| ╞══════╡
| │ true │
| └──────┘
|
| Check if a column is sorted in either direction.
|
| >>> df = pl.DataFrame({"a": [4, 3, 2, 1]})
| >>> df.select(pl.col("a").is_sorted(descending=None))
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ bool │
| ╞══════╡
| │ true │
| └──────┘
|
| is_unique(self) -> 'Expr'
| Get mask of unique values.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
| >>> df.select(pl.col("a").is_unique())
| shape: (3, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ false │
| │ true │
| └───────┘
|
| item(self, *, allow_empty: 'bool' = False) -> 'Expr'
| Get the single value.
|
| This raises an error if there is not exactly one value.
|
| Parameters
| ----------
| allow_empty
| Allow having no values to return `null`.
|
| See Also
| --------
| :meth:`Expr.get` : Get a single value by index.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1]})
| >>> df.select(pl.col("a").item())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| └─────┘
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").item())
| Traceback (most recent call last):
| ...
| polars.exceptions.ComputeError: aggregation 'item' expected a single value, got 3 values
| ...
| >>> df.head(0).select(pl.col("a").item(allow_empty=True))
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| └──────┘
|
| kurtosis(self, *, fisher: 'bool' = True, bias: 'bool' = True) -> 'Expr'
| Compute the kurtosis (Fisher or Pearson) of a dataset.
|
| Kurtosis is the fourth central moment divided by the square of the
| variance. If Fisher's definition is used, then 3.0 is subtracted from
| the result to give 0.0 for a normal distribution.
| If bias is False then the kurtosis is calculated using k statistics to
| eliminate bias coming from biased moment estimators.
|
| See scipy.stats for more information
|
| Parameters
| ----------
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").kurtosis())
| shape: (1, 1)
| ┌───────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═══════════╡
| │ -1.153061 │
| └───────────┘
|
| last(self, *, ignore_nulls: 'bool' = False) -> 'Expr'
| Get the last value.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the last non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 3, 2]})
| >>> df.select(pl.col("a").last())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| └─────┘
|
| le(self, other: 'Any') -> 'Expr'
| Method equivalent of "less than or equal" operator `expr <= other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5.0, 4.0, float("nan"), 0.5],
| ... "y": [5.0, 3.5, float("nan"), 2.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").le(pl.col("y")).alias("x <= y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x <= y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 5.0 ┆ 5.0 ┆ true │
| │ 4.0 ┆ 3.5 ┆ false │
| │ NaN ┆ NaN ┆ true │
| │ 0.5 ┆ 2.0 ┆ true │
| └─────┴─────┴────────┘
|
| len(self) -> 'Expr'
| Return the number of elements in the column.
|
| Null values count towards the total.
|
| Returns
| -------
| Expr
| Expression of data type :class:`UInt32`.
|
| See Also
| --------
| count
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3], "b": [None, 4, 4]})
| >>> df.select(pl.all().len())
| shape: (1, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞═════╪═════╡
| │ 3 ┆ 3 │
| └─────┴─────┘
|
| limit(self, n: 'int | Expr' = 10) -> 'Expr'
| Get the first `n` rows (alias for :func:`Expr.head`).
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7]})
| >>> df.select(pl.col("foo").limit(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| log(self, base: 'float | IntoExpr' = 2.718281828459045) -> 'Expr'
| Compute the logarithm to a given base.
|
| Parameters
| ----------
| base
| Given base, defaults to `e`
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").log(base=2))
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.0 │
| │ 1.0 │
| │ 1.584963 │
| └──────────┘
|
| log10(self) -> 'Expr'
| Compute the base 10 logarithm of the input array, element-wise.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").log10())
| shape: (3, 1)
| ┌─────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞═════════╡
| │ 0.0 │
| │ 0.30103 │
| │ 0.60206 │
| └─────────┘
|
| log1p(self) -> 'Expr'
| Compute the natural logarithm of each element plus one.
|
| This computes `log(1 + x)` but is more numerically stable for `x` close to zero.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").log1p())
| shape: (3, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.693147 │
| │ 1.098612 │
| │ 1.386294 │
| └──────────┘
|
| lower_bound(self) -> 'Expr'
| Calculate the lower bound.
|
| Returns a unit Series with the lowest value possible for the dtype of this
| expression.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").lower_bound())
| shape: (1, 1)
| ┌──────────────────────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════════════════════╡
| │ -9223372036854775808 │
| └──────────────────────┘
|
| lt(self, other: 'Any') -> 'Expr'
| Method equivalent of "less than" operator `expr < other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 3.0],
| ... "y": [2.0, 2.0, float("nan"), 4.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").lt(pl.col("y")).alias("x < y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────┐
| │ x ┆ y ┆ x < y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪═══════╡
| │ 1.0 ┆ 2.0 ┆ true │
| │ 2.0 ┆ 2.0 ┆ false │
| │ NaN ┆ NaN ┆ false │
| │ 3.0 ┆ 4.0 ┆ true │
| └─────┴─────┴───────┘
|
| map_batches(self, function: 'Callable[[Series], Series | Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, agg_list: 'bool' = False, is_elementwise: 'bool' = False, returns_scalar: 'bool' = False) -> 'Expr'
| Apply a custom python function to a whole Series or sequence of Series.
|
| The output of this custom function is presumed to be either a Series,
| or a NumPy array (in which case it will be automatically converted into
| a Series), or a scalar that will be converted into a Series. If the
| result is a scalar and you want it to stay as a scalar, pass in
| ``returns_scalar=True``. If you want to apply a
| custom function elementwise over single values, see :func:`map_elements`.
| A reasonable use case for `map` functions is transforming the values
| represented by an expression using a third-party library.
|
| Parameters
| ----------
| function
| Lambda/function to apply.
| return_dtype
| Datatype of the output Series.
|
| It is recommended to set this whenever possible. If this is `None`, it tries
| to infer the datatype by calling the function with dummy data and looking at
| the output.
| agg_list
| First implode when in a group-by aggregation.
|
| .. deprecated:: 1.32.0
|
| Use `expr.implode().map_batches(..)` instead.
| is_elementwise
| Set to true if the operations is elementwise for better performance
| and optimization.
|
| An elementwise operations has unit or equal length for all inputs
| and can be ran sequentially on slices without results being affected.
| returns_scalar
| If the function returns a scalar, by default it will be wrapped in
| a list in the output, since the assumption is that the function
| always returns something Series-like. If you want to keep the
| result as a scalar, set this argument to True.
|
| Notes
| -----
| A UDF passed to `map_batches` must be pure, meaning that it cannot modify
| or depend on state other than its arguments. Polars may call the function
| with arbitrary input data.
|
| See Also
| --------
| map_elements
| replace
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "sine": [0.0, 1.0, 0.0, -1.0],
| ... "cosine": [1.0, 0.0, -1.0, 0.0],
| ... }
| ... )
| >>> df.select(
| ... pl.all().map_batches(
| ... lambda x: x.to_numpy().argmax(),
| ... returns_scalar=True,
| ... )
| ... )
| shape: (1, 2)
| ┌──────┬────────┐
| │ sine ┆ cosine │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞══════╪════════╡
| │ 1 ┆ 0 │
| └──────┴────────┘
|
| Here's an example of a function that returns a scalar, where we want it
| to stay as a scalar:
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [0, 1, 0, 1],
| ... "b": [1, 2, 3, 4],
| ... }
| ... )
| >>> df.group_by("a").agg(
| ... pl.col("b").map_batches(
| ... lambda x: x.max(), returns_scalar=True, return_dtype=pl.self_dtype()
| ... )
| ... ) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 4 │
| │ 0 ┆ 3 │
| └─────┴─────┘
|
| Call a function that takes multiple arguments by creating a `struct` and
| referencing its fields inside the function call.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": [5, 1, 0, 3],
| ... "b": [4, 2, 3, 4],
| ... }
| ... )
| >>> df.with_columns(
| ... a_times_b=pl.struct("a", "b").map_batches(
| ... lambda x: np.multiply(x.struct.field("a"), x.struct.field("b")),
| ... return_dtype=pl.Int64,
| ... )
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────────┐
| │ a ┆ b ┆ a_times_b │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═══════════╡
| │ 5 ┆ 4 ┆ 20 │
| │ 1 ┆ 2 ┆ 2 │
| │ 0 ┆ 3 ┆ 0 │
| │ 3 ┆ 4 ┆ 12 │
| └─────┴─────┴───────────┘
|
| map_elements(self, function: 'Callable[[Any], Any]', return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None, *, skip_nulls: 'bool' = True, pass_name: 'bool' = False, strategy: 'MapElementsStrategy' = 'thread_local', returns_scalar: 'bool' = False) -> 'Expr'
| Map a custom/user-defined function (UDF) to each element of a column.
|
| .. warning::
| This method is much slower than the native expressions API.
| Only use it if you cannot implement your logic otherwise.
|
| Suppose that the function is: `x ↦ sqrt(x)`:
|
| - For mapping elements of a series, consider:
| `pl.col("col_name").sqrt()`.
| - For mapping inner elements of lists, consider:
| `pl.col("col_name").list.eval(pl.element().sqrt())`.
| - For mapping elements of struct fields, consider:
| `pl.col("col_name").struct.field("field_name").sqrt()`.
|
| If you want to replace the original column or field,
| consider :meth:`.with_columns <polars.DataFrame.with_columns>`
| and :meth:`.with_fields <polars.Expr.struct.with_fields>`.
|
| Parameters
| ----------
| function
| Lambda/function to map.
| return_dtype
| Datatype of the output Series.
|
| It is recommended to set this whenever possible. If this is `None`, it tries
| to infer the datatype by calling the function with dummy data and looking at
| the output.
| skip_nulls
| Don't map the function over values that contain nulls (this is faster).
| pass_name
| Pass the Series name to the custom function (this is more expensive).
| returns_scalar
| .. deprecated:: 1.32.0
| Is ignored and will be removed in 2.0.
| strategy : {'thread_local', 'threading'}
| The threading strategy to use.
|
| - 'thread_local': run the python function on a single thread.
| - 'threading': run the python function on separate threads. Use with
| care as this can slow performance. This might only speed up
| your code if the amount of work per element is significant
| and the python function releases the GIL (e.g. via calling
| a c function)
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Notes
| -----
| * Using `map_elements` is strongly discouraged as you will be effectively
| running python "for" loops, which will be very slow. Wherever possible you
| should prefer the native expression API to achieve the best performance.
|
| * If your function is expensive and you don't want it to be called more than
| once for a given input, consider applying an `@lru_cache` decorator to it.
| If your data is suitable you may achieve *significant* speedups.
|
| * Window function application using `over` is considered a GroupBy context
| here, so `map_elements` can be used to map functions over window groups.
|
| * A UDF passed to `map_elements` must be pure, meaning that it cannot modify or
| depend on state other than its arguments. Polars may call the function
| with arbitrary input data.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 1],
| ... "b": ["a", "b", "c", "c"],
| ... }
| ... )
|
| The function is applied to each element of column `'a'`:
|
| >>> df.with_columns( # doctest: +SKIP
| ... pl.col("a")
| ... .map_elements(lambda x: x * 2, return_dtype=pl.self_dtype())
| ... .alias("a_times_2"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬───────────┐
| │ a ┆ b ┆ a_times_2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ i64 │
| ╞═════╪═════╪═══════════╡
| │ 1 ┆ a ┆ 2 │
| │ 2 ┆ b ┆ 4 │
| │ 3 ┆ c ┆ 6 │
| │ 1 ┆ c ┆ 2 │
| └─────┴─────┴───────────┘
|
| Tip: it is better to implement this with an expression:
|
| >>> df.with_columns(
| ... (pl.col("a") * 2).alias("a_times_2"),
| ... ) # doctest: +IGNORE_RESULT
|
| >>> (
| ... df.lazy()
| ... .group_by("b")
| ... .agg(
| ... pl.col("a")
| ... .implode()
| ... .map_elements(lambda x: x.sum(), return_dtype=pl.Int64)
| ... )
| ... .collect()
| ... ) # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌─────┬─────┐
| │ b ┆ a │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ b ┆ 2 │
| │ c ┆ 4 │
| └─────┴─────┘
|
| Tip: again, it is better to implement this with an expression:
|
| >>> (
| ... df.lazy()
| ... .group_by("b", maintain_order=True)
| ... .agg(pl.col("a").sum())
| ... .collect()
| ... ) # doctest: +IGNORE_RESULT
|
| Window function application using `over` will behave as a GroupBy
| context, with your function receiving individual window groups:
|
| >>> df = pl.DataFrame(
| ... {
| ... "key": ["x", "x", "y", "x", "y", "z"],
| ... "val": [1, 1, 1, 1, 1, 1],
| ... }
| ... )
| >>> df.with_columns(
| ... scaled=pl.col("val")
| ... .implode()
| ... .map_elements(lambda s: s * len(s), return_dtype=pl.List(pl.Int64))
| ... .explode(empty_as_null=False)
| ... .over("key"),
| ... ).sort("key")
| shape: (6, 3)
| ┌─────┬─────┬────────┐
| │ key ┆ val ┆ scaled │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═════╪═════╪════════╡
| │ x ┆ 1 ┆ 3 │
| │ x ┆ 1 ┆ 3 │
| │ x ┆ 1 ┆ 3 │
| │ y ┆ 1 ┆ 2 │
| │ y ┆ 1 ┆ 2 │
| │ z ┆ 1 ┆ 1 │
| └─────┴─────┴────────┘
|
| Note that this function would *also* be better-implemented natively:
|
| >>> df.with_columns(
| ... scaled=(pl.col("val") * pl.col("val").count()).over("key"),
| ... ).sort("key") # doctest: +IGNORE_RESULT
|
| max(self) -> 'Expr'
| Get maximum value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0]})
| >>> df.select(pl.col("a").max())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
|
| max_by(self, by: 'IntoExpr') -> 'Expr'
| Get maximum value, ordered by another expression.
|
| If the by expression has multiple values equal to the maximum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the largest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0], "b": ["x", "y", "z"]})
| >>> df.select(pl.col("b").max_by("a"))
| shape: (1, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ str │
| ╞═════╡
| │ z │
| └─────┘
|
| mean(self) -> 'Expr'
| Get mean value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").mean())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| └─────┘
|
| median(self) -> 'Expr'
| Get median value using linear interpolation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").median())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| └─────┘
|
| min(self) -> 'Expr'
| Get minimum value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0]})
| >>> df.select(pl.col("a").min())
| shape: (1, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ -1.0 │
| └──────┘
|
| min_by(self, by: 'IntoExpr') -> 'Expr'
| Get minimum value, ordered by another expression.
|
| If the by expression has multiple values equal to the minimum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the smallest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1.0, float("nan"), 1.0], "b": ["x", "y", "z"]})
| >>> df.select(pl.col("b").min_by("a"))
| shape: (1, 1)
| ┌─────┐
| │ b │
| │ --- │
| │ str │
| ╞═════╡
| │ x │
| └─────┘
|
| mod(self, other: 'Any') -> 'Expr'
| Method equivalent of modulus operator `expr % other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [0, 1, 2, 3, 4]})
| >>> df.with_columns(pl.col("x").mod(2).alias("x%2"))
| shape: (5, 2)
| ┌─────┬─────┐
| │ x ┆ x%2 │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 0 ┆ 0 │
| │ 1 ┆ 1 │
| │ 2 ┆ 0 │
| │ 3 ┆ 1 │
| │ 4 ┆ 0 │
| └─────┴─────┘
|
| mode(self, *, maintain_order: 'bool' = False) -> 'Expr'
| Compute the most occurring value(s).
|
| Can return multiple Values.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 1, 2, 3],
| ... "b": [1, 1, 2, 2],
| ... }
| ... )
| >>> df.select(pl.all().mode().first()) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 1 │
| └─────┴─────┘
|
| mul(self, other: 'Any') -> 'Expr'
| Method equivalent of multiplication operator `expr * other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 4, 8, 16]})
| >>> df.with_columns(
| ... pl.col("x").mul(2).alias("x*2"),
| ... pl.col("x").mul(pl.col("x").log(2)).alias("x * xlog2"),
| ... )
| shape: (5, 3)
| ┌─────┬─────┬───────────┐
| │ x ┆ x*2 ┆ x * xlog2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪═════╪═══════════╡
| │ 1 ┆ 2 ┆ 0.0 │
| │ 2 ┆ 4 ┆ 2.0 │
| │ 4 ┆ 8 ┆ 8.0 │
| │ 8 ┆ 16 ┆ 24.0 │
| │ 16 ┆ 32 ┆ 64.0 │
| └─────┴─────┴───────────┘
|
| n_unique(self) -> 'Expr'
| Count unique values.
|
| Notes
| -----
| `null` is considered to be a unique value for the purposes of this operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 1, 2, 2, 3], "y": [1, 1, 1, None, None]})
| >>> df.select(
| ... x_unique=pl.col("x").n_unique(),
| ... y_unique=pl.col("y").n_unique(),
| ... )
| shape: (1, 2)
| ┌──────────┬──────────┐
| │ x_unique ┆ y_unique │
| │ --- ┆ --- │
| │ u32 ┆ u32 │
| ╞══════════╪══════════╡
| │ 3 ┆ 2 │
| └──────────┴──────────┘
|
| nan_max(self) -> 'Expr'
| Get maximum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.0, float("nan")]})
| >>> df.select(pl.col("a").nan_max())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ NaN │
| └─────┘
|
| nan_min(self) -> 'Expr'
| Get minimum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.0, float("nan")]})
| >>> df.select(pl.col("a").nan_min())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ NaN │
| └─────┘
|
| ne(self, other: 'Any') -> 'Expr'
| Method equivalent of inequality operator `expr != other`.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0],
| ... "y": [2.0, 2.0, float("nan"), 4.0],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").ne(pl.col("y")).alias("x != y"),
| ... )
| shape: (4, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ y ┆ x != y │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool │
| ╞═════╪═════╪════════╡
| │ 1.0 ┆ 2.0 ┆ true │
| │ 2.0 ┆ 2.0 ┆ false │
| │ NaN ┆ NaN ┆ false │
| │ 4.0 ┆ 4.0 ┆ false │
| └─────┴─────┴────────┘
|
| ne_missing(self, other: 'Any') -> 'Expr'
| Method equivalent of equality operator `expr != other` where `None == None`.
|
| This differs from default `ne` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [1.0, 2.0, float("nan"), 4.0, None, None],
| ... "y": [2.0, 2.0, float("nan"), 4.0, 5.0, None],
| ... }
| ... )
| >>> df.with_columns(
| ... pl.col("x").ne(pl.col("y")).alias("x ne y"),
| ... pl.col("x").ne_missing(pl.col("y")).alias("x ne_missing y"),
| ... )
| shape: (6, 4)
| ┌──────┬──────┬────────┬────────────────┐
| │ x ┆ y ┆ x ne y ┆ x ne_missing y │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ bool ┆ bool │
| ╞══════╪══════╪════════╪════════════════╡
| │ 1.0 ┆ 2.0 ┆ true ┆ true │
| │ 2.0 ┆ 2.0 ┆ false ┆ false │
| │ NaN ┆ NaN ┆ false ┆ false │
| │ 4.0 ┆ 4.0 ┆ false ┆ false │
| │ null ┆ 5.0 ┆ null ┆ true │
| │ null ┆ null ┆ null ┆ false │
| └──────┴──────┴────────┴────────────────┘
|
| neg(self) -> 'Expr'
| Method equivalent of unary minus operator `-expr`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 2, None]})
| >>> df.with_columns(pl.col("a").neg())
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 1 │
| │ 0 │
| │ -2 │
| │ null │
| └──────┘
|
| not_(self) -> 'Expr'
| Method equivalent of bitwise "not" operator `~expr`.
|
| This has the effect of negating logical boolean expressions,
| but operates bitwise on integers.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "label": ["aa", "bb", "cc", "dd", "ee"],
| ... "valid": [True, False, None, False, True],
| ... "int_code": [1, 0, 2, None, -1],
| ... }
| ... )
|
| Apply "not" to boolean expression (negates the value) and
| integer expression (operates bitwise):
|
| >>> df.with_columns(
| ... not_valid=pl.col("valid").not_(),
| ... not_int_code=pl.col("int_code").not_(),
| ... )
| shape: (5, 5)
| ┌───────┬───────┬──────────┬───────────┬──────────────┐
| │ label ┆ valid ┆ int_code ┆ not_valid ┆ not_int_code │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ bool ┆ i64 ┆ bool ┆ i64 │
| ╞═══════╪═══════╪══════════╪═══════════╪══════════════╡
| │ aa ┆ true ┆ 1 ┆ false ┆ -2 │
| │ bb ┆ false ┆ 0 ┆ true ┆ -1 │
| │ cc ┆ null ┆ 2 ┆ null ┆ -3 │
| │ dd ┆ false ┆ null ┆ true ┆ null │
| │ ee ┆ true ┆ -1 ┆ false ┆ 0 │
| └───────┴───────┴──────────┴───────────┴──────────────┘
|
| null_count(self) -> 'Expr'
| Count null values.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [None, 1, None],
| ... "b": [10, None, 300],
| ... "c": [350, 650, 850],
| ... }
| ... )
| >>> df.select(pl.all().null_count())
| shape: (1, 3)
| ┌─────┬─────┬─────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞═════╪═════╪═════╡
| │ 2 ┆ 1 ┆ 0 │
| └─────┴─────┴─────┘
|
| or_(self, *others: 'Any') -> 'Expr'
| Method equivalent of bitwise "or" operator `expr | other | ...`.
|
| This has the effect of combining logical boolean expressions,
| but operates bitwise on integers.
|
| Parameters
| ----------
| *others
| One or more integer or boolean expressions to evaluate/combine.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={
| ... "x": [5, 6, 7, 4, 8],
| ... "y": [1.5, 2.5, 1.0, 4.0, -5.75],
| ... "z": [-9, 2, -1, 4, 8],
| ... }
| ... )
|
| Combine logical "or" conditions:
|
| >>> df.select(
| ... (pl.col("x") == pl.col("y"))
| ... .or_(
| ... pl.col("x") == pl.col("y"),
| ... pl.col("y") == pl.col("z"),
| ... pl.col("y").cast(int) == pl.col("z"),
| ... )
| ... .alias("any")
| ... )
| shape: (5, 1)
| ┌───────┐
| │ any │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ true │
| │ false │
| │ true │
| │ false │
| └───────┘
|
| Bitwise "or" operation on integer columns:
|
| >>> df.select("x", "z", x_or_z=pl.col("x").or_(pl.col("z")))
| shape: (5, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ z ┆ x_or_z │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪════════╡
| │ 5 ┆ -9 ┆ -9 │
| │ 6 ┆ 2 ┆ 6 │
| │ 7 ┆ -1 ┆ -1 │
| │ 4 ┆ 4 ┆ 4 │
| │ 8 ┆ 8 ┆ 8 │
| └─────┴─────┴────────┘
|
| over(self, partition_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, *more_exprs: 'IntoExpr', order_by: 'IntoExpr | Iterable[IntoExpr] | None' = None, descending: 'bool' = False, nulls_last: 'bool' = False, mapping_strategy: 'WindowMappingStrategy' = 'group_to_rows') -> 'Expr'
| Compute expressions over the given groups.
|
| This expression is similar to performing a group by aggregation and joining the
| result back into the original DataFrame.
|
| The outcome is similar to how `window functions
| <https://www.postgresql.org/docs/current/tutorial-window.html>`_
| work in PostgreSQL.
|
| Parameters
| ----------
| partition_by
| Column(s) to group by. Accepts expression input. Strings are parsed as
| column names.
| *more_exprs
| Additional columns to group by, specified as positional arguments.
| order_by
| Order rows within each partition group before evaluating the expression.
| Useful for order-sensitive operations such as
| :func:`cum_sum` or :func:`diff`.
| descending
| In case 'order_by' is given, indicate whether to order in
| ascending or descending order.
| nulls_last
| In case 'order_by' is given, indicate whether to order
| the nulls in last position.
| mapping_strategy: {'group_to_rows', 'join', 'explode'}
| - group_to_rows
| If the aggregation results in multiple values per group, map them back
| to their row position in the DataFrame. This can only be done if each
| group yields the same number of elements before aggregation as after. If
| the aggregation results in one scalar value per group, this value will
| be mapped to every row.
| - join
| If the aggregation may result in multiple values per group, join the
| values as 'List<group_dtype>' to each row position. Warning: this can be
| memory intensive. If the aggregation always results in one scalar value
| per group, join this value as '<group_dtype>' to each row position.
| - explode
| If the aggregation may result in multiple values per group, map each
| value to a new row, similar to the results of `group_by` + `agg` +
| `explode`. If the aggregation always results in one scalar value per
| group, map this value to one row position. Sorting of the given groups
| is required if the groups are not part of the window operation for the
| operation, otherwise the result would not make sense. This operation
| changes the number of rows.
|
| Examples
| --------
| Pass the name of a column to compute the expression over that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["a", "a", "b", "b", "b"],
| ... "b": [1, 2, 3, 5, 3],
| ... "c": [5, 4, 3, 2, 1],
| ... }
| ... )
| >>> df.with_columns(c_max=pl.col("c").max().over("a"))
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────┐
| │ a ┆ b ┆ c ┆ c_max │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ a ┆ 1 ┆ 5 ┆ 5 │
| │ a ┆ 2 ┆ 4 ┆ 5 │
| │ b ┆ 3 ┆ 3 ┆ 3 │
| │ b ┆ 5 ┆ 2 ┆ 3 │
| │ b ┆ 3 ┆ 1 ┆ 3 │
| └─────┴─────┴─────┴───────┘
|
| Expression input is also supported.
|
| >>> df.with_columns(c_max=pl.col("c").max().over(pl.col("b") // 2))
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────┐
| │ a ┆ b ┆ c ┆ c_max │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ a ┆ 1 ┆ 5 ┆ 5 │
| │ a ┆ 2 ┆ 4 ┆ 4 │
| │ b ┆ 3 ┆ 3 ┆ 4 │
| │ b ┆ 5 ┆ 2 ┆ 2 │
| │ b ┆ 3 ┆ 1 ┆ 4 │
| └─────┴─────┴─────┴───────┘
|
| Group by multiple columns by passing multiple column names or expressions.
|
| >>> df.with_columns(c_min=pl.col("c").min().over("a", pl.col("b") % 2))
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────┐
| │ a ┆ b ┆ c ┆ c_min │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪═════╪═══════╡
| │ a ┆ 1 ┆ 5 ┆ 5 │
| │ a ┆ 2 ┆ 4 ┆ 4 │
| │ b ┆ 3 ┆ 3 ┆ 1 │
| │ b ┆ 5 ┆ 2 ┆ 1 │
| │ b ┆ 3 ┆ 1 ┆ 1 │
| └─────┴─────┴─────┴───────┘
|
| Mapping strategy `join` joins the values by group.
|
| >>> df.with_columns(
| ... c_pairs=pl.col("c").head(2).over("a", mapping_strategy="join")
| ... )
| shape: (5, 4)
| ┌─────┬─────┬─────┬───────────┐
| │ a ┆ b ┆ c ┆ c_pairs │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 ┆ list[i64] │
| ╞═════╪═════╪═════╪═══════════╡
| │ a ┆ 1 ┆ 5 ┆ [5, 4] │
| │ a ┆ 2 ┆ 4 ┆ [5, 4] │
| │ b ┆ 3 ┆ 3 ┆ [3, 2] │
| │ b ┆ 5 ┆ 2 ┆ [3, 2] │
| │ b ┆ 3 ┆ 1 ┆ [3, 2] │
| └─────┴─────┴─────┴───────────┘
|
| Mapping strategy `explode` maps the values to new rows, changing the shape.
|
| >>> df.select(
| ... c_first_2=pl.col("c").head(2).over("a", mapping_strategy="explode")
| ... )
| shape: (4, 1)
| ┌───────────┐
| │ c_first_2 │
| │ --- │
| │ i64 │
| ╞═══════════╡
| │ 5 │
| │ 4 │
| │ 3 │
| │ 2 │
| └───────────┘
|
| You can use non-elementwise expressions with `over` too. By default they are
| evaluated using row-order, but you can specify a different one using `order_by`.
|
| >>> from datetime import date
| >>> df = pl.DataFrame(
| ... {
| ... "store_id": ["a", "a", "b", "b"],
| ... "date": [
| ... date(2024, 9, 18),
| ... date(2024, 9, 17),
| ... date(2024, 9, 18),
| ... date(2024, 9, 16),
| ... ],
| ... "sales": [7, 9, 8, 10],
| ... }
| ... )
| >>> df.with_columns(
| ... cumulative_sales=pl.col("sales")
| ... .cum_sum()
| ... .over("store_id", order_by="date")
| ... )
| shape: (4, 4)
| ┌──────────┬────────────┬───────┬──────────────────┐
| │ store_id ┆ date ┆ sales ┆ cumulative_sales │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 ┆ i64 │
| ╞══════════╪════════════╪═══════╪══════════════════╡
| │ a ┆ 2024-09-18 ┆ 7 ┆ 16 │
| │ a ┆ 2024-09-17 ┆ 9 ┆ 9 │
| │ b ┆ 2024-09-18 ┆ 8 ┆ 18 │
| │ b ┆ 2024-09-16 ┆ 10 ┆ 10 │
| └──────────┴────────────┴───────┴──────────────────┘
|
| If you don't require that the group order be preserved, then the more performant
| option is to use `mapping_strategy='explode'` - be careful however to only ever
| use this in a `select` statement, not a `with_columns` one.
|
| >>> window = {
| ... "partition_by": "store_id",
| ... "order_by": "date",
| ... "mapping_strategy": "explode",
| ... }
| >>> df.select(
| ... pl.all().over(**window),
| ... cumulative_sales=pl.col("sales").cum_sum().over(**window),
| ... )
| shape: (4, 4)
| ┌──────────┬────────────┬───────┬──────────────────┐
| │ store_id ┆ date ┆ sales ┆ cumulative_sales │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ str ┆ date ┆ i64 ┆ i64 │
| ╞══════════╪════════════╪═══════╪══════════════════╡
| │ a ┆ 2024-09-17 ┆ 9 ┆ 9 │
| │ a ┆ 2024-09-18 ┆ 7 ┆ 16 │
| │ b ┆ 2024-09-16 ┆ 10 ┆ 10 │
| │ b ┆ 2024-09-18 ┆ 8 ┆ 18 │
| └──────────┴────────────┴───────┴──────────────────┘
|
| pct_change(self, n: 'int | IntoExprColumn' = 1) -> 'Expr'
| Computes percentage change between values.
|
| Percentage change (as fraction) between current element and most-recent
| non-null element at least `n` period(s) before the current element.
|
| Computes the change from the previous row by default.
|
| Parameters
| ----------
| n
| periods to shift for forming percent change.
|
| Notes
| -----
| Null values are preserved. If you're coming from pandas, this matches
| their ``fill_method=None`` behaviour.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [10, 11, 12, None, 12],
| ... }
| ... )
| >>> df.with_columns(pl.col("a").pct_change().alias("pct_change"))
| shape: (5, 2)
| ┌──────┬────────────┐
| │ a ┆ pct_change │
| │ --- ┆ --- │
| │ i64 ┆ f64 │
| ╞══════╪════════════╡
| │ 10 ┆ null │
| │ 11 ┆ 0.1 │
| │ 12 ┆ 0.090909 │
| │ null ┆ null │
| │ 12 ┆ null │
| └──────┴────────────┘
|
| peak_max(self) -> 'Expr'
| Get a boolean mask of the local maximum peaks.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4, 5]})
| >>> df.select(pl.col("a").peak_max())
| shape: (5, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ false │
| │ false │
| │ false │
| │ true │
| └───────┘
|
| peak_min(self) -> 'Expr'
| Get a boolean mask of the local minimum peaks.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [4, 1, 3, 2, 5]})
| >>> df.select(pl.col("a").peak_min())
| shape: (5, 1)
| ┌───────┐
| │ a │
| │ --- │
| │ bool │
| ╞═══════╡
| │ false │
| │ true │
| │ false │
| │ true │
| │ false │
| └───────┘
|
| pipe(self, function: 'Callable[Concatenate[Expr, P], T]', *args: 'P.args', **kwargs: 'P.kwargs') -> 'T'
| Offers a structured way to apply a sequence of user-defined functions (UDFs).
|
| Parameters
| ----------
| function
| Callable; will receive the expression as the first parameter,
| followed by any given args/kwargs.
| *args
| Arguments to pass to the UDF.
| **kwargs
| Keyword arguments to pass to the UDF.
|
| Examples
| --------
| >>> def extract_number(expr: pl.Expr) -> pl.Expr:
| ... """Extract the digits from a string."""
| ... return expr.str.extract(r"\d+", 0).cast(pl.Int64)
| >>>
| >>> def scale_negative_even(expr: pl.Expr, *, n: int = 1) -> pl.Expr:
| ... """Set even numbers negative, and scale by a user-supplied value."""
| ... expr = pl.when(expr % 2 == 0).then(-expr).otherwise(expr)
| ... return expr * n
| >>>
| >>> df = pl.DataFrame({"val": ["a: 1", "b: 2", "c: 3", "d: 4"]})
| >>> df.with_columns(
| ... udfs=(
| ... pl.col("val").pipe(extract_number).pipe(scale_negative_even, n=5)
| ... ),
| ... )
| shape: (4, 2)
| ┌──────┬──────┐
| │ val ┆ udfs │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞══════╪══════╡
| │ a: 1 ┆ 5 │
| │ b: 2 ┆ -10 │
| │ c: 3 ┆ 15 │
| │ d: 4 ┆ -20 │
| └──────┴──────┘
|
| pow(self, exponent: 'IntoExprColumn | int | float') -> 'Expr'
| Method equivalent of exponentiation operator `expr ** exponent`.
|
| If the exponent is float, the result follows the dtype of exponent.
| Otherwise, it follows dtype of base.
|
| Parameters
| ----------
| exponent
| Numeric literal or expression exponent value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [1, 2, 4, 8]})
| >>> df.with_columns(
| ... pl.col("x").pow(3).alias("cube"),
| ... pl.col("x").pow(pl.col("x").log(2)).alias("x ** xlog2"),
| ... )
| shape: (4, 3)
| ┌─────┬──────┬────────────┐
| │ x ┆ cube ┆ x ** xlog2 │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪══════╪════════════╡
| │ 1 ┆ 1 ┆ 1.0 │
| │ 2 ┆ 8 ┆ 2.0 │
| │ 4 ┆ 64 ┆ 16.0 │
| │ 8 ┆ 512 ┆ 512.0 │
| └─────┴──────┴────────────┘
|
| Raising an integer to a positive integer results in an integer - in order
| to raise to a negative integer, you can cast either the base or the exponent
| to float first:
|
| >>> df.with_columns(
| ... x_squared=pl.col("x").pow(2),
| ... x_inverse=pl.col("x").pow(-1.0),
| ... )
| shape: (4, 3)
| ┌─────┬───────────┬───────────┐
| │ x ┆ x_squared ┆ x_inverse │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪═══════════╪═══════════╡
| │ 1 ┆ 1 ┆ 1.0 │
| │ 2 ┆ 4 ┆ 0.5 │
| │ 4 ┆ 16 ┆ 0.25 │
| │ 8 ┆ 64 ┆ 0.125 │
| └─────┴───────────┴───────────┘
|
| product(self) -> 'Expr'
| Compute the product of an expression.
|
| Notes
| -----
| If there are no non-null values, then the output is `1`.
| If you would prefer empty products to return `None`, you can
| use `pl.when(expr.count()>0).then(expr.product())` instead
| of `expr.product()`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").product())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 6 │
| └─────┘
|
| qcut(self, quantiles: 'Sequence[float] | int', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, allow_duplicates: 'bool' = False, include_breaks: 'bool' = False) -> 'Expr'
| Bin continuous values into discrete categories based on their quantiles.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| quantiles
| Either a list of quantile probabilities between 0 and 1 or a positive
| integer determining the number of bins with uniform probability.
| labels
| Names of the categories. The number of labels must be equal to the number
| of categories.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| allow_duplicates
| If set to `True`, duplicates in the resulting quantiles are dropped,
| rather than raising a `DuplicateError`. This can happen even with unique
| probabilities, depending on the data.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from a
| :class:`Categorical` to a :class:`Struct`.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Categorical` if `include_breaks` is set to
| `False` (default), otherwise an expression of data type :class:`Struct`.
|
| See Also
| --------
| cut
|
| Examples
| --------
| Divide a column into three categories according to pre-defined quantile
| probabilities.
|
| >>> df = pl.DataFrame({"foo": [-2, -1, 0, 1, 2]})
| >>> df.with_columns(
| ... pl.col("foo").qcut([0.25, 0.75], labels=["a", "b", "c"]).alias("qcut")
| ... )
| shape: (5, 2)
| ┌─────┬──────┐
| │ foo ┆ qcut │
| │ --- ┆ --- │
| │ i64 ┆ cat │
| ╞═════╪══════╡
| │ -2 ┆ a │
| │ -1 ┆ a │
| │ 0 ┆ b │
| │ 1 ┆ b │
| │ 2 ┆ c │
| └─────┴──────┘
|
| Divide a column into two categories using uniform quantile probabilities.
|
| >>> df.with_columns(
| ... pl.col("foo")
| ... .qcut(2, labels=["low", "high"], left_closed=True)
| ... .alias("qcut")
| ... )
| shape: (5, 2)
| ┌─────┬──────┐
| │ foo ┆ qcut │
| │ --- ┆ --- │
| │ i64 ┆ cat │
| ╞═════╪══════╡
| │ -2 ┆ low │
| │ -1 ┆ low │
| │ 0 ┆ high │
| │ 1 ┆ high │
| │ 2 ┆ high │
| └─────┴──────┘
|
| Add both the category and the breakpoint.
|
| >>> df.with_columns(
| ... pl.col("foo").qcut([0.25, 0.75], include_breaks=True).alias("qcut")
| ... ).unnest("qcut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ cat │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| quantile(self, quantile: 'float | list_[float] | Expr', interpolation: 'QuantileMethod' = 'nearest') -> 'Expr'
| Get quantile value.
|
| Parameters
| ----------
| quantile
| Quantile(s) between 0.0 and 1.0. Can be a single float or a list of floats.
|
| - If a single float, returns a single f64 value per row.
| - If a list of floats, returns a list of f64 values per row (one value per quantile).
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0, 1, 2, 3, 4, 5]})
| >>> df.select(pl.col("a").quantile(0.3))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 2.0 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="higher"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 2.0 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="lower"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="midpoint"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.5 │
| └─────┘
| >>> df.select(pl.col("a").quantile(0.3, interpolation="linear"))
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.5 │
| └─────┘
| >>> df.select(pl.col("a").quantile([0.25, 0.75], interpolation="linear"))
| shape: (1, 1)
| ┌──────────────┐
| │ a │
| │ --- │
| │ list[f64] │
| ╞══════════════╡
| │ [1.25, 3.75] │
| └──────────────┘
|
| radians(self) -> 'Expr'
| Convert from degrees to radians.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-720, -540, -360, -180, 0, 180, 360, 540, 720]})
| >>> df.select(pl.col("a").radians())
| shape: (9, 1)
| ┌────────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞════════════╡
| │ -12.566371 │
| │ -9.424778 │
| │ -6.283185 │
| │ -3.141593 │
| │ 0.0 │
| │ 3.141593 │
| │ 6.283185 │
| │ 9.424778 │
| │ 12.566371 │
| └────────────┘
|
| rank(self, method: 'RankMethod' = 'average', *, descending: 'bool' = False, seed: 'int | None' = None) -> 'Expr'
| Assign ranks to data, dealing with ties appropriately.
|
| Parameters
| ----------
| method : {'average', 'min', 'max', 'dense', 'ordinal', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'ordinal' : All values are given a distinct rank, corresponding to
| the order that the values occur in the Series.
| - 'random' : Like 'ordinal', but the rank for ties is not dependent
| on the order that the values occur in the Series.
| descending
| Rank in descending order.
| seed
| If `method="random"`, use this as seed.
|
| Notes
| -----
| If you're coming from SQL, you may be expecting null values to be ranked last.
| Polars, however, only ranks non-null values and preserves the null ones.
|
| Examples
| --------
| The 'average' method:
|
| >>> df = pl.DataFrame({"a": [3, 6, 1, 1, 6]})
| >>> df.select(pl.col("a").rank())
| shape: (5, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 3.0 │
| │ 4.5 │
| │ 1.5 │
| │ 1.5 │
| │ 4.5 │
| └─────┘
|
| The 'ordinal' method:
|
| >>> df = pl.DataFrame({"a": [3, 6, 1, 1, 6]})
| >>> df.select(pl.col("a").rank("ordinal"))
| shape: (5, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 3 │
| │ 4 │
| │ 1 │
| │ 2 │
| │ 5 │
| └─────┘
|
| Use 'rank' with 'over' to rank within groups:
|
| >>> df = pl.DataFrame({"a": [1, 1, 2, 2, 2], "b": [6, 7, 5, 14, 11]})
| >>> df.with_columns(pl.col("b").rank().over("a").alias("rank"))
| shape: (5, 3)
| ┌─────┬─────┬──────┐
| │ a ┆ b ┆ rank │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ f64 │
| ╞═════╪═════╪══════╡
| │ 1 ┆ 6 ┆ 1.0 │
| │ 1 ┆ 7 ┆ 2.0 │
| │ 2 ┆ 5 ┆ 1.0 │
| │ 2 ┆ 14 ┆ 3.0 │
| │ 2 ┆ 11 ┆ 2.0 │
| └─────┴─────┴──────┘
|
| Divide by the length or number of non-null values
| to compute the percentile rank.
|
| >>> df = pl.DataFrame({"a": [6, 7, None, 14, 11]})
| >>> df.with_columns(
| ... pct=pl.col("a").rank() / pl.len(),
| ... pct_valid=pl.col("a").rank() / pl.count("a"),
| ... )
| shape: (5, 3)
| ┌──────┬──────┬───────────┐
| │ a ┆ pct ┆ pct_valid │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 │
| ╞══════╪══════╪═══════════╡
| │ 6 ┆ 0.2 ┆ 0.25 │
| │ 7 ┆ 0.4 ┆ 0.5 │
| │ null ┆ null ┆ null │
| │ 14 ┆ 0.8 ┆ 1.0 │
| │ 11 ┆ 0.6 ┆ 0.75 │
| └──────┴──────┴───────────┘
|
| rechunk(self) -> 'Expr'
| Create a single chunk of memory for this Series.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
|
| Create a Series with 3 nulls, append column `a`, then rechunk.
|
| >>> df.select(pl.repeat(None, 3).append(pl.col("a")).rechunk())
| shape: (6, 1)
| ┌────────┐
| │ repeat │
| │ --- │
| │ i64 │
| ╞════════╡
| │ null │
| │ null │
| │ null │
| │ 1 │
| │ 1 │
| │ 2 │
| └────────┘
|
| register_plugin(self, *, lib: 'str_', symbol: 'str_', args: 'list_[IntoExpr] | None' = None, kwargs: 'dict[Any, Any] | None' = None, is_elementwise: 'bool' = False, input_wildcard_expansion: 'bool' = False, returns_scalar: 'bool' = False, cast_to_supertypes: 'bool' = False, pass_name_to_apply: 'bool' = False, changes_length: 'bool' = False) -> 'Expr'
| Register a plugin function.
|
| .. deprecated:: 0.20.16
| Use :func:`polars.plugins.register_plugin_function` instead.
|
| See the `user guide <https://docs.pola.rs/user-guide/plugins/>`_
| for more information about plugins.
|
| Warnings
| --------
| This method is deprecated. Use the new `polars.plugins.register_plugin_function`
| function instead.
|
| This is highly unsafe as this will call the C function loaded by
| `lib::symbol`.
|
| The parameters you set dictate how Polars will handle the function.
| Make sure they are correct!
|
| Parameters
| ----------
| lib
| Library to load.
| symbol
| Function to load.
| args
| Arguments (other than self) passed to this function.
| These arguments have to be of type Expression.
| kwargs
| Non-expression arguments. They must be JSON serializable.
| is_elementwise
| If the function only operates on scalars
| this will trigger fast paths.
| input_wildcard_expansion
| Expand expressions as input of this function.
| returns_scalar
| Automatically explode on unit length if it ran as final aggregation.
| this is the case for aggregations like `sum`, `min`, `covariance` etc.
| cast_to_supertypes
| Cast the input datatypes to their supertype.
| pass_name_to_apply
| if set, then the `Series` passed to the function in the group_by operation
| will ensure the name is set. This is an extra heap allocation per group.
| changes_length
| For example a `unique` or a `slice`
|
| reinterpret(self, *, signed: 'bool | None' = None, dtype: 'PolarsDataType | None' = None) -> 'Expr'
| Reinterpret the underlying bits as a signed/unsigned integer or float.
|
| This operation is only allowed for numeric types of the same size.
| For lower bits numbers, you can safely use the cast operation.
|
| Either `signed` or `dtype` can be specified.
| Defaults to `signed=True` otherwise.
|
| Parameters
| ----------
| signed
| If True, reinterpret as signed integer. Otherwise, reinterpret
| as unsigned integer.
| dtype
| DataType to reinterpret to.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 1, 2], dtype=pl.UInt64)
| >>> df = pl.DataFrame([s])
| >>> df.select(
| ... [
| ... pl.col("a").reinterpret(dtype=pl.Int64).alias("reinterpreted"),
| ... pl.col("a").alias("original"),
| ... ]
| ... )
| shape: (3, 2)
| ┌───────────────┬──────────┐
| │ reinterpreted ┆ original │
| │ --- ┆ --- │
| │ i64 ┆ u64 │
| ╞═══════════════╪══════════╡
| │ 1 ┆ 1 │
| │ 1 ┆ 1 │
| │ 2 ┆ 2 │
| └───────────────┴──────────┘
|
| repeat_by(self, by: 'pl.Series | Expr | str_ | int') -> 'Expr'
| Repeat the elements in this Series as specified in the given expression.
|
| The repeated elements are expanded into a `List`.
|
| Parameters
| ----------
| by
| Numeric column that determines how often the values will be repeated.
| The column will be coerced to UInt32. Give this dtype to make the coercion a
| no-op.
|
| Returns
| -------
| Expr
| Expression of data type :class:`List`, where the inner data type is equal
| to the original data type.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": ["x", "y", "z"],
| ... "n": [1, 2, 3],
| ... }
| ... )
| >>> df.select(pl.col("a").repeat_by("n"))
| shape: (3, 1)
| ┌─────────────────┐
| │ a │
| │ --- │
| │ list[str] │
| ╞═════════════════╡
| │ ["x"] │
| │ ["y", "y"] │
| │ ["z", "z", "z"] │
| └─────────────────┘
|
| replace(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | None' = None) -> 'Expr'
| Replace the given values by different values of the same data type.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Length must match the length of `old` or have length 1.
|
| default
| Set values that were not replaced to this value.
| Defaults to keeping the original value.
| Accepts expression input. Non-expression inputs are parsed as literals.
|
| .. deprecated:: 1.0.0
| Use :meth:`replace_strict` instead to set a default while replacing
| values.
|
| return_dtype
| The data type of the resulting expression. If set to `None` (default),
| the data type of the original column is preserved.
|
| .. deprecated:: 1.0.0
| Use :meth:`replace_strict` instead to set a return data type while
| replacing values, or explicitly call :meth:`cast` on the output.
|
| See Also
| --------
| replace_strict
| str.replace
|
| Examples
| --------
| Replace a single value by another value. Values that were not replaced remain
| unchanged.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3]})
| >>> df.with_columns(replaced=pl.col("a").replace(2, 100))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 100 │
| │ 2 ┆ 100 │
| │ 3 ┆ 3 │
| └─────┴──────────┘
|
| Replace multiple values by passing sequences to the `old` and `new` parameters.
|
| >>> df.with_columns(replaced=pl.col("a").replace([2, 3], [100, 200]))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 100 │
| │ 2 ┆ 100 │
| │ 3 ┆ 200 │
| └─────┴──────────┘
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {2: 100, 3: 200}
| >>> df.with_columns(replaced=pl.col("a").replace(mapping))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 1 │
| │ 2 ┆ 100 │
| │ 2 ┆ 100 │
| │ 3 ┆ 200 │
| └─────┴──────────┘
|
| The original data type is preserved when replacing by values of a different
| data type. Use :meth:`replace_strict` to replace and change the return data
| type.
|
| >>> df = pl.DataFrame({"a": ["x", "y", "z"]})
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> df.with_columns(replaced=pl.col("a").replace(mapping))
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
|
| Expression input is supported.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3], "b": [1.5, 2.5, 5.0, 1.0]})
| >>> df.with_columns(
| ... replaced=pl.col("a").replace(
| ... old=pl.col("a").max(),
| ... new=pl.col("b").sum(),
| ... )
| ... )
| shape: (4, 3)
| ┌─────┬─────┬──────────┐
| │ a ┆ b ┆ replaced │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ i64 │
| ╞═════╪═════╪══════════╡
| │ 1 ┆ 1.5 ┆ 1 │
| │ 2 ┆ 2.5 ┆ 2 │
| │ 2 ┆ 5.0 ┆ 2 │
| │ 3 ┆ 1.0 ┆ 10 │
| └─────┴─────┴──────────┘
|
| replace_strict(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | pl.DataTypeExpr | None' = None) -> 'Expr'
| Replace all values by different values.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace_strict(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Accepts expression input. Sequences are parsed as Series,
| other non-expression inputs are parsed as literals.
| Length must match the length of `old` or have length 1.
| default
| Set values that were not replaced to this value. If no default is specified,
| (default), an error is raised if any values were not replaced.
| Accepts expression input. Non-expression inputs are parsed as literals.
| return_dtype
| The data type of the resulting expression. If set to `None` (default),
| the data type is determined automatically based on the other inputs.
|
| Raises
| ------
| InvalidOperationError
| If any non-null values in the original column were not replaced, and no
| `default` was specified.
|
| See Also
| --------
| replace
| str.replace
|
| Examples
| --------
| Replace values by passing sequences to the `old` and `new` parameters.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3]})
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict([1, 2, 3], [100, 200, 300])
| ... )
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 100 │
| │ 2 ┆ 200 │
| │ 2 ┆ 200 │
| │ 3 ┆ 300 │
| └─────┴──────────┘
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {1: 100, 2: 200, 3: 300}
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ 100 │
| │ 2 ┆ 200 │
| │ 2 ┆ 200 │
| │ 3 ┆ 300 │
| └─────┴──────────┘
|
| By default, an error is raised if any non-null values were not replaced.
| Specify a default to set all values that were not matched.
|
| >>> mapping = {2: 200, 3: 300}
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict(mapping)
| ... ) # doctest: +SKIP
| Traceback (most recent call last):
| ...
| polars.exceptions.InvalidOperationError: incomplete mapping specified for `replace_strict`
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping, default=-1))
| shape: (4, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪══════════╡
| │ 1 ┆ -1 │
| │ 2 ┆ 200 │
| │ 2 ┆ 200 │
| │ 3 ┆ 300 │
| └─────┴──────────┘
|
| Replacing by values of a different data type sets the return type based on
| a combination of the `new` data type and the `default` data type.
|
| >>> df = pl.DataFrame({"a": ["x", "y", "z"]})
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping))
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ i64 │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
| >>> df.with_columns(replaced=pl.col("a").replace_strict(mapping, default="x"))
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
|
| Set the `return_dtype` parameter to control the resulting data type directly.
|
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict(mapping, return_dtype=pl.UInt8)
| ... )
| shape: (3, 2)
| ┌─────┬──────────┐
| │ a ┆ replaced │
| │ --- ┆ --- │
| │ str ┆ u8 │
| ╞═════╪══════════╡
| │ x ┆ 1 │
| │ y ┆ 2 │
| │ z ┆ 3 │
| └─────┴──────────┘
|
| Expression input is supported for all parameters.
|
| >>> df = pl.DataFrame({"a": [1, 2, 2, 3], "b": [1.5, 2.5, 5.0, 1.0]})
| >>> df.with_columns(
| ... replaced=pl.col("a").replace_strict(
| ... old=pl.col("a").max(),
| ... new=pl.col("b").sum(),
| ... default=pl.col("b"),
| ... )
| ... )
| shape: (4, 3)
| ┌─────┬─────┬──────────┐
| │ a ┆ b ┆ replaced │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 │
| ╞═════╪═════╪══════════╡
| │ 1 ┆ 1.5 ┆ 1.5 │
| │ 2 ┆ 2.5 ┆ 2.5 │
| │ 2 ┆ 5.0 ┆ 5.0 │
| │ 3 ┆ 1.0 ┆ 10.0 │
| └─────┴─────┴──────────┘
|
| reshape(self, dimensions: 'tuple[int, ...]') -> 'Expr'
| Reshape this Expr to a flat column or an Array column.
|
| Parameters
| ----------
| dimensions
| Tuple of the dimension sizes. If -1 is used as the value for the
| first dimension, that dimension is inferred.
| Because the size of the Column may not be known in advance, it is
| only possible to use -1 for the first dimension.
|
| Returns
| -------
| Expr
| If a single dimension is given, results in an expression of the original
| data type.
| If a multiple dimensions are given, results in an expression of data type
| :class:`Array` with shape `dimensions`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7, 8, 9]})
| >>> square = df.select(pl.col("foo").reshape((3, 3)))
| >>> square
| shape: (3, 1)
| ┌───────────────┐
| │ foo │
| │ --- │
| │ array[i64, 3] │
| ╞═══════════════╡
| │ [1, 2, 3] │
| │ [4, 5, 6] │
| │ [7, 8, 9] │
| └───────────────┘
| >>> square.select(pl.col("foo").reshape((9,)))
| shape: (9, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| │ 4 │
| │ 5 │
| │ 6 │
| │ 7 │
| │ 8 │
| │ 9 │
| └─────┘
|
| See Also
| --------
| Expr.list.explode : Explode a list column.
|
| reverse(self) -> 'Expr'
| Reverse the selection.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "A": [1, 2, 3, 4, 5],
| ... "fruits": ["banana", "banana", "apple", "apple", "banana"],
| ... "B": [5, 4, 3, 2, 1],
| ... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"],
| ... }
| ... )
| >>> df.select(
| ... [
| ... pl.all(),
| ... pl.all().reverse().name.suffix("_reverse"),
| ... ]
| ... )
| shape: (5, 8)
| ┌─────┬────────┬─────┬────────┬───────────┬────────────────┬───────────┬──────────────┐
| │ A ┆ fruits ┆ B ┆ cars ┆ A_reverse ┆ fruits_reverse ┆ B_reverse ┆ cars_reverse │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ i64 ┆ str ┆ i64 ┆ str ┆ i64 ┆ str │
| ╞═════╪════════╪═════╪════════╪═══════════╪════════════════╪═══════════╪══════════════╡
| │ 1 ┆ banana ┆ 5 ┆ beetle ┆ 5 ┆ banana ┆ 1 ┆ beetle │
| │ 2 ┆ banana ┆ 4 ┆ audi ┆ 4 ┆ apple ┆ 2 ┆ beetle │
| │ 3 ┆ apple ┆ 3 ┆ beetle ┆ 3 ┆ apple ┆ 3 ┆ beetle │
| │ 4 ┆ apple ┆ 2 ┆ beetle ┆ 2 ┆ banana ┆ 4 ┆ audi │
| │ 5 ┆ banana ┆ 1 ┆ beetle ┆ 1 ┆ banana ┆ 5 ┆ beetle │
| └─────┴────────┴─────┴────────┴───────────┴────────────────┴───────────┴──────────────┘
|
| rle(self) -> 'Expr'
| Compress the column data using run-length encoding.
|
| Run-length encoding (RLE) encodes data by storing each *run* of identical values
| as a single value and its length.
|
| Returns
| -------
| Expr
| Expression of data type `Struct` with fields `len` of data type `UInt32`
| and `value` of the original data type.
|
| See Also
| --------
| rle_id
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2, 1, None, 1, 3, 3]})
| >>> df.select(pl.col("a").rle()).unnest("a")
| shape: (6, 2)
| ┌─────┬───────┐
| │ len ┆ value │
| │ --- ┆ --- │
| │ u32 ┆ i64 │
| ╞═════╪═══════╡
| │ 2 ┆ 1 │
| │ 1 ┆ 2 │
| │ 1 ┆ 1 │
| │ 1 ┆ null │
| │ 1 ┆ 1 │
| │ 2 ┆ 3 │
| └─────┴───────┘
|
| rle_id(self) -> 'Expr'
| Get a distinct integer ID for each run of identical values.
|
| The ID starts at 0 and increases by one each time the value of the column
| changes.
|
| Returns
| -------
| Expr
| Expression of data type `UInt32`.
|
| See Also
| --------
| rle
|
| Notes
| -----
| This functionality is especially useful for defining a new group for every time
| a column's value changes, rather than for every distinct value of that column.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 1, 1, 1],
| ... "b": ["x", "x", None, "y", "y"],
| ... }
| ... )
| >>> df.with_columns(
| ... rle_id_a=pl.col("a").rle_id(),
| ... rle_id_ab=pl.struct("a", "b").rle_id(),
| ... )
| shape: (5, 4)
| ┌─────┬──────┬──────────┬───────────┐
| │ a ┆ b ┆ rle_id_a ┆ rle_id_ab │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ str ┆ u32 ┆ u32 │
| ╞═════╪══════╪══════════╪═══════════╡
| │ 1 ┆ x ┆ 0 ┆ 0 │
| │ 2 ┆ x ┆ 1 ┆ 1 │
| │ 1 ┆ null ┆ 2 ┆ 2 │
| │ 1 ┆ y ┆ 2 ┆ 3 │
| │ 1 ┆ y ┆ 2 ┆ 3 │
| └─────┴──────┴──────────┴───────────┘
|
| rolling(self, index_column: 'IntoExprColumn', *, period: 'str_ | timedelta', offset: 'str_ | timedelta | None' = None, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Create rolling groups based on a temporal or integer column.
|
| If you have a time series `<t_0, t_1, ..., t_n>`, then by default the
| windows created will be
|
| * (t_0 - period, t_0]
| * (t_1 - period, t_1]
| * ...
| * (t_n - period, t_n]
|
| whereas if you pass a non-default `offset`, then the windows will be
|
| * (t_0 + offset, t_0 + offset + period]
| * (t_1 + offset, t_1 + offset + period]
| * ...
| * (t_n + offset, t_n + offset + period]
|
| The `period` and `offset` arguments are created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| By "calendar day", we mean the corresponding time on the next day (which may
| not be 24 hours, due to daylight savings). Similarly for "calendar week",
| "calendar month", "calendar quarter", and "calendar year".
|
| Parameters
| ----------
| index_column
| Column used to group based on the time window.
| Often of type Date/Datetime.
| This column must be sorted in ascending order.
| In case of a rolling group by on indices, dtype needs to be one of
| {UInt32, UInt64, Int32, Int64}. Note that the first three get temporarily
| cast to Int64, so if performance matters use an Int64 column.
| period
| Length of the window - must be non-negative.
| offset
| Offset of the window. Default is `-period`.
| closed : {'right', 'left', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive).
|
| Examples
| --------
| >>> dates = [
| ... "2020-01-01 13:45:48",
| ... "2020-01-01 16:42:13",
| ... "2020-01-01 16:45:09",
| ... "2020-01-02 18:12:48",
| ... "2020-01-03 19:45:32",
| ... "2020-01-08 23:16:43",
| ... ]
| >>> df = pl.DataFrame({"dt": dates, "a": [3, 7, 5, 9, 2, 1]}).with_columns(
| ... pl.col("dt").str.strptime(pl.Datetime).set_sorted()
| ... )
| >>> df.with_columns(
| ... sum_a=pl.sum("a").rolling(index_column="dt", period="2d"),
| ... min_a=pl.min("a").rolling(index_column="dt", period="2d"),
| ... max_a=pl.max("a").rolling(index_column="dt", period="2d"),
| ... )
| shape: (6, 5)
| ┌─────────────────────┬─────┬───────┬───────┬───────┐
| │ dt ┆ a ┆ sum_a ┆ min_a ┆ max_a │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ datetime[μs] ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
| ╞═════════════════════╪═════╪═══════╪═══════╪═══════╡
| │ 2020-01-01 13:45:48 ┆ 3 ┆ 3 ┆ 3 ┆ 3 │
| │ 2020-01-01 16:42:13 ┆ 7 ┆ 10 ┆ 3 ┆ 7 │
| │ 2020-01-01 16:45:09 ┆ 5 ┆ 15 ┆ 3 ┆ 7 │
| │ 2020-01-02 18:12:48 ┆ 9 ┆ 24 ┆ 3 ┆ 9 │
| │ 2020-01-03 19:45:32 ┆ 2 ┆ 11 ┆ 2 ┆ 9 │
| │ 2020-01-08 23:16:43 ┆ 1 ┆ 1 ┆ 1 ┆ 1 │
| └─────────────────────┴─────┴───────┴───────┴───────┘
|
| rolling_kurtosis(self, window_size: 'int', *, fisher: 'bool' = True, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling kurtosis.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Expr.kurtosis
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 4, 2, 9]})
| >>> df.select(pl.col("a").rolling_kurtosis(3))
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ null │
| │ -1.5 │
| │ -1.5 │
| └──────┘
|
| rolling_map(self, function: 'Callable[[Series], Any]', window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a custom rolling window function.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| function
| Custom aggregation function.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Warnings
| --------
| Computing custom functions is extremely slow. Use specialized rolling
| functions such as :func:`Expr.rolling_sum` if at all possible.
|
| Examples
| --------
| >>> from numpy import nansum
| >>> df = pl.DataFrame({"a": [11.0, 2.0, 9.0, float("nan"), 8.0]})
| >>> df.select(pl.col("a").rolling_map(nansum, window_size=3))
| shape: (5, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ null │
| │ 22.0 │
| │ 11.0 │
| │ 17.0 │
| └──────┘
|
| rolling_max(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling max (moving max) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their max.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_max=pl.col("A").rolling_max(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_max │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 2.0 │
| │ 3.0 ┆ 3.0 │
| │ 4.0 ┆ 4.0 │
| │ 5.0 ┆ 5.0 │
| │ 6.0 ┆ 6.0 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_max=pl.col("A").rolling_max(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_max │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.25 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ 3.75 │
| │ 6.0 ┆ 4.5 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_max=pl.col("A").rolling_max(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_max │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 3.0 │
| │ 3.0 ┆ 4.0 │
| │ 4.0 ┆ 5.0 │
| │ 5.0 ┆ 6.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_max_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling max based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling max with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_max=pl.col("index").rolling_max_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_max │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 2 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 3 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 4 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 20 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 21 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 22 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 23 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 24 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling max with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_max=pl.col("index").rolling_max_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_max │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 2 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 3 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 4 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 20 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 21 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 22 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 23 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 24 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_mean(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling mean (moving mean) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their mean. Weights
| are normalized to sum to 1.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window, after being normalized to sum to
| 1.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_mean=pl.col("A").rolling_mean(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────┐
| │ A ┆ rolling_mean │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.5 │
| │ 4.0 ┆ 3.5 │
| │ 5.0 ┆ 4.5 │
| │ 6.0 ┆ 5.5 │
| └─────┴──────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_mean=pl.col("A").rolling_mean(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────┐
| │ A ┆ rolling_mean │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.75 │
| │ 3.0 ┆ 2.75 │
| │ 4.0 ┆ 3.75 │
| │ 5.0 ┆ 4.75 │
| │ 6.0 ┆ 5.75 │
| └─────┴──────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_mean=pl.col("A").rolling_mean(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────┐
| │ A ┆ rolling_mean │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 2.0 │
| │ 3.0 ┆ 3.0 │
| │ 4.0 ┆ 4.0 │
| │ 5.0 ┆ 5.0 │
| │ 6.0 ┆ null │
| └─────┴──────────────┘
|
| rolling_mean_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling mean based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling mean with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_mean=pl.col("index").rolling_mean_by(
| ... "date", window_size="2h"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬──────────────────┐
| │ index ┆ date ┆ rolling_row_mean │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪══════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.5 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.5 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.5 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.5 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.5 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.5 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.5 │
| └───────┴─────────────────────┴──────────────────┘
|
| Compute the rolling mean with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_mean=pl.col("index").rolling_mean_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬──────────────────┐
| │ index ┆ date ┆ rolling_row_mean │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪══════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.0 │
| └───────┴─────────────────────┴──────────────────┘
|
| rolling_median(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling median.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their median.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_median=pl.col("A").rolling_median(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬────────────────┐
| │ A ┆ rolling_median │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.5 │
| │ 4.0 ┆ 3.5 │
| │ 5.0 ┆ 4.5 │
| │ 6.0 ┆ 5.5 │
| └─────┴────────────────┘
|
| Specify weights for the values in each window:
|
| >>> df.with_columns(
| ... rolling_median=pl.col("A").rolling_median(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬────────────────┐
| │ A ┆ rolling_median │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.5 │
| │ 3.0 ┆ 2.5 │
| │ 4.0 ┆ 3.5 │
| │ 5.0 ┆ 4.5 │
| │ 6.0 ┆ 5.5 │
| └─────┴────────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_median=pl.col("A").rolling_median(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬────────────────┐
| │ A ┆ rolling_median │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 2.0 │
| │ 3.0 ┆ 3.0 │
| │ 4.0 ┆ 4.0 │
| │ 5.0 ┆ 5.0 │
| │ 6.0 ┆ null │
| └─────┴────────────────┘
|
| rolling_median_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Compute a rolling median based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling median with the temporal windows closed on the right:
|
| >>> df_temporal.with_columns(
| ... rolling_row_median=pl.col("index").rolling_median_by(
| ... "date", window_size="2h"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬────────────────────┐
| │ index ┆ date ┆ rolling_row_median │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.5 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.5 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.5 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.5 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.5 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.5 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.5 │
| └───────┴─────────────────────┴────────────────────┘
|
| rolling_min(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling min (moving min) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their min.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_min=pl.col("A").rolling_min(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_min │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 2.0 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ 4.0 │
| │ 6.0 ┆ 5.0 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_min=pl.col("A").rolling_min(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_min │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.25 │
| │ 3.0 ┆ 0.5 │
| │ 4.0 ┆ 0.75 │
| │ 5.0 ┆ 1.0 │
| │ 6.0 ┆ 1.25 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_min=pl.col("A").rolling_min(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_min │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 2.0 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ 4.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_min_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling min based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling min with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_min=pl.col("index").rolling_min_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_min │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_quantile(self, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', window_size: 'int' = 2, weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling quantile.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their quantile.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.25, window_size=4
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ null │
| │ 4.0 ┆ 2.0 │
| │ 5.0 ┆ 3.0 │
| │ 6.0 ┆ 4.0 │
| └─────┴──────────────────┘
|
| Specify weights for the values in each window:
|
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.25, window_size=4, weights=[0.2, 0.4, 0.4, 0.2]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ null │
| │ 4.0 ┆ 2.0 │
| │ 5.0 ┆ 3.0 │
| │ 6.0 ┆ 4.0 │
| └─────┴──────────────────┘
|
| Specify weights and interpolation method
|
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.25,
| ... window_size=4,
| ... weights=[0.2, 0.4, 0.4, 0.2],
| ... interpolation="linear",
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ null │
| │ 4.0 ┆ 1.625 │
| │ 5.0 ┆ 2.625 │
| │ 6.0 ┆ 3.625 │
| └─────┴──────────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_quantile=pl.col("A").rolling_quantile(
| ... quantile=0.2, window_size=5, center=True
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬──────────────────┐
| │ A ┆ rolling_quantile │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪══════════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ null │
| │ 3.0 ┆ 2.0 │
| │ 4.0 ┆ 3.0 │
| │ 5.0 ┆ null │
| │ 6.0 ┆ null │
| └─────┴──────────────────┘
|
| rolling_quantile_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Compute a rolling quantile based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling quantile with the temporal windows closed on the right:
|
| >>> df_temporal.with_columns(
| ... rolling_row_quantile=pl.col("index").rolling_quantile_by(
| ... "date", window_size="2h", quantile=0.3
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬──────────────────────┐
| │ index ┆ date ┆ rolling_row_quantile │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪══════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0.0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.0 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 2.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 3.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 19.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 20.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 21.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 22.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 23.0 │
| └───────┴─────────────────────┴──────────────────────┘
|
| rolling_rank(self, window_size: 'int', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling rank.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| A window of length `window_size` will traverse the array. The values
| that fill this window will be ranked according to the `method`
| parameter. The resulting values will be the rank of the value that is
| at the end of the sliding window.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Returns
| -------
| Expr
| An Expr of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 4, 4, 1, 9]})
| >>> df.select(pl.col("a").rolling_rank(3, method="average"))
| shape: (5, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ null │
| │ null │
| │ 2.5 │
| │ 1.0 │
| │ 3.0 │
| └──────┘
|
| rolling_rank_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Compute a rolling rank based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Returns
| -------
| Expr
| An Expr of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| rolling_skew(self, window_size: 'int', *, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Compute a rolling skew.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| bias
| If False, the calculations are corrected for statistical bias.
| bias: bool = True,
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Expr.skew
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 4, 2, 9]})
| >>> df.select(pl.col("a").rolling_skew(3))
| shape: (4, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ null │
| │ null │
| │ 0.381802 │
| │ 0.47033 │
| └──────────┘
|
| Note how the values match the following:
|
| >>> pl.Series([1, 4, 2]).skew(), pl.Series([4, 2, 9]).skew()
| (0.38180177416060584, 0.47033046033698594)
|
| rolling_std(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Expr'
| Compute a rolling standard deviation.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their std. Weights
| are normalized to sum to 1.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window after being normalized to sum to
| 1.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_std=pl.col("A").rolling_std(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_std │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.707107 │
| │ 3.0 ┆ 0.707107 │
| │ 4.0 ┆ 0.707107 │
| │ 5.0 ┆ 0.707107 │
| │ 6.0 ┆ 0.707107 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_std=pl.col("A").rolling_std(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_std │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.433013 │
| │ 3.0 ┆ 0.433013 │
| │ 4.0 ┆ 0.433013 │
| │ 5.0 ┆ 0.433013 │
| │ 6.0 ┆ 0.433013 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_std=pl.col("A").rolling_std(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_std │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 1.0 │
| │ 4.0 ┆ 1.0 │
| │ 5.0 ┆ 1.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_std_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Expr'
| Compute a rolling standard deviation based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling std with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_std=pl.col("index").rolling_std_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_std │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.707107 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 0.707107 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 0.707107 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 0.707107 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 0.707107 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 0.707107 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 0.707107 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 0.707107 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 0.707107 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling std with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_std=pl.col("index").rolling_std_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_std │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.707107 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 1.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 1.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 1.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 1.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 1.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 1.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 1.0 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_sum(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Expr'
| Apply a rolling sum (moving sum) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their sum.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_sum=pl.col("A").rolling_sum(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_sum │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 3.0 │
| │ 3.0 ┆ 5.0 │
| │ 4.0 ┆ 7.0 │
| │ 5.0 ┆ 9.0 │
| │ 6.0 ┆ 11.0 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_sum=pl.col("A").rolling_sum(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_sum │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.75 │
| │ 3.0 ┆ 2.75 │
| │ 4.0 ┆ 3.75 │
| │ 5.0 ┆ 4.75 │
| │ 6.0 ┆ 5.75 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_sum=pl.col("A").rolling_sum(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_sum │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 6.0 │
| │ 3.0 ┆ 9.0 │
| │ 4.0 ┆ 12.0 │
| │ 5.0 ┆ 15.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_sum_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 0, closed: 'ClosedInterval' = 'right') -> 'Expr'
| Apply a rolling sum based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling sum with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_sum=pl.col("index").rolling_sum_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_sum │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 3 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 7 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 39 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 41 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 43 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 45 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 47 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling sum with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_sum=pl.col("index").rolling_sum_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_sum │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ u32 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ 0 │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 1 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 3 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 6 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 9 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 57 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 60 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 63 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 66 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 69 │
| └───────┴─────────────────────┴─────────────────┘
|
| rolling_var(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Expr'
| Compute a rolling variance.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weights` vector. The resulting values will be aggregated to their var. Weights
| are normalized to sum to 1.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window after being normalized to sum to
| 1.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]})
| >>> df.with_columns(
| ... rolling_var=pl.col("A").rolling_var(window_size=2),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_var │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.5 │
| │ 3.0 ┆ 0.5 │
| │ 4.0 ┆ 0.5 │
| │ 5.0 ┆ 0.5 │
| │ 6.0 ┆ 0.5 │
| └─────┴─────────────┘
|
| Specify weights to multiply the values in the window with:
|
| >>> df.with_columns(
| ... rolling_var=pl.col("A").rolling_var(
| ... window_size=2, weights=[0.25, 0.75]
| ... ),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_var │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 0.1875 │
| │ 3.0 ┆ 0.1875 │
| │ 4.0 ┆ 0.1875 │
| │ 5.0 ┆ 0.1875 │
| │ 6.0 ┆ 0.1875 │
| └─────┴─────────────┘
|
| Center the values in the window
|
| >>> df.with_columns(
| ... rolling_var=pl.col("A").rolling_var(window_size=3, center=True),
| ... )
| shape: (6, 2)
| ┌─────┬─────────────┐
| │ A ┆ rolling_var │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════╪═════════════╡
| │ 1.0 ┆ null │
| │ 2.0 ┆ 1.0 │
| │ 3.0 ┆ 1.0 │
| │ 4.0 ┆ 1.0 │
| │ 5.0 ┆ 1.0 │
| │ 6.0 ┆ null │
| └─────┴─────────────┘
|
| rolling_var_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Expr'
| Compute a rolling variance based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a DataFrame with a datetime column and a row number column
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> df_temporal = pl.DataFrame(
| ... {"date": pl.datetime_range(start, stop, "1h", eager=True)}
| ... ).with_row_index()
| >>> df_temporal
| shape: (25, 2)
| ┌───────┬─────────────────────┐
| │ index ┆ date │
| │ --- ┆ --- │
| │ u32 ┆ datetime[μs] │
| ╞═══════╪═════════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 │
| │ 1 ┆ 2001-01-01 01:00:00 │
| │ 2 ┆ 2001-01-01 02:00:00 │
| │ 3 ┆ 2001-01-01 03:00:00 │
| │ 4 ┆ 2001-01-01 04:00:00 │
| │ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 │
| │ 21 ┆ 2001-01-01 21:00:00 │
| │ 22 ┆ 2001-01-01 22:00:00 │
| │ 23 ┆ 2001-01-01 23:00:00 │
| │ 24 ┆ 2001-01-02 00:00:00 │
| └───────┴─────────────────────┘
|
| Compute the rolling var with the temporal windows closed on the right (default)
|
| >>> df_temporal.with_columns(
| ... rolling_row_var=pl.col("index").rolling_var_by("date", window_size="2h")
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_var │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 0.5 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 0.5 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 0.5 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 0.5 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 0.5 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 0.5 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 0.5 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 0.5 │
| └───────┴─────────────────────┴─────────────────┘
|
| Compute the rolling var with the closure of windows on both sides
|
| >>> df_temporal.with_columns(
| ... rolling_row_var=pl.col("index").rolling_var_by(
| ... "date", window_size="2h", closed="both"
| ... )
| ... )
| shape: (25, 3)
| ┌───────┬─────────────────────┬─────────────────┐
| │ index ┆ date ┆ rolling_row_var │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ datetime[μs] ┆ f64 │
| ╞═══════╪═════════════════════╪═════════════════╡
| │ 0 ┆ 2001-01-01 00:00:00 ┆ null │
| │ 1 ┆ 2001-01-01 01:00:00 ┆ 0.5 │
| │ 2 ┆ 2001-01-01 02:00:00 ┆ 1.0 │
| │ 3 ┆ 2001-01-01 03:00:00 ┆ 1.0 │
| │ 4 ┆ 2001-01-01 04:00:00 ┆ 1.0 │
| │ … ┆ … ┆ … │
| │ 20 ┆ 2001-01-01 20:00:00 ┆ 1.0 │
| │ 21 ┆ 2001-01-01 21:00:00 ┆ 1.0 │
| │ 22 ┆ 2001-01-01 22:00:00 ┆ 1.0 │
| │ 23 ┆ 2001-01-01 23:00:00 ┆ 1.0 │
| │ 24 ┆ 2001-01-02 00:00:00 ┆ 1.0 │
| └───────┴─────────────────────┴─────────────────┘
|
| round(self, decimals: 'int' = 0, mode: 'RoundMode' = 'half_to_even') -> 'Expr'
| Round underlying floating point data by `decimals` digits.
|
| Parameters
| ----------
| decimals
| Number of decimals to round by.
| mode : {'half_to_even', 'half_away_from_zero', 'to_zero'}
| The rounding strategy used. A "rounded value" is a value with at most
| `decimals` decimal places (e.g. integers when ``decimals=0``, multiples
| of 0.1 when ``decimals=1``, 0.01 when ``decimals=2``, and so on).
|
| Strategies that start with ``half_`` round all values to the *nearest*
| rounded value, only using the strategy to break ties when a value falls
| exactly between two rounded values (e.g. 0.5 when ``decimals=0``, 0.05
| when ``decimals=1``). Other rounding strategies specify explicitly
| which rounded value is chosen and always apply (not just for tiebreaks).
|
| * *half_to_even* (default)
| Round to the nearest value; break ties by choosing the nearest
| **even** value. For example, 0.5 rounds to 0, 1.5 rounds to 2,
| 2.5 rounds to 2. Also known as "banker's rounding"; this is the
| default because it tends to minimise cumulative rounding bias.
| * *half_away_from_zero*
| Round to the nearest value; break ties by rounding **away from
| zero**. For example, 0.5 rounds to 1, -0.5 rounds to -1, 2.5
| rounds to 3. Also known as "commercial rounding".
| * *to_zero*
| Always round (truncate) **towards zero**, discarding the fractional
| part beyond `decimals`. For example, 0.9 rounds to 0, -0.9 rounds
| to 0, 1.29 rounds to 1.2 (with ``decimals=1``). Equivalent to the
| :meth:`truncate` method.
|
| See Also
| --------
| ceil : Round up to the nearest integer.
| floor : Round down to the nearest integer.
| round_sig_figs : Round to a given number of significant figures.
| truncate : Truncate to a given number of decimals.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.33, 0.52, 1.02, 1.17]})
| >>> df.select(pl.col("a").round(1))
| shape: (4, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.3 │
| │ 0.5 │
| │ 1.0 │
| │ 1.2 │
| └─────┘
|
| >>> df = pl.DataFrame(
| ... {
| ... "f64": [-3.5, -2.5, -1.5, -0.5, 0.5, 1.5, 2.5, 3.5],
| ... "d": ["-3.5", "-2.5", "-1.5", "-0.5", "0.5", "1.5", "2.5", "3.5"],
| ... },
| ... schema_overrides={"d": pl.Decimal(scale=1)},
| ... )
| >>> df.with_columns(
| ... pl.all().round(mode="half_away_from_zero").name.suffix("_away"),
| ... pl.all().round(mode="half_to_even").name.suffix("_to_even"),
| ... )
| shape: (8, 6)
| ┌──────┬───────────────┬──────────┬───────────────┬─────────────┬───────────────┐
| │ f64 ┆ d ┆ f64_away ┆ d_away ┆ f64_to_even ┆ d_to_even │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ decimal[38,1] ┆ f64 ┆ decimal[38,1] ┆ f64 ┆ decimal[38,1] │
| ╞══════╪═══════════════╪══════════╪═══════════════╪═════════════╪═══════════════╡
| │ -3.5 ┆ -3.5 ┆ -4.0 ┆ -4.0 ┆ -4.0 ┆ -4.0 │
| │ -2.5 ┆ -2.5 ┆ -3.0 ┆ -3.0 ┆ -2.0 ┆ -2.0 │
| │ -1.5 ┆ -1.5 ┆ -2.0 ┆ -2.0 ┆ -2.0 ┆ -2.0 │
| │ -0.5 ┆ -0.5 ┆ -1.0 ┆ -1.0 ┆ -0.0 ┆ 0.0 │
| │ 0.5 ┆ 0.5 ┆ 1.0 ┆ 1.0 ┆ 0.0 ┆ 0.0 │
| │ 1.5 ┆ 1.5 ┆ 2.0 ┆ 2.0 ┆ 2.0 ┆ 2.0 │
| │ 2.5 ┆ 2.5 ┆ 3.0 ┆ 3.0 ┆ 2.0 ┆ 2.0 │
| │ 3.5 ┆ 3.5 ┆ 4.0 ┆ 4.0 ┆ 4.0 ┆ 4.0 │
| └──────┴───────────────┴──────────┴───────────────┴─────────────┴───────────────┘
|
| round_sig_figs(self, digits: 'int') -> 'Expr'
| Round to a number of significant figures.
|
| Parameters
| ----------
| digits
| Number of significant figures to round to.
|
| See Also
| --------
| ceil : Round up to the nearest integer.
| floor : Round down to the nearest integer.
| round : Round to a given number of decimals.
| truncate : Truncate to a given number of decimals.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [0.01234, 3.333, 1234.0]})
| >>> df.with_columns(pl.col("a").round_sig_figs(2).alias("round_sig_figs"))
| shape: (3, 2)
| ┌─────────┬────────────────┐
| │ a ┆ round_sig_figs │
| │ --- ┆ --- │
| │ f64 ┆ f64 │
| ╞═════════╪════════════════╡
| │ 0.01234 ┆ 0.012 │
| │ 3.333 ┆ 3.3 │
| │ 1234.0 ┆ 1200.0 │
| └─────────┴────────────────┘
|
| sample(self, n: 'int | IntoExprColumn | None' = None, *, fraction: 'float | IntoExprColumn | None' = None, with_replacement: 'bool' = False, shuffle: 'bool | None' = None, seed: 'int | None' = None) -> 'Expr'
| Sample from this expression.
|
| Parameters
| ----------
| n
| Number of items to return. Cannot be used with `fraction`. Defaults to 1 if
| `fraction` is None.
| fraction
| Fraction of items to return. Cannot be used with `n`.
| with_replacement
| Allow values to be sampled more than once.
| shuffle
| Determines the order of the sampled elements.
| If True, sampled elements are explicitly shuffled.
| If False, the relative order of the sampled elements is preserved.
| (i.e. they appear in the same order as the original input).
| If None (default), no ordering guarantee; uses the most performant
| algorithm.
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated for each sample operation.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(
| ... pl.col("a").sample(
| ... fraction=1.0, with_replacement=True, shuffle=False, seed=1
| ... )
| ... )
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 3 │
| │ 3 │
| │ 1 │
| └─────┘
|
| search_sorted(self, element: 'IntoExpr | np.ndarray[Any, Any]', side: 'SearchSortedSide' = 'any', *, descending: 'bool' = False) -> 'Expr'
| Find indices where elements should be inserted to maintain order.
|
| .. math:: a[i-1] < v <= a[i]
|
| Parameters
| ----------
| element
| Expression or scalar value.
| side : {'any', 'left', 'right'}
| If 'any', the index of the first suitable location found is given.
| If 'left', the index of the leftmost suitable location found is given.
| If 'right', return the rightmost suitable location found is given.
| descending
| Boolean indicating whether the values are descending or not (they
| are required to be sorted either way).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "values": [1, 2, 3, 5],
| ... }
| ... )
| >>> df.select(
| ... [
| ... pl.col("values").search_sorted(0).alias("zero"),
| ... pl.col("values").search_sorted(3).alias("three"),
| ... pl.col("values").search_sorted(6).alias("six"),
| ... ]
| ... )
| shape: (1, 3)
| ┌──────┬───────┬─────┐
| │ zero ┆ three ┆ six │
| │ --- ┆ --- ┆ --- │
| │ u32 ┆ u32 ┆ u32 │
| ╞══════╪═══════╪═════╡
| │ 0 ┆ 2 ┆ 4 │
| └──────┴───────┴─────┘
|
| set_sorted(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Expr'
| Flags the expression as 'sorted'.
|
| Enables downstream code to user fast paths for sorted arrays. It is
| recommended to also set whether `nulls_last` is `True` or `False`, as
| this enables many internal optimizations.
|
| Parameters
| ----------
| descending
| Whether the `Series` order is descending.
| nulls_last
| Whether the nulls are at the end.
|
| Warnings
| --------
| This can lead to incorrect results if the data is NOT sorted!!
| Use with care!
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1, 2, 3]})
| >>> df.select(pl.col("values").set_sorted().max())
| shape: (1, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ i64 │
| ╞════════╡
| │ 3 │
| └────────┘
|
| shift(self, n: 'int | IntoExprColumn' = 1, *, fill_value: 'IntoExpr | None' = None) -> 'Expr'
| Shift values by the given number of indices.
|
| Parameters
| ----------
| n
| Number of indices to shift forward. If a negative value is passed, values
| are shifted in the opposite direction instead.
| fill_value
| Fill the resulting null values with this scalar value.
|
| Notes
| -----
| This method is similar to the `LAG` operation in SQL when the value for `n`
| is positive. With a negative value for `n`, it is similar to `LEAD`.
|
| See Also
| --------
| fill_null
|
| Examples
| --------
| By default, values are shifted forward by one index.
|
| >>> df = pl.DataFrame({"a": [1, 2, 3, 4]})
| >>> df.with_columns(shift=pl.col("a").shift())
| shape: (4, 2)
| ┌─────┬───────┐
| │ a ┆ shift │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═══════╡
| │ 1 ┆ null │
| │ 2 ┆ 1 │
| │ 3 ┆ 2 │
| │ 4 ┆ 3 │
| └─────┴───────┘
|
| Pass a negative value to shift in the opposite direction instead.
|
| >>> df.with_columns(shift=pl.col("a").shift(-2))
| shape: (4, 2)
| ┌─────┬───────┐
| │ a ┆ shift │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═══════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| │ 3 ┆ null │
| │ 4 ┆ null │
| └─────┴───────┘
|
| Specify `fill_value` to fill the resulting null values.
|
| >>> df.with_columns(shift=pl.col("a").shift(-2, fill_value=100))
| shape: (4, 2)
| ┌─────┬───────┐
| │ a ┆ shift │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═══════╡
| │ 1 ┆ 3 │
| │ 2 ┆ 4 │
| │ 3 ┆ 100 │
| │ 4 ┆ 100 │
| └─────┴───────┘
|
| shrink_dtype(self) -> 'Expr'
| Shrink numeric columns to the minimal required datatype.
|
| Shrink to the dtype needed to fit the extrema of this [`Series`].
| This can be used to reduce memory pressure.
|
| .. versionchanged:: 1.33.0
| Deprecated and turned into a no-op. The operation does not match the
| Polars data-model during lazy execution since the output datatype
| cannot be known without inspecting the data.
|
| Use `Series.shrink_dtype` instead.
|
| Examples
| --------
| >>> pl.DataFrame(
| ... {
| ... "a": [1, 2, 3],
| ... "b": [1, 2, 2 << 32],
| ... "c": [-1, 2, 1 << 30],
| ... "d": [-112, 2, 112],
| ... "e": [-112, 2, 129],
| ... "f": ["a", "b", "c"],
| ... "g": [0.1, 1.32, 0.12],
| ... "h": [True, None, False],
| ... }
| ... ).select(pl.all().shrink_dtype()) # doctest: +SKIP
| shape: (3, 8)
| ┌─────┬────────────┬────────────┬──────┬──────┬─────┬──────┬───────┐
| │ a ┆ b ┆ c ┆ d ┆ e ┆ f ┆ g ┆ h │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i8 ┆ i64 ┆ i32 ┆ i8 ┆ i16 ┆ str ┆ f32 ┆ bool │
| ╞═════╪════════════╪════════════╪══════╪══════╪═════╪══════╪═══════╡
| │ 1 ┆ 1 ┆ -1 ┆ -112 ┆ -112 ┆ a ┆ 0.1 ┆ true │
| │ 2 ┆ 2 ┆ 2 ┆ 2 ┆ 2 ┆ b ┆ 1.32 ┆ null │
| │ 3 ┆ 8589934592 ┆ 1073741824 ┆ 112 ┆ 129 ┆ c ┆ 0.12 ┆ false │
| └─────┴────────────┴────────────┴──────┴──────┴─────┴──────┴───────┘
|
| shuffle(self, seed: 'int | None' = None) -> 'Expr'
| Shuffle the contents of this expression.
|
| Note this is shuffled independently of any other column or Expression. If you
| want each row to stay the same use df.sample(shuffle=True)
|
| Parameters
| ----------
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated each time the shuffle is called.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3]})
| >>> df.select(pl.col("a").shuffle(seed=1))
| shape: (3, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 3 │
| │ 1 │
| └─────┘
|
| sign(self) -> 'Expr'
| Compute the element-wise sign function on numeric types.
|
| The returned value is computed as follows:
|
| * -1 if x < 0.
| * 1 if x > 0.
| * x otherwise (typically 0, but could be NaN if the input is).
|
| Null values are preserved as-is, and the dtype of the input is preserved.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-9.0, -0.0, 0.0, 4.0, float("nan"), None]})
| >>> df.select(pl.col.a.sign())
| shape: (6, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════╡
| │ -1.0 │
| │ -0.0 │
| │ 0.0 │
| │ 1.0 │
| │ NaN │
| │ null │
| └──────┘
|
| sin(self) -> 'Expr'
| Compute the element-wise value for the sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 2]})
| >>> df.select(pl.col("a").sin())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
| >>> df = pl.DataFrame({"a": [0.0, 90]})
| >>> df.select(pl.col("a").radians().sin())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
|
| sinh(self) -> 'Expr'
| Compute the element-wise value for the hyperbolic sine.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").sinh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.175201 │
| └──────────┘
|
| skew(self, *, bias: 'bool' = True) -> 'Expr'
| Compute the sample skewness of a data set.
|
| For normally distributed data, the skewness should be about zero. For
| unimodal continuous distributions, a skewness value greater than zero means
| that there is more weight in the right tail of the distribution. The
| function `skewtest` can be used to determine if the skewness value
| is close enough to zero, statistically speaking.
|
|
| See scipy.stats for more information.
|
| Parameters
| ----------
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Notes
| -----
| The sample skewness is computed as the Fisher-Pearson coefficient
| of skewness, i.e.
|
| .. math:: g_1=\frac{m_3}{m_2^{3/2}}
|
| where
|
| .. math:: m_i=\frac{1}{N}\sum_{n=1}^N(x[n]-\bar{x})^i
|
| is the biased sample :math:`i\texttt{th}` central moment, and
| :math:`\bar{x}` is
| the sample mean. If `bias` is False, the calculations are
| corrected for bias and the value computed is the adjusted
| Fisher-Pearson standardized moment coefficient, i.e.
|
| .. math::
| G_1 = \frac{k_3}{k_2^{3/2}} = \frac{\sqrt{N(N-1)}}{N-2}\frac{m_3}{m_2^{3/2}}
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").skew())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.343622 │
| └──────────┘
|
| slice(self, offset: 'int | Expr', length: 'int | Expr | None' = None) -> 'Expr'
| Get a slice of this expression.
|
| Parameters
| ----------
| offset
| Start index. Negative indexing is supported.
| length
| Length of the slice. If set to `None`, all rows starting at the offset
| will be selected.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [8, 9, 10, 11],
| ... "b": [None, 4, 4, 4],
| ... }
| ... )
| >>> df.select(pl.all().slice(1, 2))
| shape: (2, 2)
| ┌─────┬─────┐
| │ a ┆ b │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 9 ┆ 4 │
| │ 10 ┆ 4 │
| └─────┴─────┘
|
| sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Expr'
| Sort this column.
|
| When used in a projection/selection context, the whole column is sorted.
| When used in a group by context, the groups are sorted.
|
| Parameters
| ----------
| descending
| Sort in descending order.
| nulls_last
| Place null values last.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, None, 3, 2],
| ... }
| ... )
| >>> df.select(pl.col("a").sort())
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| │ 1 │
| │ 2 │
| │ 3 │
| └──────┘
| >>> df.select(pl.col("a").sort(descending=True))
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ null │
| │ 3 │
| │ 2 │
| │ 1 │
| └──────┘
| >>> df.select(pl.col("a").sort(nulls_last=True))
| shape: (4, 1)
| ┌──────┐
| │ a │
| │ --- │
| │ i64 │
| ╞══════╡
| │ 1 │
| │ 2 │
| │ 3 │
| │ null │
| └──────┘
|
| When sorting in a group by context, the groups are sorted.
|
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["one", "one", "one", "two", "two", "two"],
| ... "value": [1, 98, 2, 3, 99, 4],
| ... }
| ... )
| >>> df.group_by("group").agg(pl.col("value").sort()) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌───────┬────────────┐
| │ group ┆ value │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪════════════╡
| │ two ┆ [3, 4, 99] │
| │ one ┆ [1, 2, 98] │
| └───────┴────────────┘
|
| sort_by(self, by: 'IntoExpr | Iterable[IntoExpr]', *more_by: 'IntoExpr', descending: 'bool | Sequence[bool]' = False, nulls_last: 'bool | Sequence[bool]' = False, multithreaded: 'bool' = True, maintain_order: 'bool' = False) -> 'Expr'
| Sort this column by the ordering of other columns.
|
| When used in a projection/selection context, the whole column is sorted.
| When used in a group by context, the groups are sorted.
|
| Parameters
| ----------
| by
| Column(s) to sort by. Accepts expression input. Strings are parsed as column
| names.
| *more_by
| Additional columns to sort by, specified as positional arguments.
| descending
| Sort in descending order. When sorting by multiple columns, can be specified
| per column by passing a sequence of booleans.
| nulls_last
| Place null values last; can specify a single boolean applying to all columns
| or a sequence of booleans for per-column control.
| multithreaded
| Sort using multiple threads.
| maintain_order
| Whether the order should be maintained if elements are equal.
|
| Examples
| --------
| Pass a single column name to sort by that column.
|
| >>> df = pl.DataFrame(
| ... {
| ... "group": ["a", "a", "b", "b"],
| ... "value1": [1, 3, 4, 2],
| ... "value2": [8, 7, 6, 5],
| ... }
| ... )
| >>> df.select(pl.col("group").sort_by("value1"))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ a │
| │ b │
| │ a │
| │ b │
| └───────┘
|
| Sorting by expressions is also supported.
|
| >>> df.select(pl.col("group").sort_by(pl.col("value1") + pl.col("value2")))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ b │
| │ a │
| │ a │
| │ b │
| └───────┘
|
| Sort by multiple columns by passing a list of columns.
|
| >>> df.select(pl.col("group").sort_by(["value1", "value2"], descending=True))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ b │
| │ a │
| │ b │
| │ a │
| └───────┘
|
| Or use positional arguments to sort by multiple columns in the same way.
|
| >>> df.select(pl.col("group").sort_by("value1", "value2"))
| shape: (4, 1)
| ┌───────┐
| │ group │
| │ --- │
| │ str │
| ╞═══════╡
| │ a │
| │ b │
| │ a │
| │ b │
| └───────┘
|
| When sorting in a group by context, the groups are sorted.
|
| >>> df.group_by("group").agg(
| ... pl.col("value1").sort_by("value2")
| ... ) # doctest: +IGNORE_RESULT
| shape: (2, 2)
| ┌───────┬───────────┐
| │ group ┆ value1 │
| │ --- ┆ --- │
| │ str ┆ list[i64] │
| ╞═══════╪═══════════╡
| │ a ┆ [3, 1] │
| │ b ┆ [2, 4] │
| └───────┴───────────┘
|
| Take a single row from each group where a column attains its minimal value
| within that group.
|
| >>> df.group_by("group").agg(
| ... pl.all().sort_by("value2").first()
| ... ) # doctest: +IGNORE_RESULT
| shape: (2, 3)
| ┌───────┬────────┬────────┐
| │ group ┆ value1 ┆ value2 |
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 |
| ╞═══════╪════════╪════════╡
| │ a ┆ 3 ┆ 7 |
| │ b ┆ 2 ┆ 5 |
| └───────┴────────┴────────┘
|
| sqrt(self) -> 'Expr'
| Compute the square root of the elements.
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": [1.0, 2.0, 4.0]})
| >>> df.select(pl.col("values").sqrt())
| shape: (3, 1)
| ┌──────────┐
| │ values │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 1.0 │
| │ 1.414214 │
| │ 2.0 │
| └──────────┘
|
| std(self, ddof: 'int' = 1) -> 'Expr'
| Get standard deviation.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").std())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
|
| sub(self, other: 'Any') -> 'Expr'
| Method equivalent of subtraction operator `expr - other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Examples
| --------
| >>> df = pl.DataFrame({"x": [0, 1, 2, 3, 4]})
| >>> df.with_columns(
| ... pl.col("x").sub(2).alias("x-2"),
| ... pl.col("x").sub(pl.col("x").cum_sum()).alias("x-expr"),
| ... )
| shape: (5, 3)
| ┌─────┬─────┬────────┐
| │ x ┆ x-2 ┆ x-expr │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ i64 │
| ╞═════╪═════╪════════╡
| │ 0 ┆ -2 ┆ 0 │
| │ 1 ┆ -1 ┆ 0 │
| │ 2 ┆ 0 ┆ -1 │
| │ 3 ┆ 1 ┆ -3 │
| │ 4 ┆ 2 ┆ -6 │
| └─────┴─────┴────────┘
|
| sum(self) -> 'Expr'
| Get sum value.
|
| Notes
| -----
| * Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
| * If there are no non-null values, then the output is `0`.
| If you would prefer empty sums to return `None`, you can
| use `pl.when(expr.count()>0).then(expr.sum())` instead
| of `expr.sum()`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").sum())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 0 │
| └─────┘
|
| tail(self, n: 'int | Expr' = 10) -> 'Expr'
| Get the last `n` rows.
|
| Parameters
| ----------
| n
| Number of rows to return.
|
| Examples
| --------
| >>> df = pl.DataFrame({"foo": [1, 2, 3, 4, 5, 6, 7]})
| >>> df.select(pl.col("foo").tail(3))
| shape: (3, 1)
| ┌─────┐
| │ foo │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 5 │
| │ 6 │
| │ 7 │
| └─────┘
|
| tan(self) -> 'Expr'
| Compute the element-wise value for the tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Notes
| -----
| The argument must be in radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Expr.radians>`.
|
| Examples
| --------
| >>> from math import pi
| >>> df = pl.DataFrame({"a": [0.0, pi / 4]})
| >>> df.select(pl.col("a").tan())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
| >>> df = pl.DataFrame({"a": [0.0, 45]})
| >>> df.select(pl.col("a").radians().tan())
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 0.0 │
| │ 1.0 │
| └─────┘
|
| tanh(self) -> 'Expr'
| Compute the element-wise value for the hyperbolic tangent.
|
| Returns
| -------
| Expr
| Expression of data type :class:`Float64`.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1.0]})
| >>> df.select(pl.col("a").tanh())
| shape: (1, 1)
| ┌──────────┐
| │ a │
| │ --- │
| │ f64 │
| ╞══════════╡
| │ 0.761594 │
| └──────────┘
|
| to_physical(self) -> 'Expr'
| Cast to physical representation of the logical dtype.
|
| - :func:`polars.datatypes.Date` -> :func:`polars.datatypes.Int32`
| - :func:`polars.datatypes.Datetime` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Time` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Duration` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Categorical` -> :func:`polars.datatypes.UInt32`
| - `List(inner)` -> `List(physical of inner)`
| - `Array(inner)` -> `Struct(physical of inner)`
| - `Struct(fields)` -> `Array(physical of fields)`
|
| Other data types will be left unchanged.
|
| Warnings
| --------
| The physical representations are an implementation detail
| and not guaranteed to be stable.
|
| Examples
| --------
| Replicating the pandas
| `pd.factorize
| <https://pandas.pydata.org/docs/reference/api/pandas.factorize.html>`_
| function.
|
| >>> pl.DataFrame({"vals": ["a", "x", None, "a"]}).with_columns(
| ... pl.col("vals").cast(pl.Categorical),
| ... pl.col("vals")
| ... .cast(pl.Categorical)
| ... .to_physical()
| ... .alias("vals_physical"),
| ... )
| shape: (4, 2)
| ┌──────┬───────────────┐
| │ vals ┆ vals_physical │
| │ --- ┆ --- │
| │ cat ┆ u32 │
| ╞══════╪═══════════════╡
| │ a ┆ 0 │
| │ x ┆ 1 │
| │ null ┆ null │
| │ a ┆ 0 │
| └──────┴───────────────┘
|
| top_k(self, k: 'int | IntoExprColumn' = 5) -> 'Expr'
| Return the `k` largest elements.
|
| Non-null elements are always preferred over null elements. The output
| is not guaranteed to be in any particular order, call :func:`sort`
| after this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k_by
| bottom_k
| bottom_k_by
|
| Examples
| --------
| Get the 5 largest values in series.
|
| >>> df = pl.DataFrame({"value": [1, 98, 2, 3, 99, 4]})
| >>> df.select(
| ... pl.col("value").top_k().alias("top_k"),
| ... pl.col("value").bottom_k().alias("bottom_k"),
| ... )
| shape: (5, 2)
| ┌───────┬──────────┐
| │ top_k ┆ bottom_k │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═══════╪══════════╡
| │ 4 ┆ 1 │
| │ 98 ┆ 98 │
| │ 2 ┆ 2 │
| │ 3 ┆ 3 │
| │ 99 ┆ 4 │
| └───────┴──────────┘
|
| top_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int | IntoExprColumn' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Expr'
| Return the elements corresponding to the `k` largest elements of the `by` column(s).
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| .. versionchanged:: 1.0.0
| The `descending` parameter was renamed to `reverse`.
|
| Parameters
| ----------
| by
| Column(s) used to determine the largest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` smallest elements of the `by` column(s) (instead of the `k`
| largest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| bottom_k
| bottom_k_by
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "a": [1, 2, 3, 4, 5, 6],
| ... "b": [6, 5, 4, 3, 2, 1],
| ... "c": ["Apple", "Orange", "Apple", "Apple", "Banana", "Banana"],
| ... }
| ... )
| >>> df
| shape: (6, 3)
| ┌─────┬─────┬────────┐
| │ a ┆ b ┆ c │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str │
| ╞═════╪═════╪════════╡
| │ 1 ┆ 6 ┆ Apple │
| │ 2 ┆ 5 ┆ Orange │
| │ 3 ┆ 4 ┆ Apple │
| │ 4 ┆ 3 ┆ Apple │
| │ 5 ┆ 2 ┆ Banana │
| │ 6 ┆ 1 ┆ Banana │
| └─────┴─────┴────────┘
|
| Get the top 2 rows by column `a` or `b`.
|
| >>> df.select(
| ... pl.all().top_k_by("a", 2).name.suffix("_top_by_a"),
| ... pl.all().top_k_by("b", 2).name.suffix("_top_by_b"),
| ... )
| shape: (2, 6)
| ┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
| │ a_top_by_a ┆ b_top_by_a ┆ c_top_by_a ┆ a_top_by_b ┆ b_top_by_b ┆ c_top_by_b │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞════════════╪════════════╪════════════╪════════════╪════════════╪════════════╡
| │ 6 ┆ 1 ┆ Banana ┆ 1 ┆ 6 ┆ Apple │
| │ 5 ┆ 2 ┆ Banana ┆ 2 ┆ 5 ┆ Orange │
| └────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
|
| Get the top 2 rows by multiple columns with given order.
|
| >>> df.select(
| ... pl.all()
| ... .top_k_by(["c", "a"], 2, reverse=[False, True])
| ... .name.suffix("_by_ca"),
| ... pl.all()
| ... .top_k_by(["c", "b"], 2, reverse=[False, True])
| ... .name.suffix("_by_cb"),
| ... )
| shape: (2, 6)
| ┌─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
| │ a_by_ca ┆ b_by_ca ┆ c_by_ca ┆ a_by_cb ┆ b_by_cb ┆ c_by_cb │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
| ╞═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
| │ 2 ┆ 5 ┆ Orange ┆ 2 ┆ 5 ┆ Orange │
| │ 5 ┆ 2 ┆ Banana ┆ 6 ┆ 1 ┆ Banana │
| └─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
|
| Get the top 2 rows by column `a` in each group.
|
| >>> (
| ... df.group_by("c", maintain_order=True)
| ... .agg(pl.all().top_k_by("a", 2))
| ... .explode(pl.all().exclude("c"))
| ... )
| shape: (5, 3)
| ┌────────┬─────┬─────┐
| │ c ┆ a ┆ b │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞════════╪═════╪═════╡
| │ Apple ┆ 4 ┆ 3 │
| │ Apple ┆ 3 ┆ 4 │
| │ Orange ┆ 2 ┆ 5 │
| │ Banana ┆ 6 ┆ 1 │
| │ Banana ┆ 5 ┆ 2 │
| └────────┴─────┴─────┘
|
| truediv(self, other: 'Any') -> 'Expr'
| Method equivalent of float division operator `expr / other`.
|
| Parameters
| ----------
| other
| Numeric literal or expression value.
|
| Notes
| -----
| Zero-division behaviour follows IEEE-754:
|
| 0/0: Invalid operation - mathematically undefined, returns NaN.
| n/0: On finite operands gives an exact infinite result, eg: ±infinity.
|
| See Also
| --------
| floordiv
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... data={"x": [-2, -1, 0, 1, 2], "y": [0.5, 0.0, 0.0, -4.0, -0.5]}
| ... )
| >>> df.with_columns(
| ... pl.col("x").truediv(2).alias("x/2"),
| ... pl.col("x").truediv(pl.col("y")).alias("x/y"),
| ... )
| shape: (5, 4)
| ┌─────┬──────┬──────┬───────┐
| │ x ┆ y ┆ x/2 ┆ x/y │
| │ --- ┆ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ f64 ┆ f64 │
| ╞═════╪══════╪══════╪═══════╡
| │ -2 ┆ 0.5 ┆ -1.0 ┆ -4.0 │
| │ -1 ┆ 0.0 ┆ -0.5 ┆ -inf │
| │ 0 ┆ 0.0 ┆ 0.0 ┆ NaN │
| │ 1 ┆ -4.0 ┆ 0.5 ┆ -0.25 │
| │ 2 ┆ -0.5 ┆ 1.0 ┆ -4.0 │
| └─────┴──────┴──────┴───────┘
|
| truncate(self, decimals: 'int' = 0) -> 'Expr'
| Truncate numeric data toward zero to `decimals` number of decimal places.
|
| Parameters
| ----------
| decimals
| Number of decimal places to truncate to.
|
| Notes
| -----
| * Truncation discards the fractional part beyond the given number of decimals.
| For example, when rounding to 0 decimals 0.25, -0.25, 0.99, and -0.99 will
| all round to 0. When rounding to 1 decimal 1.9999 rounds to 1.9 and -1.9999
| rounds to -1.9. There is no tiebreak behaviour at midpoint values as there
| is with :meth:`round` so 0.5 and -0.5 will also round to 0 when decimals=1.
|
| * This method performs numeric truncation. For truncating temporal
| data (dates/datetimes), use :func:`Expr.dt.truncate` instead.
|
| See Also
| --------
| ceil : Round up to the nearest integer.
| floor : Round down to the nearest integer.
| round : Round to a given number of decimals.
| round_sig_figs : Round to a given number of significant figures.
|
| Examples
| --------
| >>> df = pl.DataFrame({"n": [-9.9999, 0.12345, 1.0251, 8.8765]})
| >>> df.with_columns(
| ... t0=pl.col("n").truncate(0),
| ... t1=pl.col("n").truncate(1),
| ... t2=pl.col("n").truncate(2),
| ... t3=pl.col("n").truncate(3),
| ... t4=pl.col("n").truncate(4),
| ... )
| shape: (4, 6)
| ┌─────────┬──────┬──────┬───────┬────────┬─────────┐
| │ n ┆ t0 ┆ t1 ┆ t2 ┆ t3 ┆ t4 │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
| ╞═════════╪══════╪══════╪═══════╪════════╪═════════╡
| │ -9.9999 ┆ -9.0 ┆ -9.9 ┆ -9.99 ┆ -9.999 ┆ -9.9999 │
| │ 0.12345 ┆ 0.0 ┆ 0.1 ┆ 0.12 ┆ 0.123 ┆ 0.1234 │
| │ 1.0251 ┆ 1.0 ┆ 1.0 ┆ 1.02 ┆ 1.025 ┆ 1.025 │
| │ 8.8765 ┆ 8.0 ┆ 8.8 ┆ 8.87 ┆ 8.876 ┆ 8.8765 │
| └─────────┴──────┴──────┴───────┴────────┴─────────┘
|
| unique(self, *, maintain_order: 'bool' = False) -> 'Expr'
| Get unique values of this expression.
|
| `null` is considered to be a unique value for the purposes of this operation.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 1, 2]})
| >>> df.select(pl.col("a").unique()) # doctest: +IGNORE_RESULT
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 2 │
| │ 1 │
| └─────┘
| >>> df.select(pl.col("a").unique(maintain_order=True))
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 1 │
| │ 2 │
| └─────┘
|
| unique_counts(self) -> 'Expr'
| Return a count of the unique values in the order of appearance.
|
| This method differs from `value_counts` in that it does not return the
| values, only the counts and might be faster
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "id": ["a", "b", "b", "c", "c", "c"],
| ... }
| ... )
| >>> df.select(pl.col("id").unique_counts())
| shape: (3, 1)
| ┌─────┐
| │ id │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| Note that `group_by` can be used to generate counts.
|
| >>> df.group_by("id", maintain_order=True).len().select("len")
| shape: (3, 1)
| ┌─────┐
| │ len │
| │ --- │
| │ u32 │
| ╞═════╡
| │ 1 │
| │ 2 │
| │ 3 │
| └─────┘
|
| To add counts as a new column `pl.len()` can be used as a window function.
|
| >>> df.with_columns(pl.len().over("id"))
| shape: (6, 2)
| ┌─────┬─────┐
| │ id ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═════╪═════╡
| │ a ┆ 1 │
| │ b ┆ 2 │
| │ b ┆ 2 │
| │ c ┆ 3 │
| │ c ┆ 3 │
| │ c ┆ 3 │
| └─────┴─────┘
|
| upper_bound(self) -> 'Expr'
| Calculate the upper bound.
|
| Returns a unit Series with the highest value possible for the dtype of this
| expression.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [1, 2, 3, 2, 1]})
| >>> df.select(pl.col("a").upper_bound())
| shape: (1, 1)
| ┌─────────────────────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════════════════════╡
| │ 9223372036854775807 │
| └─────────────────────┘
|
| value_counts(self, *, sort: 'bool' = False, parallel: 'bool' = False, name: 'str_ | None' = None, normalize: 'bool' = False) -> 'Expr'
| Count the occurrence of unique values.
|
| Parameters
| ----------
| sort
| Sort the output by count, in descending order.
| If set to `False` (default), the order is non-deterministic.
| parallel
| Execute the computation in parallel.
|
| .. note::
| This option should likely *not* be enabled in a `group_by` context,
| as the computation will already be parallelized per group.
| name
| Give the resulting count column a specific name; if `normalize` is
| True this defaults to "proportion", otherwise defaults to "count".
| normalize
| If True, the count is returned as the relative frequency of unique
| values normalized to 1.0.
|
| Returns
| -------
| Expr
| Expression of type :class:`Struct`, mapping unique values to their
| count (or proportion).
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"color": ["red", "blue", "red", "green", "blue", "blue"]}
| ... )
| >>> df_count = df.select(pl.col("color").value_counts())
| >>> df_count # doctest: +IGNORE_RESULT
| shape: (3, 1)
| ┌─────────────┐
| │ color │
| │ --- │
| │ struct[2] │
| ╞═════════════╡
| │ {"green",1} │
| │ {"blue",3} │
| │ {"red",2} │
| └─────────────┘
|
| >>> df_count.unnest("color") # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌───────┬───────┐
| │ color ┆ count │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═══════╡
| │ green ┆ 1 │
| │ blue ┆ 3 │
| │ red ┆ 2 │
| └───────┴───────┘
|
| Sort the output by (descending) count, customize the field name,
| and normalize the count to its relative proportion (of 1.0).
|
| >>> df_count = df.select(
| ... pl.col("color").value_counts(
| ... name="fraction",
| ... normalize=True,
| ... sort=True,
| ... )
| ... )
| >>> df_count
| shape: (3, 1)
| ┌────────────────────┐
| │ color │
| │ --- │
| │ struct[2] │
| ╞════════════════════╡
| │ {"blue",0.5} │
| │ {"red",0.333333} │
| │ {"green",0.166667} │
| └────────────────────┘
|
| >>> df_count.unnest("color")
| shape: (3, 2)
| ┌───────┬──────────┐
| │ color ┆ fraction │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪══════════╡
| │ blue ┆ 0.5 │
| │ red ┆ 0.333333 │
| │ green ┆ 0.166667 │
| └───────┴──────────┘
|
| Note that `group_by` can be used to generate counts.
|
| >>> df.group_by("color").len() # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌───────┬─────┐
| │ color ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═════╡
| │ red ┆ 2 │
| │ green ┆ 1 │
| │ blue ┆ 3 │
| └───────┴─────┘
|
| To add counts as a new column `pl.len()` can be used as a window function.
|
| >>> df.with_columns(pl.len().over("color"))
| shape: (6, 2)
| ┌───────┬─────┐
| │ color ┆ len │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═════╡
| │ red ┆ 2 │
| │ blue ┆ 3 │
| │ red ┆ 2 │
| │ green ┆ 1 │
| │ blue ┆ 3 │
| │ blue ┆ 3 │
| └───────┴─────┘
|
| >>> df.with_columns((pl.len().over("color") / pl.len()).alias("fraction"))
| shape: (6, 2)
| ┌───────┬──────────┐
| │ color ┆ fraction │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪══════════╡
| │ red ┆ 0.333333 │
| │ blue ┆ 0.5 │
| │ red ┆ 0.333333 │
| │ green ┆ 0.166667 │
| │ blue ┆ 0.5 │
| │ blue ┆ 0.5 │
| └───────┴──────────┘
|
| var(self, ddof: 'int' = 1) -> 'Expr'
| Get variance.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> df = pl.DataFrame({"a": [-1, 0, 1]})
| >>> df.select(pl.col("a").var())
| shape: (1, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ f64 │
| ╞═════╡
| │ 1.0 │
| └─────┘
|
| where(self, predicate: 'Expr') -> 'Expr'
| Filter a single column.
|
| .. deprecated:: 0.20.4
| Use the :func:`filter` method instead.
|
| Alias for :func:`filter`.
|
| Parameters
| ----------
| predicate
| Boolean expression.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "group_col": ["g1", "g1", "g2"],
| ... "b": [1, 2, 3],
| ... }
| ... )
| >>> df.group_by("group_col").agg( # doctest: +SKIP
| ... [
| ... pl.col("b").where(pl.col("b") < 2).sum().alias("lt"),
| ... pl.col("b").where(pl.col("b") >= 2).sum().alias("gte"),
| ... ]
| ... ).sort("group_col")
| shape: (2, 3)
| ┌───────────┬─────┬─────┐
| │ group_col ┆ lt ┆ gte │
| │ --- ┆ --- ┆ --- │
| │ str ┆ i64 ┆ i64 │
| ╞═══════════╪═════╪═════╡
| │ g1 ┆ 1 ┆ 2 │
| │ g2 ┆ 0 ┆ 3 │
| └───────────┴─────┴─────┘
|
| xor(self, other: 'Any') -> 'Expr'
| Method equivalent of bitwise exclusive-or operator `expr ^ other`.
|
| Parameters
| ----------
| other
| Integer or boolean value; accepts expression input.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {"x": [True, False, True, False], "y": [True, True, False, False]}
| ... )
| >>> df.with_columns(pl.col("x").xor(pl.col("y")).alias("x ^ y"))
| shape: (4, 3)
| ┌───────┬───────┬───────┐
| │ x ┆ y ┆ x ^ y │
| │ --- ┆ --- ┆ --- │
| │ bool ┆ bool ┆ bool │
| ╞═══════╪═══════╪═══════╡
| │ true ┆ true ┆ false │
| │ false ┆ true ┆ true │
| │ true ┆ false ┆ true │
| │ false ┆ false ┆ false │
| └───────┴───────┴───────┘
|
| >>> def binary_string(n: int) -> str:
| ... return bin(n)[2:].zfill(8)
| >>>
| >>> df = pl.DataFrame(
| ... data={"x": [10, 8, 250, 66], "y": [1, 2, 3, 4]},
| ... schema={"x": pl.UInt8, "y": pl.UInt8},
| ... )
| >>> df.with_columns(
| ... pl.col("x")
| ... .map_elements(binary_string, return_dtype=pl.String)
| ... .alias("bin_x"),
| ... pl.col("y")
| ... .map_elements(binary_string, return_dtype=pl.String)
| ... .alias("bin_y"),
| ... pl.col("x").xor(pl.col("y")).alias("xor_xy"),
| ... pl.col("x")
| ... .xor(pl.col("y"))
| ... .map_elements(binary_string, return_dtype=pl.String)
| ... .alias("bin_xor_xy"),
| ... )
| shape: (4, 6)
| ┌─────┬─────┬──────────┬──────────┬────────┬────────────┐
| │ x ┆ y ┆ bin_x ┆ bin_y ┆ xor_xy ┆ bin_xor_xy │
| │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ str ┆ str ┆ u8 ┆ str │
| ╞═════╪═════╪══════════╪══════════╪════════╪════════════╡
| │ 10 ┆ 1 ┆ 00001010 ┆ 00000001 ┆ 11 ┆ 00001011 │
| │ 8 ┆ 2 ┆ 00001000 ┆ 00000010 ┆ 10 ┆ 00001010 │
| │ 250 ┆ 3 ┆ 11111010 ┆ 00000011 ┆ 249 ┆ 11111001 │
| │ 66 ┆ 4 ┆ 01000010 ┆ 00000100 ┆ 70 ┆ 01000110 │
| └─────┴─────┴──────────┴──────────┴────────┴────────────┘
|
| ----------------------------------------------------------------------
| Class methods inherited from polars.expr.expr.Expr:
|
| deserialize(source: 'str_ | Path | IOBase | bytes', *, format: 'SerializationFormat' = 'binary') -> 'Expr'
| Read a serialized expression from a file.
|
| Parameters
| ----------
| source
| Path to a file or a file-like object (by file-like object, we refer to
| objects that have a `read()` method, such as a file handler (e.g.
| via builtin `open` function) or `BytesIO`).
| format
| The format with which the Expr was serialized. Options:
|
| - `"binary"`: Deserialize from binary format (bytes). This is the default.
| - `"json"`: Deserialize from JSON format (string).
|
| Warnings
| --------
| This function uses :mod:`pickle` if the logical plan contains Python UDFs,
| and as such inherits the security implications. Deserializing can execute
| arbitrary code, so it should only be attempted on trusted data.
|
| See Also
| --------
| Expr.meta.serialize
|
| Notes
| -----
| Serialization is not stable across Polars versions: a LazyFrame serialized
| in one Polars version may not be deserializable in another Polars version.
|
| Examples
| --------
| >>> import io
| >>> expr = pl.col("foo").sum().over("bar")
| >>> bytes = expr.meta.serialize()
| >>> pl.Expr.deserialize(io.BytesIO(bytes))
| <Expr ['col("foo").sum().over([col("ba…'] at ...>
|
| from_json(value: 'str_') -> 'Expr'
| Read an expression from a JSON encoded string to construct an Expression.
|
| .. deprecated:: 0.20.11
| This method has been renamed to :meth:`deserialize`.
| Note that the new method operates on file-like inputs rather than strings.
| Enclose your input in `io.StringIO` to keep the same behavior.
|
| Parameters
| ----------
| value
| JSON encoded string value
|
| ----------------------------------------------------------------------
| Readonly properties inherited from polars.expr.expr.Expr:
|
| arr
| Create an object namespace of all array related methods.
|
| See the individual method pages for full details.
|
| bin
| Create an object namespace of all binary related methods.
|
| See the individual method pages for full details
|
| cat
| Create an object namespace of all categorical related methods.
|
| See the individual method pages for full details
|
| Examples
| --------
| >>> df = pl.DataFrame({"values": ["a", "b"]}).select(
| ... pl.col("values").cast(pl.Categorical)
| ... )
| >>> df.select(pl.col("values").cat.get_categories())
| shape: (2, 1)
| ┌────────┐
| │ values │
| │ --- │
| │ str │
| ╞════════╡
| │ a │
| │ b │
| └────────┘
|
| dt
| Create an object namespace of all datetime related methods.
|
| ext
| Create an object namespace of all extension type related expressions.
|
| See the individual method pages for full details.
|
| list
| Create an object namespace of all list related methods.
|
| See the individual method pages for full details.
|
| meta
| Create an object namespace of all meta related expression methods.
|
| This can be used to modify and traverse existing expressions.
|
| name
| Create an object namespace of all expressions that modify expression names.
|
| See the individual method pages for full details.
|
| str
| Create an object namespace of all string related methods.
|
| See the individual method pages for full details.
|
| Examples
| --------
| >>> df = pl.DataFrame({"letters": ["a", "b"]})
| >>> df.select(pl.col("letters").str.to_uppercase())
| shape: (2, 1)
| ┌─────────┐
| │ letters │
| │ --- │
| │ str │
| ╞═════════╡
| │ A │
| │ B │
| └─────────┘
|
| struct
| Create an object namespace of all struct related methods.
|
| See the individual method pages for full details.
|
| Examples
| --------
| >>> df = (
| ... pl.DataFrame(
| ... {
| ... "int": [1, 2],
| ... "str": ["a", "b"],
| ... "bool": [True, None],
| ... "list": [[1, 2], [3]],
| ... }
| ... )
| ... .to_struct("my_struct")
| ... .to_frame()
| ... )
| >>> df.select(pl.col("my_struct").struct.field("str"))
| shape: (2, 1)
| ┌─────┐
| │ str │
| │ --- │
| │ str │
| ╞═════╡
| │ a │
| │ b │
| └─────┘
|
| ----------------------------------------------------------------------
| Data descriptors inherited from polars.expr.expr.Expr:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
all() -> 'Selector'
Select all columns.
See Also
--------
first : Select the first column in the current scope.
last : Select the last column in the current scope.
Examples
--------
>>> from datetime import date
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "dt": [date(1999, 12, 31), date(2024, 1, 1)],
... "value": [1_234_500, 5_000_555],
... },
... schema_overrides={"value": pl.Int32},
... )
Select all columns, casting them to string:
>>> df.select(cs.all().cast(pl.String))
shape: (2, 2)
┌────────────┬─────────┐
│ dt ┆ value │
│ --- ┆ --- │
│ str ┆ str │
╞════════════╪═════════╡
│ 1999-12-31 ┆ 1234500 │
│ 2024-01-01 ┆ 5000555 │
└────────────┴─────────┘
Select all columns *except* for those matching the given dtypes:
>>> df.select(cs.all() - cs.numeric())
shape: (2, 1)
┌────────────┐
│ dt │
│ --- │
│ date │
╞════════════╡
│ 1999-12-31 │
│ 2024-01-01 │
└────────────┘
alpha(ascii_only: 'bool' = False, *, ignore_spaces: 'bool' = False) -> 'Selector'
Select all columns with alphabetic names (eg: only letters).
Parameters
----------
ascii_only
Indicate whether to consider only ASCII alphabetic characters, or the full
Unicode range of valid letters (accented, idiographic, etc).
ignore_spaces
Indicate whether to ignore the presence of spaces in column names; if so,
only the other (non-space) characters are considered.
Notes
-----
Matching column names cannot contain *any* non-alphabetic characters. Note
that the definition of "alphabetic" consists of all valid Unicode alphabetic
characters (`\p{Alphabetic}`) by default; this can be changed by setting
`ascii_only=True`.
Examples
--------
>>> import polars as pl
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "no1": [100, 200, 300],
... "café": ["espresso", "latte", "mocha"],
... "t or f": [True, False, None],
... "hmm": ["aaa", "bbb", "ccc"],
... "都市": ["東京", "大阪", "京都"],
... }
... )
Select columns with alphabetic names; note that accented
characters and kanji are recognised as alphabetic here:
>>> df.select(cs.alpha())
shape: (3, 3)
┌──────────┬─────┬──────┐
│ café ┆ hmm ┆ 都市 │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str │
╞══════════╪═════╪══════╡
│ espresso ┆ aaa ┆ 東京 │
│ latte ┆ bbb ┆ 大阪 │
│ mocha ┆ ccc ┆ 京都 │
└──────────┴─────┴──────┘
Constrain the definition of "alphabetic" to ASCII characters only:
>>> df.select(cs.alpha(ascii_only=True))
shape: (3, 1)
┌─────┐
│ hmm │
│ --- │
│ str │
╞═════╡
│ aaa │
│ bbb │
│ ccc │
└─────┘
>>> df.select(cs.alpha(ascii_only=True, ignore_spaces=True))
shape: (3, 2)
┌────────┬─────┐
│ t or f ┆ hmm │
│ --- ┆ --- │
│ bool ┆ str │
╞════════╪═════╡
│ true ┆ aaa │
│ false ┆ bbb │
│ null ┆ ccc │
└────────┴─────┘
Select all columns *except* for those with alphabetic names:
>>> df.select(~cs.alpha())
shape: (3, 2)
┌─────┬────────┐
│ no1 ┆ t or f │
│ --- ┆ --- │
│ i64 ┆ bool │
╞═════╪════════╡
│ 100 ┆ true │
│ 200 ┆ false │
│ 300 ┆ null │
└─────┴────────┘
>>> df.select(~cs.alpha(ignore_spaces=True))
shape: (3, 1)
┌─────┐
│ no1 │
│ --- │
│ i64 │
╞═════╡
│ 100 │
│ 200 │
│ 300 │
└─────┘
alphanumeric(ascii_only: 'bool' = False, *, ignore_spaces: 'bool' = False) -> 'Selector'
Select all columns with alphanumeric names (eg: only letters and the digits 0-9).
Parameters
----------
ascii_only
Indicate whether to consider only ASCII alphabetic characters, or the full
Unicode range of valid letters (accented, idiographic, etc).
ignore_spaces
Indicate whether to ignore the presence of spaces in column names; if so,
only the other (non-space) characters are considered.
Notes
-----
Matching column names cannot contain *any* non-alphabetic or integer characters.
Note that the definition of "alphabetic" consists of all valid Unicode alphabetic
characters (`\p{Alphabetic}`) and digit characters (`\d`) by default; this
can be changed by setting `ascii_only=True`.
Examples
--------
>>> import polars as pl
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "1st_col": [100, 200, 300],
... "flagged": [True, False, True],
... "00prefix": ["01:aa", "02:bb", "03:cc"],
... "last col": ["x", "y", "z"],
... }
... )
Select columns with alphanumeric names:
>>> df.select(cs.alphanumeric())
shape: (3, 2)
┌─────────┬──────────┐
│ flagged ┆ 00prefix │
│ --- ┆ --- │
│ bool ┆ str │
╞═════════╪══════════╡
│ true ┆ 01:aa │
│ false ┆ 02:bb │
│ true ┆ 03:cc │
└─────────┴──────────┘
>>> df.select(cs.alphanumeric(ignore_spaces=True))
shape: (3, 3)
┌─────────┬──────────┬──────────┐
│ flagged ┆ 00prefix ┆ last col │
│ --- ┆ --- ┆ --- │
│ bool ┆ str ┆ str │
╞═════════╪══════════╪══════════╡
│ true ┆ 01:aa ┆ x │
│ false ┆ 02:bb ┆ y │
│ true ┆ 03:cc ┆ z │
└─────────┴──────────┴──────────┘
Select all columns *except* for those with alphanumeric names:
>>> df.select(~cs.alphanumeric())
shape: (3, 2)
┌─────────┬──────────┐
│ 1st_col ┆ last col │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════════╪══════════╡
│ 100 ┆ x │
│ 200 ┆ y │
│ 300 ┆ z │
└─────────┴──────────┘
>>> df.select(~cs.alphanumeric(ignore_spaces=True))
shape: (3, 1)
┌─────────┐
│ 1st_col │
│ --- │
│ i64 │
╞═════════╡
│ 100 │
│ 200 │
│ 300 │
└─────────┘
array(inner: 'Selector | None' = None, *, width: 'int | None' = None) -> 'Selector'
Select all array columns.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
See Also
--------
by_dtype : Select all columns matching the given dtype(s).
list : Select all list columns.
nested : Select all nested columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": [["xx", "yy"], ["x", "y"]],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... },
... schema_overrides={"foo": pl.Array(pl.String, 2)},
... )
Select all array columns:
>>> df.select(cs.array())
shape: (2, 1)
┌───────────────┐
│ foo │
│ --- │
│ array[str, 2] │
╞═══════════════╡
│ ["xx", "yy"] │
│ ["x", "y"] │
└───────────────┘
Select all columns *except* for those that are array:
>>> df.select(~cs.array())
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 123 ┆ 2.0 │
│ 456 ┆ 5.5 │
└─────┴─────┘
Select all array columns with a certain matching inner type:
>>> df.select(cs.array(cs.string()))
shape: (2, 1)
┌───────────────┐
│ foo │
│ --- │
│ array[str, 2] │
╞═══════════════╡
│ ["xx", "yy"] │
│ ["x", "y"] │
└───────────────┘
>>> df.select(cs.array(cs.integer()))
shape: (0, 0)
┌┐
╞╡
└┘
>>> df.select(cs.array(width=2))
shape: (2, 1)
┌───────────────┐
│ foo │
│ --- │
│ array[str, 2] │
╞═══════════════╡
│ ["xx", "yy"] │
│ ["x", "y"] │
└───────────────┘
>>> df.select(cs.array(width=3))
shape: (0, 0)
┌┐
╞╡
└┘
binary() -> 'Selector'
Select all binary columns.
See Also
--------
by_dtype : Select all columns matching the given dtype(s).
string : Select all string columns (optionally including categoricals).
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame({"a": [b"hello"], "b": ["world"], "c": [b"!"], "d": [":)"]})
>>> df
shape: (1, 4)
┌──────────┬───────┬────────┬─────┐
│ a ┆ b ┆ c ┆ d │
│ --- ┆ --- ┆ --- ┆ --- │
│ binary ┆ str ┆ binary ┆ str │
╞══════════╪═══════╪════════╪═════╡
│ b"hello" ┆ world ┆ b"!" ┆ :) │
└──────────┴───────┴────────┴─────┘
Select binary columns and export as a dict:
>>> df.select(cs.binary()).to_dict(as_series=False)
{'a': [b'hello'], 'c': [b'!']}
Select all columns *except* for those that are binary:
>>> df.select(~cs.binary()).to_dict(as_series=False)
{'b': ['world'], 'd': [':)']}
boolean() -> 'Selector'
Select all boolean columns.
See Also
--------
by_dtype : Select all columns matching the given dtype(s).
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame({"n": range(1, 5)}).with_columns(n_even=pl.col("n") % 2 == 0)
>>> df
shape: (4, 2)
┌─────┬────────┐
│ n ┆ n_even │
│ --- ┆ --- │
│ i64 ┆ bool │
╞═════╪════════╡
│ 1 ┆ false │
│ 2 ┆ true │
│ 3 ┆ false │
│ 4 ┆ true │
└─────┴────────┘
Select and invert boolean columns:
>>> df.with_columns(is_odd=cs.boolean().not_())
shape: (4, 3)
┌─────┬────────┬────────┐
│ n ┆ n_even ┆ is_odd │
│ --- ┆ --- ┆ --- │
│ i64 ┆ bool ┆ bool │
╞═════╪════════╪════════╡
│ 1 ┆ false ┆ true │
│ 2 ┆ true ┆ false │
│ 3 ┆ false ┆ true │
│ 4 ┆ true ┆ false │
└─────┴────────┴────────┘
Select all columns *except* for those that are boolean:
>>> df.select(~cs.boolean())
shape: (4, 1)
┌─────┐
│ n │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 2 │
│ 3 │
│ 4 │
└─────┘
by_dtype(*dtypes: 'PolarsDataType | PythonDataType | Iterable[PolarsDataType] | Iterable[PythonDataType]') -> 'Selector'
Select all columns matching the given dtypes.
See Also
--------
by_name : Select all columns matching the given names.
by_index : Select all columns matching the given indices.
Examples
--------
>>> from datetime import date
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "dt": [date(1999, 12, 31), date(2024, 1, 1), date(2010, 7, 5)],
... "value": [1_234_500, 5_000_555, -4_500_000],
... "other": ["foo", "bar", "foo"],
... }
... )
Select all columns with date or string dtypes:
>>> df.select(cs.by_dtype(pl.Date, pl.String))
shape: (3, 2)
┌────────────┬───────┐
│ dt ┆ other │
│ --- ┆ --- │
│ date ┆ str │
╞════════════╪═══════╡
│ 1999-12-31 ┆ foo │
│ 2024-01-01 ┆ bar │
│ 2010-07-05 ┆ foo │
└────────────┴───────┘
Select all columns that are not of date or string dtype:
>>> df.select(~cs.by_dtype(pl.Date, pl.String))
shape: (3, 1)
┌──────────┐
│ value │
│ --- │
│ i64 │
╞══════════╡
│ 1234500 │
│ 5000555 │
│ -4500000 │
└──────────┘
Group by string columns and sum the numeric columns:
>>> df.group_by(cs.string()).agg(cs.numeric().sum()).sort(by="other")
shape: (2, 2)
┌───────┬──────────┐
│ other ┆ value │
│ --- ┆ --- │
│ str ┆ i64 │
╞═══════╪══════════╡
│ bar ┆ 5000555 │
│ foo ┆ -3265500 │
└───────┴──────────┘
by_index(*indices: 'int | range | Sequence[int | range]', require_all: 'bool' = True) -> 'Selector'
Select all columns matching the given indices (or range objects).
Parameters
----------
*indices
One or more column indices (or range objects).
Negative indexing is supported.
require_all
By default, all specified indices must be valid; if any index is out of bounds,
an error is raised. If set to `False`, out-of-bounds indices are ignored
Notes
-----
Matching columns are returned in the order in which their indexes
appear in the selector, not the underlying schema order.
See Also
--------
by_dtype : Select all columns matching the given dtypes.
by_name : Select all columns matching the given names.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "key": ["abc"],
... **{f"c{i:02}": [0.5 * i] for i in range(100)},
... },
... )
>>> print(df)
shape: (1, 101)
┌─────┬─────┬─────┬─────┬───┬──────┬──────┬──────┬──────┐
│ key ┆ c00 ┆ c01 ┆ c02 ┆ … ┆ c96 ┆ c97 ┆ c98 ┆ c99 │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ f64 ┆ f64 ┆ ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞═════╪═════╪═════╪═════╪═══╪══════╪══════╪══════╪══════╡
│ abc ┆ 0.0 ┆ 0.5 ┆ 1.0 ┆ … ┆ 48.0 ┆ 48.5 ┆ 49.0 ┆ 49.5 │
└─────┴─────┴─────┴─────┴───┴──────┴──────┴──────┴──────┘
Select columns by index ("key" column and the two first/last columns):
>>> df.select(cs.by_index(0, 1, 2, -2, -1))
shape: (1, 5)
┌─────┬─────┬─────┬──────┬──────┐
│ key ┆ c00 ┆ c01 ┆ c98 ┆ c99 │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞═════╪═════╪═════╪══════╪══════╡
│ abc ┆ 0.0 ┆ 0.5 ┆ 49.0 ┆ 49.5 │
└─────┴─────┴─────┴──────┴──────┘
Select the "key" column and use a `range` object to select various columns.
Note that you can freely mix and match integer indices and `range` objects:
>>> df.select(cs.by_index(0, range(1, 101, 20)))
shape: (1, 6)
┌─────┬─────┬──────┬──────┬──────┬──────┐
│ key ┆ c00 ┆ c20 ┆ c40 ┆ c60 ┆ c80 │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞═════╪═════╪══════╪══════╪══════╪══════╡
│ abc ┆ 0.0 ┆ 10.0 ┆ 20.0 ┆ 30.0 ┆ 40.0 │
└─────┴─────┴──────┴──────┴──────┴──────┘
>>> df.select(cs.by_index(0, range(101, 0, -25), require_all=False))
shape: (1, 5)
┌─────┬──────┬──────┬──────┬─────┐
│ key ┆ c75 ┆ c50 ┆ c25 ┆ c00 │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞═════╪══════╪══════╪══════╪═════╡
│ abc ┆ 37.5 ┆ 25.0 ┆ 12.5 ┆ 0.0 │
└─────┴──────┴──────┴──────┴─────┘
Select all columns *except* for the even-indexed ones:
>>> df.select(~cs.by_index(range(1, 100, 2)))
shape: (1, 51)
┌─────┬─────┬─────┬─────┬───┬──────┬──────┬──────┬──────┐
│ key ┆ c01 ┆ c03 ┆ c05 ┆ … ┆ c93 ┆ c95 ┆ c97 ┆ c99 │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ f64 ┆ f64 ┆ ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞═════╪═════╪═════╪═════╪═══╪══════╪══════╪══════╪══════╡
│ abc ┆ 0.5 ┆ 1.5 ┆ 2.5 ┆ … ┆ 46.5 ┆ 47.5 ┆ 48.5 ┆ 49.5 │
└─────┴─────┴─────┴─────┴───┴──────┴──────┴──────┴──────┘
by_name(*names: 'str | Collection[str]', require_all: 'bool' = True) -> 'Selector'
Select all columns matching the given names.
.. versionadded:: 0.20.27
The `require_all` parameter was added.
Parameters
----------
*names
One or more names of columns to select.
require_all
Whether to match *all* names (the default) or *any* of the names.
Notes
-----
Matching columns are returned in the order in which they are declared in
the selector, not the underlying schema order.
See Also
--------
by_dtype : Select all columns matching the given dtypes.
by_index : Select all columns matching the given indices.
matches: Select columns matching the given regex pattern.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "zap": [False, True],
... }
... )
Select columns by name:
>>> df.select(cs.by_name("foo", "bar"))
shape: (2, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═════╡
│ x ┆ 123 │
│ y ┆ 456 │
└─────┴─────┘
Match *any* of the given columns by name:
>>> df.select(cs.by_name("baz", "moose", "foo", "bear", require_all=False))
shape: (2, 2)
┌─────┬─────┐
│ baz ┆ foo │
│ --- ┆ --- │
│ f64 ┆ str │
╞═════╪═════╡
│ 2.0 ┆ x │
│ 5.5 ┆ y │
└─────┴─────┘
Match all columns *except* for those given:
>>> df.select(~cs.by_name("foo", "bar"))
shape: (2, 2)
┌─────┬───────┐
│ baz ┆ zap │
│ --- ┆ --- │
│ f64 ┆ bool │
╞═════╪═══════╡
│ 2.0 ┆ false │
│ 5.5 ┆ true │
└─────┴───────┘
categorical() -> 'Selector'
Select all categorical columns.
See Also
--------
by_dtype : Select all columns matching the given dtype(s).
string : Select all string columns (optionally including categoricals).
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["xx", "yy"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... },
... schema_overrides={"foo": pl.Categorical},
... )
Select all categorical columns:
>>> df.select(cs.categorical())
shape: (2, 1)
┌─────┐
│ foo │
│ --- │
│ cat │
╞═════╡
│ xx │
│ yy │
└─────┘
Select all columns *except* for those that are categorical:
>>> df.select(~cs.categorical())
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 123 ┆ 2.0 │
│ 456 ┆ 5.5 │
└─────┴─────┘
contains(*substring: 'str') -> 'Selector'
Select columns whose names contain the given literal substring(s).
Parameters
----------
substring
Substring(s) that matching column names should contain.
See Also
--------
matches : Select all columns that match the given regex pattern.
ends_with : Select columns that end with the given substring(s).
starts_with : Select columns that start with the given substring(s).
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "zap": [False, True],
... }
... )
Select columns that contain the substring 'ba':
>>> df.select(cs.contains("ba"))
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 123 ┆ 2.0 │
│ 456 ┆ 5.5 │
└─────┴─────┘
Select columns that contain the substring 'ba' or the letter 'z':
>>> df.select(cs.contains("ba", "z"))
shape: (2, 3)
┌─────┬─────┬───────┐
│ bar ┆ baz ┆ zap │
│ --- ┆ --- ┆ --- │
│ i64 ┆ f64 ┆ bool │
╞═════╪═════╪═══════╡
│ 123 ┆ 2.0 ┆ false │
│ 456 ┆ 5.5 ┆ true │
└─────┴─────┴───────┘
Select all columns *except* for those that contain the substring 'ba':
>>> df.select(~cs.contains("ba"))
shape: (2, 2)
┌─────┬───────┐
│ foo ┆ zap │
│ --- ┆ --- │
│ str ┆ bool │
╞═════╪═══════╡
│ x ┆ false │
│ y ┆ true │
└─────┴───────┘
date() -> 'Selector'
Select all date columns.
See Also
--------
datetime : Select all datetime columns, optionally filtering by time unit/zone.
duration : Select all duration columns, optionally filtering by time unit.
temporal : Select all temporal columns.
time : Select all time columns.
Examples
--------
>>> from datetime import date, datetime, time
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "dtm": [datetime(2001, 5, 7, 10, 25), datetime(2031, 12, 31, 0, 30)],
... "dt": [date(1999, 12, 31), date(2024, 8, 9)],
... "tm": [time(0, 0, 0), time(23, 59, 59)],
... },
... )
Select all date columns:
>>> df.select(cs.date())
shape: (2, 1)
┌────────────┐
│ dt │
│ --- │
│ date │
╞════════════╡
│ 1999-12-31 │
│ 2024-08-09 │
└────────────┘
Select all columns *except* for those that are dates:
>>> df.select(~cs.date())
shape: (2, 2)
┌─────────────────────┬──────────┐
│ dtm ┆ tm │
│ --- ┆ --- │
│ datetime[μs] ┆ time │
╞═════════════════════╪══════════╡
│ 2001-05-07 10:25:00 ┆ 00:00:00 │
│ 2031-12-31 00:30:00 ┆ 23:59:59 │
└─────────────────────┴──────────┘
datetime(time_unit: 'TimeUnit | Collection[TimeUnit] | None' = None, time_zone: 'str | pydatetime.timezone | Collection[str | pydatetime.timezone | None] | None' = ('*', None)) -> 'Selector'
Select all datetime columns, optionally filtering by time unit/zone.
Parameters
----------
time_unit
One (or more) of the allowed timeunit precision strings, "ms", "us", and "ns".
Omit to select columns with any valid timeunit.
time_zone
* One or more timezone strings, as defined in zoneinfo (to see valid options
run `import zoneinfo; zoneinfo.available_timezones()` for a full list).
* Set `None` to select Datetime columns that do not have a timezone.
* Set "*" to select Datetime columns that have *any* timezone.
See Also
--------
date : Select all date columns.
duration : Select all duration columns, optionally filtering by time unit.
temporal : Select all temporal columns.
time : Select all time columns.
Examples
--------
>>> from datetime import datetime, date, timezone
>>> import polars.selectors as cs
>>> from zoneinfo import ZoneInfo
>>> tokyo_tz = ZoneInfo("Asia/Tokyo")
>>> utc_tz = timezone.utc
>>> df = pl.DataFrame(
... {
... "tstamp_tokyo": [
... datetime(1999, 7, 21, 5, 20, 16, 987654, tzinfo=tokyo_tz),
... datetime(2000, 5, 16, 6, 21, 21, 123465, tzinfo=tokyo_tz),
... ],
... "tstamp_utc": [
... datetime(2023, 4, 10, 12, 14, 16, 999000, tzinfo=utc_tz),
... datetime(2025, 8, 25, 14, 18, 22, 666000, tzinfo=utc_tz),
... ],
... "tstamp": [
... datetime(2000, 11, 20, 18, 12, 16, 600000),
... datetime(2020, 10, 30, 10, 20, 25, 123000),
... ],
... "dt": [date(1999, 12, 31), date(2010, 7, 5)],
... },
... schema_overrides={
... "tstamp_tokyo": pl.Datetime("ns", "Asia/Tokyo"),
... "tstamp_utc": pl.Datetime("us", "UTC"),
... },
... )
Select all datetime columns:
>>> df.select(cs.datetime())
shape: (2, 3)
┌────────────────────────────────┬─────────────────────────────┬─────────────────────────┐
│ tstamp_tokyo ┆ tstamp_utc ┆ tstamp │
│ --- ┆ --- ┆ --- │
│ datetime[ns, Asia/Tokyo] ┆ datetime[μs, UTC] ┆ datetime[μs] │
╞════════════════════════════════╪═════════════════════════════╪═════════════════════════╡
│ 1999-07-21 05:20:16.987654 JST ┆ 2023-04-10 12:14:16.999 UTC ┆ 2000-11-20 18:12:16.600 │
│ 2000-05-16 06:21:21.123465 JST ┆ 2025-08-25 14:18:22.666 UTC ┆ 2020-10-30 10:20:25.123 │
└────────────────────────────────┴─────────────────────────────┴─────────────────────────┘
Select all datetime columns that have 'us' precision:
>>> df.select(cs.datetime("us"))
shape: (2, 2)
┌─────────────────────────────┬─────────────────────────┐
│ tstamp_utc ┆ tstamp │
│ --- ┆ --- │
│ datetime[μs, UTC] ┆ datetime[μs] │
╞═════════════════════════════╪═════════════════════════╡
│ 2023-04-10 12:14:16.999 UTC ┆ 2000-11-20 18:12:16.600 │
│ 2025-08-25 14:18:22.666 UTC ┆ 2020-10-30 10:20:25.123 │
└─────────────────────────────┴─────────────────────────┘
Select all datetime columns that have *any* timezone:
>>> df.select(cs.datetime(time_zone="*"))
shape: (2, 2)
┌────────────────────────────────┬─────────────────────────────┐
│ tstamp_tokyo ┆ tstamp_utc │
│ --- ┆ --- │
│ datetime[ns, Asia/Tokyo] ┆ datetime[μs, UTC] │
╞════════════════════════════════╪═════════════════════════════╡
│ 1999-07-21 05:20:16.987654 JST ┆ 2023-04-10 12:14:16.999 UTC │
│ 2000-05-16 06:21:21.123465 JST ┆ 2025-08-25 14:18:22.666 UTC │
└────────────────────────────────┴─────────────────────────────┘
Select all datetime columns that have a *specific* timezone:
>>> df.select(cs.datetime(time_zone="UTC"))
shape: (2, 1)
┌─────────────────────────────┐
│ tstamp_utc │
│ --- │
│ datetime[μs, UTC] │
╞═════════════════════════════╡
│ 2023-04-10 12:14:16.999 UTC │
│ 2025-08-25 14:18:22.666 UTC │
└─────────────────────────────┘
Select all datetime columns that have NO timezone:
>>> df.select(cs.datetime(time_zone=None))
shape: (2, 1)
┌─────────────────────────┐
│ tstamp │
│ --- │
│ datetime[μs] │
╞═════════════════════════╡
│ 2000-11-20 18:12:16.600 │
│ 2020-10-30 10:20:25.123 │
└─────────────────────────┘
Select all columns *except* for datetime columns:
>>> df.select(~cs.datetime())
shape: (2, 1)
┌────────────┐
│ dt │
│ --- │
│ date │
╞════════════╡
│ 1999-12-31 │
│ 2010-07-05 │
└────────────┘
decimal() -> 'Selector'
Select all decimal columns.
See Also
--------
float : Select all float columns.
integer : Select all integer columns.
numeric : Select all numeric columns.
Examples
--------
>>> from decimal import Decimal as D
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [D(123), D(456)],
... "baz": [D("2.0005"), D("-50.5555")],
... },
... schema_overrides={"baz": pl.Decimal(scale=5, precision=10)},
... )
Select all decimal columns:
>>> df.select(cs.decimal())
shape: (2, 2)
┌───────────────┬───────────────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ decimal[38,0] ┆ decimal[10,5] │
╞═══════════════╪═══════════════╡
│ 123 ┆ 2.00050 │
│ 456 ┆ -50.55550 │
└───────────────┴───────────────┘
Select all columns *except* the decimal ones:
>>> df.select(~cs.decimal())
shape: (2, 1)
┌─────┐
│ foo │
│ --- │
│ str │
╞═════╡
│ x │
│ y │
└─────┘
digit(ascii_only: 'bool' = False) -> 'Selector'
Select all columns having names consisting only of digits.
Notes
-----
Matching column names cannot contain *any* non-digit characters. Note that the
definition of "digit" consists of all valid Unicode digit characters (`\d`)
by default; this can be changed by setting `ascii_only=True`.
Examples
--------
>>> import polars as pl
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "key": ["aaa", "bbb", "aaa", "bbb", "bbb"],
... "year": [2001, 2001, 2025, 2025, 2001],
... "value": [-25, 100, 75, -15, -5],
... }
... ).pivot(
... values="value",
... index="key",
... on="year",
... aggregate_function="sum",
... )
>>> print(df)
shape: (2, 3)
┌─────┬──────┬──────┐
│ key ┆ 2001 ┆ 2025 │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞═════╪══════╪══════╡
│ aaa ┆ -25 ┆ 75 │
│ bbb ┆ 95 ┆ -15 │
└─────┴──────┴──────┘
Select columns with digit names:
>>> df.select(cs.digit())
shape: (2, 2)
┌──────┬──────┐
│ 2001 ┆ 2025 │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞══════╪══════╡
│ -25 ┆ 75 │
│ 95 ┆ -15 │
└──────┴──────┘
Select all columns *except* for those with digit names:
>>> df.select(~cs.digit())
shape: (2, 1)
┌─────┐
│ key │
│ --- │
│ str │
╞═════╡
│ aaa │
│ bbb │
└─────┘
Demonstrate use of `ascii_only` flag (by default all valid unicode digits
are considered, but this can be constrained to ascii 0-9):
>>> df = pl.DataFrame({"१९९९": [1999], "२०७७": [2077], "3000": [3000]})
>>> df.select(cs.digit())
shape: (1, 3)
┌──────┬──────┬──────┐
│ १९९९ ┆ २०७७ ┆ 3000 │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞══════╪══════╪══════╡
│ 1999 ┆ 2077 ┆ 3000 │
└──────┴──────┴──────┘
>>> df.select(cs.digit(ascii_only=True))
shape: (1, 1)
┌──────┐
│ 3000 │
│ --- │
│ i64 │
╞══════╡
│ 3000 │
└──────┘
duration(time_unit: 'TimeUnit | Collection[TimeUnit] | None' = None) -> 'Selector'
Select all duration columns, optionally filtering by time unit.
Parameters
----------
time_unit
One (or more) of the allowed timeunit precision strings, "ms", "us", and "ns".
Omit to select columns with any valid timeunit.
See Also
--------
date : Select all date columns.
datetime : Select all datetime columns, optionally filtering by time unit/zone.
temporal : Select all temporal columns.
time : Select all time columns.
Examples
--------
>>> from datetime import date, timedelta
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "dt": [date(2022, 1, 31), date(2025, 7, 5)],
... "td1": [
... timedelta(days=1, milliseconds=123456),
... timedelta(days=1, hours=23, microseconds=987000),
... ],
... "td2": [
... timedelta(days=7, microseconds=456789),
... timedelta(days=14, minutes=999, seconds=59),
... ],
... "td3": [
... timedelta(weeks=4, days=-10, microseconds=999999),
... timedelta(weeks=3, milliseconds=123456, microseconds=1),
... ],
... },
... schema_overrides={
... "td1": pl.Duration("ms"),
... "td2": pl.Duration("us"),
... "td3": pl.Duration("ns"),
... },
... )
Select all duration columns:
>>> df.select(cs.duration())
shape: (2, 3)
┌────────────────┬─────────────────┬────────────────────┐
│ td1 ┆ td2 ┆ td3 │
│ --- ┆ --- ┆ --- │
│ duration[ms] ┆ duration[μs] ┆ duration[ns] │
╞════════════════╪═════════════════╪════════════════════╡
│ 1d 2m 3s 456ms ┆ 7d 456789µs ┆ 18d 999999µs │
│ 1d 23h 987ms ┆ 14d 16h 39m 59s ┆ 21d 2m 3s 456001µs │
└────────────────┴─────────────────┴────────────────────┘
Select all duration columns that have 'ms' precision:
>>> df.select(cs.duration("ms"))
shape: (2, 1)
┌────────────────┐
│ td1 │
│ --- │
│ duration[ms] │
╞════════════════╡
│ 1d 2m 3s 456ms │
│ 1d 23h 987ms │
└────────────────┘
Select all duration columns that have 'ms' OR 'ns' precision:
>>> df.select(cs.duration(["ms", "ns"]))
shape: (2, 2)
┌────────────────┬────────────────────┐
│ td1 ┆ td3 │
│ --- ┆ --- │
│ duration[ms] ┆ duration[ns] │
╞════════════════╪════════════════════╡
│ 1d 2m 3s 456ms ┆ 18d 999999µs │
│ 1d 23h 987ms ┆ 21d 2m 3s 456001µs │
└────────────────┴────────────────────┘
Select all columns *except* for duration columns:
>>> df.select(~cs.duration())
shape: (2, 1)
┌────────────┐
│ dt │
│ --- │
│ date │
╞════════════╡
│ 2022-01-31 │
│ 2025-07-05 │
└────────────┘
ends_with(*suffix: 'str') -> 'Selector'
Select columns that end with the given substring(s).
See Also
--------
contains : Select columns that contain the given literal substring(s).
matches : Select all columns that match the given regex pattern.
starts_with : Select columns that start with the given substring(s).
Parameters
----------
suffix
Substring(s) that matching column names should end with.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "zap": [False, True],
... }
... )
Select columns that end with the substring 'z':
>>> df.select(cs.ends_with("z"))
shape: (2, 1)
┌─────┐
│ baz │
│ --- │
│ f64 │
╞═════╡
│ 2.0 │
│ 5.5 │
└─────┘
Select columns that end with *either* the letter 'z' or 'r':
>>> df.select(cs.ends_with("z", "r"))
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 123 ┆ 2.0 │
│ 456 ┆ 5.5 │
└─────┴─────┘
Select all columns *except* for those that end with the substring 'z':
>>> df.select(~cs.ends_with("z"))
shape: (2, 3)
┌─────┬─────┬───────┐
│ foo ┆ bar ┆ zap │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ bool │
╞═════╪═════╪═══════╡
│ x ┆ 123 ┆ false │
│ y ┆ 456 ┆ true │
└─────┴─────┴───────┘
enum() -> 'Selector'
Select all enum columns.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
See Also
--------
by_dtype : Select all columns matching the given dtype(s).
categorical : Select all categorical columns.
string : Select all string columns (optionally including categoricals).
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["xx", "yy"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... },
... schema_overrides={"foo": pl.Enum(["xx", "yy"])},
... )
Select all enum columns:
>>> df.select(cs.enum())
shape: (2, 1)
┌──────┐
│ foo │
│ --- │
│ enum │
╞══════╡
│ xx │
│ yy │
└──────┘
Select all columns *except* for those that are enum:
>>> df.select(~cs.enum())
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 123 ┆ 2.0 │
│ 456 ┆ 5.5 │
└─────┴─────┘
exclude(columns: 'str | PolarsDataType | Selector | Expr | Collection[str | PolarsDataType | Selector | Expr]', *more_columns: 'str | PolarsDataType | Selector | Expr') -> 'Selector'
Select all columns except those matching the given columns, datatypes, or selectors.
Parameters
----------
columns
One or more columns (col or name), datatypes, columns, or selectors representing
the columns to exclude.
*more_columns
Additional columns, datatypes, or selectors to exclude, specified as positional
arguments.
Notes
-----
If excluding a single selector it is simpler to write as `~selector` instead.
Examples
--------
Exclude by column name(s):
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "aa": [1, 2, 3],
... "ba": ["a", "b", None],
... "cc": [None, 2.5, 1.5],
... }
... )
>>> df.select(cs.exclude("ba", "xx"))
shape: (3, 2)
┌─────┬──────┐
│ aa ┆ cc │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪══════╡
│ 1 ┆ null │
│ 2 ┆ 2.5 │
│ 3 ┆ 1.5 │
└─────┴──────┘
Exclude using a column name, a selector, and a dtype:
>>> df.select(cs.exclude("aa", cs.string(), pl.UInt32))
shape: (3, 1)
┌──────┐
│ cc │
│ --- │
│ f64 │
╞══════╡
│ null │
│ 2.5 │
│ 1.5 │
└──────┘
expand_selector(target: 'DataFrame | LazyFrame | Mapping[str, PolarsDataType]', selector: 'Selector | Expr', *, strict: 'bool' = True) -> 'tuple[str, ...]'
Expand selector to column names, with respect to a specific frame or target schema.
.. versionadded:: 0.20.30
The `strict` parameter was added.
Parameters
----------
target
A Polars DataFrame, LazyFrame or Schema.
selector
An arbitrary polars selector (or compound selector).
strict
Setting False additionally allows for a broader range of column selection
expressions (such as bare columns or use of `.exclude()`) to be expanded,
not just the dedicated selectors.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "colx": ["a", "b", "c"],
... "coly": [123, 456, 789],
... "colz": [2.0, 5.5, 8.0],
... }
... )
Expand selector with respect to an existing `DataFrame`:
>>> cs.expand_selector(df, cs.numeric())
('coly', 'colz')
>>> cs.expand_selector(df, cs.first() | cs.last())
('colx', 'colz')
This also works with `LazyFrame`:
>>> cs.expand_selector(df.lazy(), ~(cs.first() | cs.last()))
('coly',)
Expand selector with respect to a standalone `Schema` dict:
>>> schema = {
... "id": pl.Int64,
... "desc": pl.String,
... "count": pl.UInt32,
... "value": pl.Float64,
... }
>>> cs.expand_selector(schema, cs.string() | cs.float())
('desc', 'value')
Allow for non-strict selection expressions (such as those
including use of an `.exclude()` constraint) to be expanded:
>>> cs.expand_selector(schema, cs.numeric().exclude("id"), strict=False)
('count', 'value')
first(*, strict: 'bool' = True) -> 'Selector'
Select the first column in the current scope.
See Also
--------
all : Select all columns.
last : Select the last column in the current scope.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "zap": [0, 1],
... }
... )
Select the first column:
>>> df.select(cs.first())
shape: (2, 1)
┌─────┐
│ foo │
│ --- │
│ str │
╞═════╡
│ x │
│ y │
└─────┘
Select everything *except* for the first column:
>>> df.select(~cs.first())
shape: (2, 3)
┌─────┬─────┬─────┐
│ bar ┆ baz ┆ zap │
│ --- ┆ --- ┆ --- │
│ i64 ┆ f64 ┆ i64 │
╞═════╪═════╪═════╡
│ 123 ┆ 2.0 ┆ 0 │
│ 456 ┆ 5.5 ┆ 1 │
└─────┴─────┴─────┘
float() -> 'Selector'
Select all float columns.
See Also
--------
integer : Select all integer columns.
numeric : Select all numeric columns.
signed_integer : Select all signed integer columns.
unsigned_integer : Select all unsigned integer columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "zap": [0.0, 1.0],
... },
... schema_overrides={"baz": pl.Float32, "zap": pl.Float64},
... )
Select all float columns:
>>> df.select(cs.float())
shape: (2, 2)
┌─────┬─────┐
│ baz ┆ zap │
│ --- ┆ --- │
│ f32 ┆ f64 │
╞═════╪═════╡
│ 2.0 ┆ 0.0 │
│ 5.5 ┆ 1.0 │
└─────┴─────┘
Select all columns *except* for those that are float:
>>> df.select(~cs.float())
shape: (2, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═════╡
│ x ┆ 123 │
│ y ┆ 456 │
└─────┴─────┘
integer() -> 'Selector'
Select all integer columns.
See Also
--------
by_dtype : Select columns by dtype.
float : Select all float columns.
numeric : Select all numeric columns.
signed_integer : Select all signed integer columns.
unsigned_integer : Select all unsigned integer columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "zap": [0, 1],
... }
... )
Select all integer columns:
>>> df.select(cs.integer())
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ zap │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 123 ┆ 0 │
│ 456 ┆ 1 │
└─────┴─────┘
Select all columns *except* for those that are integer :
>>> df.select(~cs.integer())
shape: (2, 2)
┌─────┬─────┐
│ foo ┆ baz │
│ --- ┆ --- │
│ str ┆ f64 │
╞═════╪═════╡
│ x ┆ 2.0 │
│ y ┆ 5.5 │
└─────┴─────┘
is_selector(obj: 'Any') -> 'TypeIs[Selector]'
Indicate whether the given object/expression is a selector.
Examples
--------
>>> from polars.selectors import is_selector
>>> import polars.selectors as cs
>>> is_selector(pl.col("colx"))
False
>>> is_selector(cs.first() | cs.last())
True
last(*, strict: 'bool' = True) -> 'Selector'
Select the last column in the current scope.
See Also
--------
all : Select all columns.
first : Select the first column in the current scope.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "zap": [0, 1],
... }
... )
Select the last column:
>>> df.select(cs.last())
shape: (2, 1)
┌─────┐
│ zap │
│ --- │
│ i64 │
╞═════╡
│ 0 │
│ 1 │
└─────┘
Select everything *except* for the last column:
>>> df.select(~cs.last())
shape: (2, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ baz │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ f64 │
╞═════╪═════╪═════╡
│ x ┆ 123 ┆ 2.0 │
│ y ┆ 456 ┆ 5.5 │
└─────┴─────┴─────┘
list(inner: 'None | Selector' = None) -> 'Selector'
Select all list columns.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
See Also
--------
by_dtype : Select all columns matching the given dtype(s).
array : Select all array columns.
nested : Select all nested columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": [["xx", "yy"], ["x"]],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... },
... )
Select all list columns:
>>> df.select(cs.list())
shape: (2, 1)
┌──────────────┐
│ foo │
│ --- │
│ list[str] │
╞══════════════╡
│ ["xx", "yy"] │
│ ["x"] │
└──────────────┘
Select all columns *except* for those that are list:
>>> df.select(~cs.list())
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 123 ┆ 2.0 │
│ 456 ┆ 5.5 │
└─────┴─────┘
Select all list columns with a certain matching inner type:
>>> df.select(cs.list(cs.string()))
shape: (2, 1)
┌──────────────┐
│ foo │
│ --- │
│ list[str] │
╞══════════════╡
│ ["xx", "yy"] │
│ ["x"] │
└──────────────┘
>>> df.select(cs.list(cs.integer()))
shape: (0, 0)
┌┐
╞╡
└┘
matches(pattern: 'str') -> 'Selector'
Select all columns that match the given regex pattern.
See Also
--------
contains : Select all columns that contain the given substring.
ends_with : Select all columns that end with the given substring(s).
starts_with : Select all columns that start with the given substring(s).
Parameters
----------
pattern
A valid regular expression pattern, compatible with the `regex crate
<https://docs.rs/regex/latest/regex/>`_.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "zap": [0, 1],
... }
... )
Match column names containing an 'a', preceded by a character that is not 'z':
>>> df.select(cs.matches("[^z]a"))
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 123 ┆ 2.0 │
│ 456 ┆ 5.5 │
└─────┴─────┘
Do not match column names ending in 'R' or 'z' (case-insensitively):
>>> df.select(~cs.matches(r"(?i)R|z$"))
shape: (2, 2)
┌─────┬─────┐
│ foo ┆ zap │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═════╡
│ x ┆ 0 │
│ y ┆ 1 │
└─────┴─────┘
nested() -> 'Selector'
Select all nested columns.
A nested column is a list, array or struct.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
See Also
--------
by_dtype : Select all columns matching the given dtype(s).
list : Select all list columns.
array : Select all array columns.
struct : Select all struct columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": [{"a": "xx", "b": "z"}, {"a": "x", "b": "y"}],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "wow": [[1, 2], [3]],
... },
... )
Select all nested columns:
>>> df.select(cs.nested())
shape: (2, 2)
┌────────────┬───────────┐
│ foo ┆ wow │
│ --- ┆ --- │
│ struct[2] ┆ list[i64] │
╞════════════╪═══════════╡
│ {"xx","z"} ┆ [1, 2] │
│ {"x","y"} ┆ [3] │
└────────────┴───────────┘
Select all columns *except* for those that are nested:
>>> df.select(~cs.nested())
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 123 ┆ 2.0 │
│ 456 ┆ 5.5 │
└─────┴─────┘
numeric() -> 'Selector'
Select all numeric columns.
See Also
--------
by_dtype : Select columns by dtype.
float : Select all float columns.
integer : Select all integer columns.
signed_integer : Select all signed integer columns.
unsigned_integer : Select all unsigned integer columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": ["x", "y"],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... "zap": [0, 0],
... },
... schema_overrides={"bar": pl.Int16, "baz": pl.Float32, "zap": pl.UInt8},
... )
Match all numeric columns:
>>> df.select(cs.numeric())
shape: (2, 3)
┌─────┬─────┬─────┐
│ bar ┆ baz ┆ zap │
│ --- ┆ --- ┆ --- │
│ i16 ┆ f32 ┆ u8 │
╞═════╪═════╪═════╡
│ 123 ┆ 2.0 ┆ 0 │
│ 456 ┆ 5.5 ┆ 0 │
└─────┴─────┴─────┘
Match all columns *except* for those that are numeric:
>>> df.select(~cs.numeric())
shape: (2, 1)
┌─────┐
│ foo │
│ --- │
│ str │
╞═════╡
│ x │
│ y │
└─────┘
signed_integer() -> 'Selector'
Select all signed integer columns.
See Also
--------
by_dtype : Select columns by dtype.
float : Select all float columns.
integer : Select all integer columns.
numeric : Select all numeric columns.
unsigned_integer : Select all unsigned integer columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": [-123, -456],
... "bar": [3456, 6789],
... "baz": [7654, 4321],
... "zap": ["ab", "cd"],
... },
... schema_overrides={"bar": pl.UInt32, "baz": pl.UInt64},
... )
Select all signed integer columns:
>>> df.select(cs.signed_integer())
shape: (2, 1)
┌──────┐
│ foo │
│ --- │
│ i64 │
╞══════╡
│ -123 │
│ -456 │
└──────┘
>>> df.select(~cs.signed_integer())
shape: (2, 3)
┌──────┬──────┬─────┐
│ bar ┆ baz ┆ zap │
│ --- ┆ --- ┆ --- │
│ u32 ┆ u64 ┆ str │
╞══════╪══════╪═════╡
│ 3456 ┆ 7654 ┆ ab │
│ 6789 ┆ 4321 ┆ cd │
└──────┴──────┴─────┘
Select all integer columns (both signed and unsigned):
>>> df.select(cs.integer())
shape: (2, 3)
┌──────┬──────┬──────┐
│ foo ┆ bar ┆ baz │
│ --- ┆ --- ┆ --- │
│ i64 ┆ u32 ┆ u64 │
╞══════╪══════╪══════╡
│ -123 ┆ 3456 ┆ 7654 │
│ -456 ┆ 6789 ┆ 4321 │
└──────┴──────┴──────┘
starts_with(*prefix: 'str') -> 'Selector'
Select columns that start with the given substring(s).
Parameters
----------
prefix
Substring(s) that matching column names should start with.
See Also
--------
contains : Select all columns that contain the given substring.
ends_with : Select all columns that end with the given substring(s).
matches : Select all columns that match the given regex pattern.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": [1.0, 2.0],
... "bar": [3.0, 4.0],
... "baz": [5, 6],
... "zap": [7, 8],
... }
... )
Match columns starting with a 'b':
>>> df.select(cs.starts_with("b"))
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ f64 ┆ i64 │
╞═════╪═════╡
│ 3.0 ┆ 5 │
│ 4.0 ┆ 6 │
└─────┴─────┘
Match columns starting with *either* the letter 'b' or 'z':
>>> df.select(cs.starts_with("b", "z"))
shape: (2, 3)
┌─────┬─────┬─────┐
│ bar ┆ baz ┆ zap │
│ --- ┆ --- ┆ --- │
│ f64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 3.0 ┆ 5 ┆ 7 │
│ 4.0 ┆ 6 ┆ 8 │
└─────┴─────┴─────┘
Match all columns *except* for those starting with 'b':
>>> df.select(~cs.starts_with("b"))
shape: (2, 2)
┌─────┬─────┐
│ foo ┆ zap │
│ --- ┆ --- │
│ f64 ┆ i64 │
╞═════╪═════╡
│ 1.0 ┆ 7 │
│ 2.0 ┆ 8 │
└─────┴─────┘
string(*, include_categorical: 'bool' = False) -> 'Selector'
Select all String (and, optionally, Categorical) string columns.
See Also
--------
binary : Select all binary columns.
by_dtype : Select all columns matching the given dtype(s).
categorical: Select all categorical columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "w": ["xx", "yy", "xx", "yy", "xx"],
... "x": [1, 2, 1, 4, -2],
... "y": [3.0, 4.5, 1.0, 2.5, -2.0],
... "z": ["a", "b", "a", "b", "b"],
... },
... ).with_columns(
... z=pl.col("z").cast(pl.Categorical()),
... )
Group by all string columns, sum the numeric columns, then sort by the string cols:
>>> df.group_by(cs.string()).agg(cs.numeric().sum()).sort(by=cs.string())
shape: (2, 3)
┌─────┬─────┬─────┐
│ w ┆ x ┆ y │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ f64 │
╞═════╪═════╪═════╡
│ xx ┆ 0 ┆ 2.0 │
│ yy ┆ 6 ┆ 7.0 │
└─────┴─────┴─────┘
Group by all string *and* categorical columns:
>>> df.group_by(cs.string(include_categorical=True)).agg(cs.numeric().sum()).sort(
... by=cs.string(include_categorical=True)
... )
shape: (3, 4)
┌─────┬─────┬─────┬──────┐
│ w ┆ z ┆ x ┆ y │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ cat ┆ i64 ┆ f64 │
╞═════╪═════╪═════╪══════╡
│ xx ┆ a ┆ 2 ┆ 4.0 │
│ xx ┆ b ┆ -2 ┆ -2.0 │
│ yy ┆ b ┆ 6 ┆ 7.0 │
└─────┴─────┴─────┴──────┘
struct() -> 'Selector'
Select all struct columns.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
See Also
--------
by_dtype : Select all columns matching the given dtype(s).
list : Select all list columns.
array : Select all array columns.
nested : Select all nested columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": [{"a": "xx", "b": "z"}, {"a": "x", "b": "y"}],
... "bar": [123, 456],
... "baz": [2.0, 5.5],
... },
... )
Select all struct columns:
>>> df.select(cs.struct())
shape: (2, 1)
┌────────────┐
│ foo │
│ --- │
│ struct[2] │
╞════════════╡
│ {"xx","z"} │
│ {"x","y"} │
└────────────┘
Select all columns *except* for those that are struct:
>>> df.select(~cs.struct())
shape: (2, 2)
┌─────┬─────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ i64 ┆ f64 │
╞═════╪═════╡
│ 123 ┆ 2.0 │
│ 456 ┆ 5.5 │
└─────┴─────┘
temporal() -> 'Selector'
Select all temporal columns.
See Also
--------
by_dtype : Select all columns matching the given dtype(s).
date : Select all date columns.
datetime : Select all datetime columns, optionally filtering by time unit/zone.
duration : Select all duration columns, optionally filtering by time unit.
time : Select all time columns.
Examples
--------
>>> from datetime import date, time
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "dt": [date(2021, 1, 1), date(2021, 1, 2)],
... "tm": [time(12, 0, 0), time(20, 30, 45)],
... "value": [1.2345, 2.3456],
... }
... )
Match all temporal columns:
>>> df.select(cs.temporal())
shape: (2, 2)
┌────────────┬──────────┐
│ dt ┆ tm │
│ --- ┆ --- │
│ date ┆ time │
╞════════════╪══════════╡
│ 2021-01-01 ┆ 12:00:00 │
│ 2021-01-02 ┆ 20:30:45 │
└────────────┴──────────┘
Match all temporal columns *except* for time columns:
>>> df.select(cs.temporal() - cs.time())
shape: (2, 1)
┌────────────┐
│ dt │
│ --- │
│ date │
╞════════════╡
│ 2021-01-01 │
│ 2021-01-02 │
└────────────┘
Match all columns *except* for temporal columns:
>>> df.select(~cs.temporal())
shape: (2, 1)
┌────────┐
│ value │
│ --- │
│ f64 │
╞════════╡
│ 1.2345 │
│ 2.3456 │
└────────┘
time() -> 'Selector'
Select all time columns.
See Also
--------
date : Select all date columns.
datetime : Select all datetime columns, optionally filtering by time unit/zone.
duration : Select all duration columns, optionally filtering by time unit.
temporal : Select all temporal columns.
Examples
--------
>>> from datetime import date, datetime, time
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "dtm": [datetime(2001, 5, 7, 10, 25), datetime(2031, 12, 31, 0, 30)],
... "dt": [date(1999, 12, 31), date(2024, 8, 9)],
... "tm": [time(0, 0, 0), time(23, 59, 59)],
... },
... )
Select all time columns:
>>> df.select(cs.time())
shape: (2, 1)
┌──────────┐
│ tm │
│ --- │
│ time │
╞══════════╡
│ 00:00:00 │
│ 23:59:59 │
└──────────┘
Select all columns *except* for those that are times:
>>> df.select(~cs.time())
shape: (2, 2)
┌─────────────────────┬────────────┐
│ dtm ┆ dt │
│ --- ┆ --- │
│ datetime[μs] ┆ date │
╞═════════════════════╪════════════╡
│ 2001-05-07 10:25:00 ┆ 1999-12-31 │
│ 2031-12-31 00:30:00 ┆ 2024-08-09 │
└─────────────────────┴────────────┘
unsigned_integer() -> 'Selector'
Select all unsigned integer columns.
See Also
--------
by_dtype : Select columns by dtype.
float : Select all float columns.
integer : Select all integer columns.
numeric : Select all numeric columns.
signed_integer : Select all signed integer columns.
Examples
--------
>>> import polars.selectors as cs
>>> df = pl.DataFrame(
... {
... "foo": [-123, -456],
... "bar": [3456, 6789],
... "baz": [7654, 4321],
... "zap": ["ab", "cd"],
... },
... schema_overrides={"bar": pl.UInt32, "baz": pl.UInt64},
... )
Select all unsigned integer columns:
>>> df.select(cs.unsigned_integer())
shape: (2, 2)
┌──────┬──────┐
│ bar ┆ baz │
│ --- ┆ --- │
│ u32 ┆ u64 │
╞══════╪══════╡
│ 3456 ┆ 7654 │
│ 6789 ┆ 4321 │
└──────┴──────┘
Select all columns *except* for those that are unsigned integers:
>>> df.select(~cs.unsigned_integer())
shape: (2, 2)
┌──────┬─────┐
│ foo ┆ zap │
│ --- ┆ --- │
│ i64 ┆ str │
╞══════╪═════╡
│ -123 ┆ ab │
│ -456 ┆ cd │
└──────┴─────┘
Select all integer columns (both signed and unsigned):
>>> df.select(cs.integer())
shape: (2, 3)
┌──────┬──────┬──────┐
│ foo ┆ bar ┆ baz │
│ --- ┆ --- ┆ --- │
│ i64 ┆ u32 ┆ u64 │
╞══════╪══════╪══════╡
│ -123 ┆ 3456 ┆ 7654 │
│ -456 ┆ 6789 ┆ 4321 │
└──────┴──────┴──────┘
DATA
__all__ = ['Selector', 'all', 'alpha', 'alphanumeric', 'array', 'binar...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\selectors.py
polars.self_dtype
Python Library Documentation: function self_dtype in module polars.functions.datatype
self_dtype() -> 'pl.DataTypeExpr'
Get the dtype of `self` in `map_elements` and `map_batches`.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
polars.series
Python Library Documentation: package polars.series in polars
NAME
polars.series
PACKAGE CONTENTS
array
binary
categorical
datetime
ext
list
plotting
series
string
struct
utils
CLASSES
builtins.object
polars.series.series.Series
class Series(builtins.object)
| Series(name: 'str_ | ArrayLike | None' = None, values: 'ArrayLike | None' = None, dtype: 'PolarsDataType | None' = None, *, strict: 'bool' = True, nan_to_null: 'bool' = False) -> 'None'
|
| A Series represents a single column in a Polars DataFrame.
|
| Parameters
| ----------
| name : str, default None
| Name of the Series. Will be used as a column name when used in a DataFrame.
| When not specified, name is set to an empty string.
| values : ArrayLike, default None
| One-dimensional data in various forms. Supported are: Sequence, Series,
| pyarrow Array, and numpy ndarray.
| dtype : DataType, default None
| Data type of the resulting Series. If set to `None` (default), the data type is
| inferred from the `values` input. The strategy for data type inference depends
| on the `strict` parameter:
|
| - If `strict` is set to True (default), the inferred data type is equal to the
| first non-null value, or `Null` if all values are null.
| - If `strict` is set to False, the inferred data type is the supertype of the
| values, or :class:`Object` if no supertype can be found. **WARNING**: A full
| pass over the values is required to determine the supertype.
| - If no values were passed, the resulting data type is :class:`Null`.
|
| strict : bool, default True
| Throw an error if any value does not exactly match the given or inferred data
| type. If set to `False`, values that do not match the data type are cast to
| that data type or, if casting is not possible, set to null instead.
| nan_to_null : bool, default False
| In case a numpy array is used to create this Series, indicate how to deal
| with np.nan values. (This parameter is a no-op on non-numpy data).
|
| Examples
| --------
| Constructing a Series by specifying name and values positionally:
|
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| Notice that the dtype is automatically inferred as a polars Int64:
|
| >>> s.dtype
| Int64
|
| Constructing a Series with a specific dtype:
|
| >>> s2 = pl.Series("a", [1, 2, 3], dtype=pl.Float32)
| >>> s2
| shape: (3,)
| Series: 'a' [f32]
| [
| 1.0
| 2.0
| 3.0
| ]
|
| It is possible to construct a Series with values as the first positional argument.
| This syntax considered an anti-pattern, but it can be useful in certain
| scenarios. You must specify any other arguments through keywords.
|
| >>> s3 = pl.Series([1, 2, 3])
| >>> s3
| shape: (3,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| ]
|
| Methods defined here:
|
| __abs__(self) -> 'Series'
|
| __add__(self, other: 'Any') -> 'Series | DataFrame | Expr'
|
| __and__(self, other: 'Any') -> 'Expr | Series'
|
| __array__(self, dtype: 'np.dtype[Any] | None' = None, copy: 'bool | None' = None) -> 'np.ndarray[Any, Any]'
| Return a NumPy ndarray with the given data type.
|
| This method ensures a Polars Series can be treated as a NumPy ndarray.
| It enables `np.asarray` and NumPy universal functions.
|
| See the NumPy documentation for more information:
| https://numpy.org/doc/stable/user/basics.interoperability.html#the-array-method
|
| See Also
| --------
| __array_ufunc__
|
| __array_ufunc__(self, ufunc: 'np.ufunc', method: 'str_', *inputs: 'Any', **kwargs: 'Any') -> 'Series'
| Numpy universal functions.
|
| __arrow_c_stream__(self, requested_schema: 'object | None' = None) -> 'object'
| Export a Series via the Arrow PyCapsule Interface.
|
| https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html
|
| __bool__(self) -> 'NoReturn'
|
| __contains__(self, item: 'Any') -> 'bool'
|
| __copy__(self) -> 'Self'
|
| __deepcopy__(self, memo: 'None' = None) -> 'Self'
|
| __eq__(self, other: 'object') -> 'Series | Expr'
| Return self==value.
|
| __floordiv__(self, other: 'Any') -> 'Series | Expr'
|
| __ge__(self, other: 'Any') -> 'Series | Expr'
| Return self>=value.
|
| __getitem__(self, key: 'SingleIndexSelector | MultiIndexSelector') -> 'Any | Series'
| Get part of the Series as a new Series or scalar.
|
| Parameters
| ----------
| key
| Row(s) to select.
|
| Returns
| -------
| Series or scalar, depending on `key`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 4, 2])
| >>> s[0]
| 1
| >>> s[0:2]
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 4
| ]
|
| __getstate__(self) -> 'bytes'
| Helper for pickle.
|
| __gt__(self, other: 'Any') -> 'Series | Expr'
| Return self>value.
|
| __init__(self, name: 'str_ | ArrayLike | None' = None, values: 'ArrayLike | None' = None, dtype: 'PolarsDataType | None' = None, *, strict: 'bool' = True, nan_to_null: 'bool' = False) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __invert__(self) -> 'Series'
|
| __iter__(self) -> 'Generator[Any]'
|
| __le__(self, other: 'Any') -> 'Series | Expr'
| Return self<=value.
|
| __len__(self) -> 'int'
|
| __lt__(self, other: 'Any') -> 'Series | Expr'
| Return self<value.
|
| __matmul__(self, other: 'Any') -> 'float | Series | None'
|
| __mod__(self, other: 'Any') -> 'Series | Expr'
|
| __mul__(self, other: 'Any') -> 'Series | DataFrame | Expr'
|
| __ne__(self, other: 'object') -> 'Series | Expr'
| Return self!=value.
|
| __neg__(self) -> 'Series'
|
| __or__(self, other: 'Any') -> 'Expr | Series'
| Return self|value.
|
| __pos__(self) -> 'Series'
|
| __pow__(self, exponent: 'int | float | Series') -> 'Series'
|
| __radd__(self, other: 'Any') -> 'Series'
|
| __rand__(self, other: 'Any') -> 'Expr | Series'
|
| __repr__(self) -> 'str_'
| Return repr(self).
|
| __rfloordiv__(self, other: 'Any') -> 'Series'
|
| __rmatmul__(self, other: 'Any') -> 'float | Series | None'
|
| __rmod__(self, other: 'Any') -> 'Series'
|
| __rmul__(self, other: 'Any') -> 'Series'
|
| __ror__(self, other: 'Any') -> 'Expr | Series'
| Return value|self.
|
| __rpow__(self, other: 'Any') -> 'Series'
|
| __rsub__(self, other: 'Any') -> 'Series'
|
| __rtruediv__(self, other: 'Any') -> 'Series'
|
| __rxor__(self, other: 'Any') -> 'Expr | Series'
|
| __setitem__(self, key: 'int | Series | np.ndarray[Any, Any] | Sequence[object] | tuple[object]', value: 'Any') -> 'None'
| Set Series values in-place using a single index, boolean mask, or index array.
|
| Parameters
| ----------
| key
| Determines which elements to update:
|
| - ``int``: a single row index.
| - ``Series`` (Boolean): a boolean mask.
| - ``Series`` (Integer): an index array.
| - ``ndarray``: a NumPy boolean mask or integer index array.
| - ``list`` / ``tuple``: an index sequence (cast to UInt32).
| value
| Scalar or sequence of values to assign.
|
| Examples
| --------
| Set a single element by index:
|
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s[0] = 10
| >>> s
| shape: (3,)
| Series: 'a' [i64]
| [
| 10
| 2
| 3
| ]
|
| Set elements with a boolean mask:
|
| >>> s[pl.Series([False, True, True])] = 99
| >>> s
| shape: (3,)
| Series: 'a' [i64]
| [
| 10
| 99
| 99
| ]
|
| __setstate__(self, state: 'bytes') -> 'None'
|
| __str__(self) -> 'str_'
| Return str(self).
|
| __sub__(self, other: 'Any') -> 'Series | Expr'
|
| __truediv__(self, other: 'Any') -> 'Series | Expr'
|
| __xor__(self, other: 'Any') -> 'Expr | Series'
|
| abs(self) -> 'Series'
| Compute absolute values.
|
| Same as `abs(series)`.
|
| Examples
| --------
| >>> s = pl.Series([1, -2, -3])
| >>> s.abs()
| shape: (3,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| ]
|
| alias(self, name: 'str_') -> 'Series'
| Rename the series.
|
| Parameters
| ----------
| name
| The new name.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.alias("b")
| shape: (3,)
| Series: 'b' [i64]
| [
| 1
| 2
| 3
| ]
|
| all(self, *, ignore_nulls: 'bool' = True) -> 'bool | None'
| Return whether all values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `True`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `False` values,
| the output is `None`.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| bool or None
|
| Examples
| --------
| >>> pl.Series([True, True]).all()
| True
| >>> pl.Series([False, True]).all()
| False
| >>> pl.Series([None, True]).all()
| True
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> pl.Series([None, True]).all(ignore_nulls=False) # Returns None
|
| any(self, *, ignore_nulls: 'bool' = True) -> 'bool | None'
| Return whether any of the values in the column are `True`.
|
| Only works on columns of data type :class:`Boolean`.
|
| Parameters
| ----------
| ignore_nulls
| * If set to `True` (default), null values are ignored. If there
| are no non-null values, the output is `False`.
| * If set to `False`, `Kleene logic`_ is used to deal with nulls:
| if the column contains any null values and no `True` values,
| the output is `None`.
|
| .. _Kleene logic: https://en.wikipedia.org/wiki/Three-valued_logic
|
| Returns
| -------
| bool or None
|
| Examples
| --------
| >>> pl.Series([True, False]).any()
| True
| >>> pl.Series([False, False]).any()
| False
| >>> pl.Series([None, False]).any()
| False
|
| Enable Kleene logic by setting `ignore_nulls=False`.
|
| >>> pl.Series([None, False]).any(ignore_nulls=False) # Returns None
|
| append(self, other: 'Series') -> 'Self'
| Append a Series to this one.
|
| The resulting series will consist of multiple chunks.
|
| Parameters
| ----------
| other
| Series to append.
|
| Warnings
| --------
| This method modifies the series in-place. The series is returned for
| convenience only.
|
| See Also
| --------
| extend
|
| Examples
| --------
| >>> a = pl.Series("a", [1, 2, 3])
| >>> b = pl.Series("b", [4, 5])
| >>> a.append(b)
| shape: (5,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| ]
|
| The resulting series will consist of multiple chunks.
|
| >>> a.n_chunks()
| 2
|
| approx_n_unique(self) -> 'PythonLiteral | None'
| Approximate count of unique values.
|
| This is done using the HyperLogLog++ algorithm for cardinality estimation.
|
| arccos(self) -> 'Series'
| Compute the element-wise value for the inverse cosine.
|
| Notes
| -----
| The returned value is in radians.
| To convert from radians to degrees,
| call :meth:`.degrees() <polars.Series.degrees>`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.arccos()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.0
| 1.570796
| 3.141593
| ]
|
| arccosh(self) -> 'Series'
| Compute the element-wise value for the inverse hyperbolic cosine.
|
| Examples
| --------
| >>> s = pl.Series("a", [5.0, 1.0, 0.0, -1.0])
| >>> s.arccosh()
| shape: (4,)
| Series: 'a' [f64]
| [
| 2.292432
| 0.0
| NaN
| NaN
| ]
|
| arcsin(self) -> 'Series'
| Compute the element-wise value for the inverse sine.
|
| Notes
| -----
| The returned value is in radians.
| To convert from radians to degrees,
| call :meth:`.degrees() <polars.Series.degrees>`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.arcsin()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.570796
| 0.0
| -1.570796
| ]
|
| arcsinh(self) -> 'Series'
| Compute the element-wise value for the inverse hyperbolic sine.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.arcsinh()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.881374
| 0.0
| -0.881374
| ]
|
| arctan(self) -> 'Series'
| Compute the element-wise value for the inverse tangent.
|
| Notes
| -----
| The returned value is in radians.
| To convert from radians to degrees,
| call :meth:`.degrees() <polars.Series.degrees>`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.arctan()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.785398
| 0.0
| -0.785398
| ]
|
| arctanh(self) -> 'Series'
| Compute the element-wise value for the inverse hyperbolic tangent.
|
| Examples
| --------
| >>> s = pl.Series("a", [2.0, 1.0, 0.5, 0.0, -0.5, -1.0, -1.1])
| >>> s.arctanh()
| shape: (7,)
| Series: 'a' [f64]
| [
| NaN
| inf
| 0.549306
| 0.0
| -0.549306
| -inf
| NaN
| ]
|
| arg_max(self) -> 'int | None'
| Get the index of the maximal value.
|
| Returns
| -------
| int
|
| Examples
| --------
| >>> s = pl.Series("a", [3, 2, 1])
| >>> s.arg_max()
| 0
|
| arg_min(self) -> 'int | None'
| Get the index of the minimal value.
|
| Returns
| -------
| int
|
| Examples
| --------
| >>> s = pl.Series("a", [3, 2, 1])
| >>> s.arg_min()
| 2
|
| arg_sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'Series'
| Get the index values that would sort this Series.
|
| Parameters
| ----------
| descending
| Sort in descending order.
| nulls_last
| Place null values last instead of first.
|
| See Also
| --------
| Series.gather: Take values by index.
| Series.rank : Get the rank of each row.
|
| Examples
| --------
| >>> s = pl.Series("a", [5, 3, 4, 1, 2])
| >>> s.arg_sort()
| shape: (5,)
| Series: 'a' [u32]
| [
| 3
| 4
| 1
| 2
| 0
| ]
|
| arg_true(self) -> 'Series'
| Get index values where Boolean Series evaluate True.
|
| Returns
| -------
| Series
| Series of data type :class:`UInt32`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> (s == 2).arg_true()
| shape: (1,)
| Series: 'a' [u32]
| [
| 1
| ]
|
| arg_unique(self) -> 'Series'
| Get unique index as Series.
|
| Returns
| -------
| Series
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.arg_unique()
| shape: (3,)
| Series: 'a' [u32]
| [
| 0
| 1
| 3
| ]
|
| backward_fill(self, limit: 'int | None' = None) -> 'Series'
| Fill missing values with the next non-null value.
|
| This is an alias of `.fill_null(strategy="backward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to backward fill.
|
| See Also
| --------
| fill_null
| forward_fill
| shift
|
| bitwise_and(self) -> 'PythonLiteral | None'
| Perform an aggregation of bitwise ANDs.
|
| bitwise_count_ones(self) -> 'Self'
| Evaluate the number of set bits.
|
| bitwise_count_zeros(self) -> 'Self'
| Evaluate the number of unset bits.
|
| bitwise_leading_ones(self) -> 'Self'
| Evaluate the number most-significant set bits before seeing an unset bit.
|
| bitwise_leading_zeros(self) -> 'Self'
| Evaluate the number most-significant unset bits before seeing a set bit.
|
| bitwise_or(self) -> 'PythonLiteral | None'
| Perform an aggregation of bitwise ORs.
|
| bitwise_trailing_ones(self) -> 'Self'
| Evaluate the number least-significant set bits before seeing an unset bit.
|
| bitwise_trailing_zeros(self) -> 'Self'
| Evaluate the number least-significant unset bits before seeing a set bit.
|
| bitwise_xor(self) -> 'PythonLiteral | None'
| Perform an aggregation of bitwise XORs.
|
| bottom_k(self, k: 'int' = 5) -> 'Series'
| Return the `k` smallest elements.
|
| Non-null elements are always preferred over null elements. The output is
| not guaranteed to be in any particular order, call :func:`sort` after
| this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k_by
|
| Examples
| --------
| >>> s = pl.Series("a", [2, 5, 1, 4, 3])
| >>> s.bottom_k(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| bottom_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Series'
| Return the `k` smallest elements of the `by` column.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| Parameters
| ----------
| by
| Column used to determine the smallest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` largest elements of the `by` column( (instead of the `k`
| smallest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| top_k_by
| bottom_k
|
| Examples
| --------
| >>> s = pl.Series("a", [2, 5, 1, 4, 3])
| >>> s.bottom_k_by("a", 3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| cast(self, dtype: 'type[int | float | str_ | bool] | PolarsDataType', *, strict: 'bool' = True, wrap_numerical: 'bool' = False) -> 'Self'
| Cast between data types.
|
| Parameters
| ----------
| dtype
| DataType to cast to.
| strict
| If True invalid casts generate exceptions instead of `null`\s.
| wrap_numerical
| If True numeric casts wrap overflowing values instead of
| marking the cast as invalid.
|
| Examples
| --------
| >>> s = pl.Series("a", [True, False, True])
| >>> s
| shape: (3,)
| Series: 'a' [bool]
| [
| true
| false
| true
| ]
|
| >>> s.cast(pl.UInt32)
| shape: (3,)
| Series: 'a' [u32]
| [
| 1
| 0
| 1
| ]
|
| cbrt(self) -> 'Series'
| Compute the cube root of the elements.
|
| Optimization for
|
| >>> pl.Series([1, 2]) ** (1.0 / 3)
| shape: (2,)
| Series: '' [f64]
| [
| 1.0
| 1.259921
| ]
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.cbrt()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 1.259921
| 1.44225
| ]
|
| ceil(self) -> 'Series'
| Rounds up to the nearest integer value.
|
| Only works on floating point Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.12345, 2.56789, 3.901234])
| >>> s.ceil()
| shape: (3,)
| Series: 'a' [f64]
| [
| 2.0
| 3.0
| 4.0
| ]
|
| chunk_lengths(self) -> 'list_[int]'
| Get the length of each individual chunk.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("a", [4, 5, 6])
|
| Concatenate Series with rechunk = True
|
| >>> pl.concat([s, s2], rechunk=True).chunk_lengths()
| [6]
|
| Concatenate Series with rechunk = False
|
| >>> pl.concat([s, s2], rechunk=False).chunk_lengths()
| [3, 3]
|
| clear(self, n: 'int' = 0) -> 'Series'
| Create an empty copy of the current Series, with zero to 'n' elements.
|
| The copy has an identical name/dtype, but no data.
|
| Parameters
| ----------
| n
| Number of (empty) elements to return in the cleared frame.
|
| See Also
| --------
| clone : Cheap deepcopy/clone.
|
| Examples
| --------
| >>> s = pl.Series("a", [None, True, False])
| >>> s.clear()
| shape: (0,)
| Series: 'a' [bool]
| [
| ]
|
| >>> s.clear(n=2)
| shape: (2,)
| Series: 'a' [bool]
| [
| null
| null
| ]
|
| clip(self, lower_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None, upper_bound: 'NumericLiteral | TemporalLiteral | IntoExprColumn | None' = None) -> 'Series'
| Set values outside the given boundaries to the boundary value.
|
| Parameters
| ----------
| lower_bound
| Lower bound. Accepts expression input.
| Non-expression inputs are parsed as literals.
| If set to `None` (default), no lower bound is applied.
| upper_bound
| Upper bound. Accepts expression input.
| Non-expression inputs are parsed as literals.
| If set to `None` (default), no upper bound is applied.
|
| See Also
| --------
| when
|
| Notes
| -----
| This method only works for numeric and temporal columns. To clip other data
| types, consider writing a `when-then-otherwise` expression. See :func:`when`.
|
| Examples
| --------
| Specifying both a lower and upper bound:
|
| >>> s = pl.Series([-50, 5, 50, None])
| >>> s.clip(1, 10)
| shape: (4,)
| Series: '' [i64]
| [
| 1
| 5
| 10
| null
| ]
|
| Specifying only a single bound:
|
| >>> s.clip(upper_bound=10)
| shape: (4,)
| Series: '' [i64]
| [
| -50
| 5
| 10
| null
| ]
|
| clone(self) -> 'Self'
| Create a copy of this Series.
|
| This is a cheap operation that does not copy data.
|
| See Also
| --------
| clear : Create an empty copy of the current Series, with identical
| schema but no data.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.clone()
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| cos(self) -> 'Series'
| Compute the element-wise value for the cosine.
|
| Notes
| -----
| Input values are interpreted as radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Series.radians>`.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [0.0, math.pi / 2.0, math.pi])
| >>> s.cos()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.0
| 6.1232e-17
| -1.0
| ]
|
| cosh(self) -> 'Series'
| Compute the element-wise value for the hyperbolic cosine.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.cosh()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.543081
| 1.0
| 1.543081
| ]
|
| cot(self) -> 'Series'
| Compute the element-wise value for the cotangent.
|
| Notes
| -----
| Input values are interpreted as radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Series.radians>`.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [0.0, math.pi / 2.0, math.pi])
| >>> s.cot()
| shape: (3,)
| Series: 'a' [f64]
| [
| inf
| 6.1232e-17
| -8.1656e15
| ]
|
| count(self) -> 'int'
| Return the number of non-null elements in the column.
|
| See Also
| --------
| len
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, None])
| >>> s.count()
| 2
|
| cum_count(self, *, reverse: 'bool' = False) -> 'Self'
| Return the cumulative count of the non-null values in the column.
|
| Parameters
| ----------
| reverse
| Reverse the operation.
|
| Examples
| --------
| >>> s = pl.Series(["x", "k", None, "d"])
| >>> s.cum_count()
| shape: (4,)
| Series: '' [u32]
| [
| 1
| 2
| 2
| 3
| ]
|
| cum_max(self, *, reverse: 'bool' = False) -> 'Series'
| Get an array with the cumulative max computed at every element.
|
| Parameters
| ----------
| reverse
| reverse the operation.
|
| Examples
| --------
| >>> s = pl.Series("s", [3, 5, 1])
| >>> s.cum_max()
| shape: (3,)
| Series: 's' [i64]
| [
| 3
| 5
| 5
| ]
|
| cum_min(self, *, reverse: 'bool' = False) -> 'Series'
| Get an array with the cumulative min computed at every element.
|
| Parameters
| ----------
| reverse
| reverse the operation.
|
| Examples
| --------
| >>> s = pl.Series("s", [1, 2, 3])
| >>> s.cum_min()
| shape: (3,)
| Series: 's' [i64]
| [
| 1
| 1
| 1
| ]
|
| cum_prod(self, *, reverse: 'bool' = False) -> 'Series'
| Get an array with the cumulative product computed at every element.
|
| Parameters
| ----------
| reverse
| reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.cum_prod()
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 6
| ]
|
| cum_sum(self, *, reverse: 'bool' = False) -> 'Series'
| Get an array with the cumulative sum computed at every element.
|
| Parameters
| ----------
| reverse
| reverse the operation.
|
| Notes
| -----
| Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.cum_sum()
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 3
| 6
| ]
|
| cumulative_eval(self, expr: 'Expr', *, min_samples: 'int' = 1, parallel: 'bool' = False) -> 'Series'
| Run an expression over a sliding window that increases `1` slot every iteration.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| expr
| Expression to evaluate
| min_samples
| Number of valid values there should be in the window before the expression
| is evaluated. valid values = `length - null_count`
| parallel
| Run in parallel. Don't do this in a group by or another operation that
| already has much parallelization.
|
| Warnings
| --------
| This can be really slow as it can have `O(n^2)` complexity. Don't use this
| for operations that visit all elements.
|
| Examples
| --------
| >>> s = pl.Series("values", [1, 2, 3, 4, 5])
| >>> s.cumulative_eval(pl.element().first() - pl.element().last() ** 2)
| shape: (5,)
| Series: 'values' [i64]
| [
| 0
| -3
| -8
| -15
| -24
| ]
|
| cut(self, breaks: 'Sequence[float]', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, include_breaks: 'bool' = False) -> 'Series'
| Bin continuous values into discrete categories.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| breaks
| List of unique cut points.
| labels
| Names of the categories. The number of labels must be equal to the number
| of cut points plus one.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from an
| :class:`Enum` to a :class:`Struct`.
|
| Returns
| -------
| Series
| Series of data type :class:`Enum` if `include_breaks` is set to
| `False` (default), otherwise a Series of data type :class:`Struct`.
|
| See Also
| --------
| qcut
|
| Examples
| --------
| Divide the column into three categories.
|
| >>> s = pl.Series("foo", [-2, -1, 0, 1, 2])
| >>> s.cut([-1, 1], labels=["a", "b", "c"])
| shape: (5,)
| Series: 'foo' [enum]
| [
| "a"
| "a"
| "b"
| "b"
| "c"
| ]
|
| Create a DataFrame with the breakpoint and category for each value.
|
| >>> cut = s.cut([-1, 1], include_breaks=True).alias("cut")
| >>> s.to_frame().with_columns(cut).unnest("cut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ enum │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| degrees(self) -> 'Series'
| Convert from radians to degrees.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [x * math.pi for x in range(-4, 5)])
| >>> s.degrees()
| shape: (9,)
| Series: 'a' [f64]
| [
| -720.0
| -540.0
| -360.0
| -180.0
| 0.0
| 180.0
| 360.0
| 540.0
| 720.0
| ]
|
| describe(self, percentiles: 'Sequence[float] | float | None' = (0.25, 0.5, 0.75), interpolation: 'QuantileMethod' = 'nearest') -> 'DataFrame'
| Quick summary statistics of a Series.
|
| Series with mixed datatypes will return summary statistics for the datatype of
| the first value.
|
| Parameters
| ----------
| percentiles
| One or more percentiles to include in the summary statistics (if the
| Series has a numeric dtype). All values must be in the range `[0, 1]`.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method used when calculating percentiles.
|
| Notes
| -----
| The median is included by default as the 50% percentile.
|
| Returns
| -------
| DataFrame
| Mapping with summary statistics of a Series.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3, 4, 5])
| >>> s.describe()
| shape: (9, 2)
| ┌────────────┬──────────┐
| │ statistic ┆ value │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞════════════╪══════════╡
| │ count ┆ 5.0 │
| │ null_count ┆ 0.0 │
| │ mean ┆ 3.0 │
| │ std ┆ 1.581139 │
| │ min ┆ 1.0 │
| │ 25% ┆ 2.0 │
| │ 50% ┆ 3.0 │
| │ 75% ┆ 4.0 │
| │ max ┆ 5.0 │
| └────────────┴──────────┘
|
| Non-numeric data types may not have all statistics available.
|
| >>> s = pl.Series(["aa", "aa", None, "bb", "cc"])
| >>> s.describe()
| shape: (4, 2)
| ┌────────────┬───────┐
| │ statistic ┆ value │
| │ --- ┆ --- │
| │ str ┆ str │
| ╞════════════╪═══════╡
| │ count ┆ 4 │
| │ null_count ┆ 1 │
| │ min ┆ aa │
| │ max ┆ cc │
| └────────────┴───────┘
|
| diff(self, n: 'int' = 1, null_behavior: 'NullBehavior' = 'ignore') -> 'Series'
| Calculate the first discrete difference between shifted items.
|
| Parameters
| ----------
| n
| Number of slots to shift.
| null_behavior : {'ignore', 'drop'}
| How to handle null values.
|
| Examples
| --------
| >>> s = pl.Series("s", values=[20, 10, 30, 25, 35], dtype=pl.Int8)
| >>> s.diff()
| shape: (5,)
| Series: 's' [i8]
| [
| null
| -10
| 20
| -5
| 10
| ]
|
| >>> s.diff(n=2)
| shape: (5,)
| Series: 's' [i8]
| [
| null
| null
| 10
| 15
| 5
| ]
|
| >>> s.diff(n=2, null_behavior="drop")
| shape: (3,)
| Series: 's' [i8]
| [
| 10
| 15
| 5
| ]
|
| dot(self, other: 'Series | ArrayLike') -> 'int | float | None'
| Compute the dot/inner product between two Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("b", [4.0, 5.0, 6.0])
| >>> s.dot(s2)
| 32.0
|
| Parameters
| ----------
| other
| Series (or array) to compute dot product with.
|
| drop_nans(self) -> 'Series'
| Drop all floating point NaN values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nulls
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To drop null values, use :func:`drop_nulls`.
|
| Examples
| --------
| >>> s = pl.Series([1.0, None, 3.0, float("nan")])
| >>> s.drop_nans()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| null
| 3.0
| ]
|
| drop_nulls(self) -> 'Series'
| Drop all null values.
|
| The original order of the remaining elements is preserved.
|
| See Also
| --------
| drop_nans
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To drop NaN values, use :func:`drop_nans`.
|
| Examples
| --------
| >>> s = pl.Series([1.0, None, 3.0, float("nan")])
| >>> s.drop_nulls()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 3.0
| NaN
| ]
|
| entropy(self, base: 'float' = 2.718281828459045, *, normalize: 'bool' = True) -> 'float | None'
| Computes the entropy.
|
| Uses the formula `-sum(pk * log(pk))` where `pk` are discrete probabilities.
|
| Parameters
| ----------
| base
| Given base, defaults to `e`
| normalize
| Normalize pk if it doesn't sum to 1.
|
| Examples
| --------
| >>> a = pl.Series([0.99, 0.005, 0.005])
| >>> a.entropy(normalize=True)
| 0.06293300616044681
| >>> b = pl.Series([0.65, 0.10, 0.25])
| >>> b.entropy(normalize=True)
| 0.8568409950394724
|
| eq(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series == other`.
|
| eq_missing(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of equality operator `series == other` where `None == None`.
|
| This differs from the standard `eq` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| See Also
| --------
| ne_missing
| eq
|
| Examples
| --------
| >>> s1 = pl.Series("a", [333, 200, None])
| >>> s2 = pl.Series("a", [100, 200, None])
| >>> s1.eq(s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| true
| null
| ]
| >>> s1.eq_missing(s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| true
| true
| ]
|
| equals(self, other: 'Series', *, check_dtypes: 'bool' = False, check_names: 'bool' = False, null_equal: 'bool' = True) -> 'bool'
| Check whether the Series is equal to another Series.
|
| .. versionchanged:: 0.20.31
| The `strict` parameter was renamed `check_dtypes`.
|
| Parameters
| ----------
| other
| Series to compare with.
| check_dtypes
| Require data types to match.
| check_names
| Require names to match.
| null_equal
| Consider null values as equal.
|
| See Also
| --------
| polars.testing.assert_series_equal
|
| Examples
| --------
| >>> s1 = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("b", [4, 5, 6])
| >>> s1.equals(s1)
| True
| >>> s1.equals(s2)
| False
|
| estimated_size(self, unit: 'SizeUnit' = 'b') -> 'int | float'
| Return an estimation of the total (heap) allocated size of the Series.
|
| Estimated size is given in the specified unit (bytes by default).
|
| This estimation is the sum of the size of its buffers, validity, including
| nested arrays. Multiple arrays may share buffers and bitmaps. Therefore, the
| size of 2 arrays is not the sum of the sizes computed from this function. In
| particular, [`StructArray`]'s size is an upper bound.
|
| When an array is sliced, its allocated size remains constant because the buffer
| unchanged. However, this function will yield a smaller number. This is because
| this function returns the visible size of the buffer, not its total capacity.
|
| FFI buffers are included in this estimation.
|
| Notes
| -----
| For data with Object dtype, the estimated size only reports the pointer
| size, which is a huge underestimation.
|
| Parameters
| ----------
| unit : {'b', 'kb', 'mb', 'gb', 'tb'}
| Scale the returned size to the given unit.
|
| Examples
| --------
| >>> s = pl.Series("values", list(range(1_000_000)), dtype=pl.UInt32)
| >>> s.estimated_size()
| 4000000
| >>> s.estimated_size("mb")
| 3.814697265625
|
| ewm_mean(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Series'
| Compute exponentially-weighted moving average.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.ewm_mean(com=1, ignore_nulls=False)
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 1.666667
| 2.428571
| ]
|
| ewm_mean_by(self, by: 'IntoExpr', *, half_life: 'str_ | timedelta') -> 'Series'
| Compute time-based exponentially weighted moving average.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMA is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \alpha_i &= 1 - \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= \alpha_i x_i + (1 - \alpha_i) y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Times to calculate average by. Should be ``DateTime``, ``Date``, ``UInt64``,
| ``UInt32``, ``Int64``, or ``Int32`` data type.
| half_life
| Unit over which observation decays to half its value.
|
| Can be created either from a timedelta, or
| by using the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 day)
| - 1w (1 week)
| - 1i (1 index count)
|
| Or combine them:
| "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
|
| Note that `half_life` is treated as a constant duration - calendar
| durations such as months (or even days in the time-zone-aware case)
| are not supported, please express your duration in an approximately
| equivalent number of hours (e.g. '370h' instead of '1mo').
|
| Returns
| -------
| Expr
| :class:`Float16` if input is `Float16`, :class:`.Float32` if input is
| `Float32`, otherwise :class:`.Float64`.
|
| Examples
| --------
| >>> from datetime import date, timedelta
| >>> df = pl.DataFrame(
| ... {
| ... "values": [0, 1, 2, None, 4],
| ... "times": [
| ... date(2020, 1, 1),
| ... date(2020, 1, 3),
| ... date(2020, 1, 10),
| ... date(2020, 1, 15),
| ... date(2020, 1, 17),
| ... ],
| ... }
| ... ).sort("times")
| >>> df["values"].ewm_mean_by(df["times"], half_life="4d")
| shape: (5,)
| Series: 'values' [f64]
| [
| 0.0
| 0.292893
| 1.492474
| null
| 3.254508
| ]
|
| ewm_std(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Series'
| Compute exponentially-weighted moving standard deviation.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.ewm_std(com=1, ignore_nulls=False)
| shape: (3,)
| Series: 'a' [f64]
| [
| null
| 0.707107
| 0.963624
| ]
|
| ewm_sum(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Series'
| Compute exponentially-weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\tau`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \tau } \right\} \;
| \forall \; \tau > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.ewm_sum(alpha=0.5)
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 2.5
| 4.25
| ]
|
| ewm_sum_by(self, by: 'IntoExpr', *, half_life: 'str_ | timedelta') -> 'Series'
| Compute time-based exponentially weighted moving sum.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given observations :math:`x_0, x_1, \ldots, x_{n-1}` at times
| :math:`t_0, t_1, \ldots, t_{n-1}`, the EWMS is calculated as
|
| .. math::
|
| y_0 &= x_0
|
| \lambda_i &= \exp \left\{ \frac{ -\ln(2)(t_i-t_{i-1}) }
| { \tau } \right\}
|
| y_i &= x_i + \lambda_i y_{i-1}; \quad i > 0
|
| where :math:`\tau` is the `half_life`.
|
| Parameters
| ----------
| by
| Times to calculate the sum by. Should be ``DateTime``, ``Date``, ``UInt64``,
| ``UInt32``, ``Int64``, or ``Int32`` data type.
| half_life
| Unit over which observation decays to half its value.
|
| Examples
| --------
| >>> df = pl.DataFrame(
| ... {
| ... "values": [1, 2, 3, 4, 5],
| ... "times": [0, 1, 2, 5, 6],
| ... }
| ... )
| >>> df["values"].ewm_sum_by(df["times"], half_life="1i")
| shape: (5,)
| Series: 'values' [f64]
| [
| 1.0
| 2.5
| 4.25
| 4.53125
| 7.265625
| ]
|
| ewm_var(self, *, com: 'float | None' = None, span: 'float | None' = None, half_life: 'float | None' = None, alpha: 'float | None' = None, adjust: 'bool' = True, bias: 'bool' = False, min_samples: 'int' = 1, ignore_nulls: 'bool' = False) -> 'Series'
| Compute exponentially-weighted moving variance.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| com
| Specify decay in terms of center of mass, :math:`\gamma`, with
|
| .. math::
| \alpha = \frac{1}{1 + \gamma} \; \forall \; \gamma \geq 0
| span
| Specify decay in terms of span, :math:`\theta`, with
|
| .. math::
| \alpha = \frac{2}{\theta + 1} \; \forall \; \theta \geq 1
| half_life
| Specify decay in terms of half-life, :math:`\lambda`, with
|
| .. math::
| \alpha = 1 - \exp \left\{ \frac{ -\ln(2) }{ \lambda } \right\} \;
| \forall \; \lambda > 0
| alpha
| Specify smoothing factor alpha directly, :math:`0 < \alpha \leq 1`.
| adjust
| Divide by decaying adjustment factor in beginning periods to account for
| imbalance in relative weightings
|
| - When `adjust=True` (the default) the EW function is calculated
| using weights :math:`w_i = (1 - \alpha)^i`
| - When `adjust=False` the EW function is calculated
| recursively by
|
| .. math::
| y_0 &= x_0 \\
| y_t &= (1 - \alpha)y_{t - 1} + \alpha x_t
| bias
| When `bias=False`, apply a correction to make the estimate statistically
| unbiased.
| min_samples
| Minimum number of observations in window required to have a value
| (otherwise result is null).
| ignore_nulls
| Ignore missing values when calculating weights.
|
| - When `ignore_nulls=False` (default), weights are based on absolute
| positions.
| For example, the weights of :math:`x_0` and :math:`x_2` used in
| calculating the final weighted average of
| [:math:`x_0`, None, :math:`x_2`] are
| :math:`(1-\alpha)^2` and :math:`1` if `adjust=True`, and
| :math:`(1-\alpha)^2` and :math:`\alpha` if `adjust=False`.
|
| - When `ignore_nulls=True`, weights are based
| on relative positions. For example, the weights of
| :math:`x_0` and :math:`x_2` used in calculating the final weighted
| average of [:math:`x_0`, None, :math:`x_2`] are
| :math:`1-\alpha` and :math:`1` if `adjust=True`,
| and :math:`1-\alpha` and :math:`\alpha` if `adjust=False`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.ewm_var(com=1, ignore_nulls=False)
| shape: (3,)
| Series: 'a' [f64]
| [
| null
| 0.5
| 0.928571
| ]
|
| exp(self) -> 'Series'
| Compute the exponential, element-wise.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.exp()
| shape: (3,)
| Series: '' [f64]
| [
| 2.718282
| 7.389056
| 20.085537
| ]
|
| explode(self, *, empty_as_null: 'bool | None' = True, keep_nulls: 'bool' = True) -> 'Series'
| Explode a list Series.
|
| This means that every item is expanded to a new row.
|
| Parameters
| ----------
| empty_as_null
| Explode an empty list into a `null`.
| keep_nulls
| Explode a `null` list into a `null`.
|
| Returns
| -------
| Series
| Series with the data type of the list elements.
|
| See Also
| --------
| Series.list.explode : Explode a list column.
|
| Examples
| --------
| >>> s = pl.Series("a", [[1, 2, 3], [4, 5, 6]])
| >>> s
| shape: (2,)
| Series: 'a' [list[i64]]
| [
| [1, 2, 3]
| [4, 5, 6]
| ]
| >>> s.explode(empty_as_null=False)
| shape: (6,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| ]
|
| extend(self, other: 'Series') -> 'Self'
| Extend the memory backed by this Series with the values from another.
|
| Different from `append`, which adds the chunks from `other` to the chunks of
| this series, `extend` appends the data from `other` to the underlying memory
| locations and thus may cause a reallocation (which is expensive).
|
| If this does `not` cause a reallocation, the resulting data structure will not
| have any extra chunks and thus will yield faster queries.
|
| Prefer `extend` over `append` when you want to do a query after a single
| append. For instance, during online operations where you add `n` rows
| and rerun a query.
|
| Prefer `append` over `extend` when you want to append many times
| before doing a query. For instance, when you read in multiple files and want
| to store them in a single `Series`. In the latter case, finish the sequence
| of `append` operations with a `rechunk`.
|
| Parameters
| ----------
| other
| Series to extend the series with.
|
| Warnings
| --------
| This method modifies the series in-place. The series is returned for
| convenience only.
|
| See Also
| --------
| append
|
| Examples
| --------
| >>> a = pl.Series("a", [1, 2, 3])
| >>> b = pl.Series("b", [4, 5])
| >>> a.extend(b)
| shape: (5,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| ]
|
| The resulting series will consist of a single chunk.
|
| >>> a.n_chunks()
| 1
|
| extend_constant(self, value: 'IntoExpr', n: 'int | IntoExprColumn') -> 'Series'
| Extremely fast method for extending the Series with 'n' copies of a value.
|
| Parameters
| ----------
| value
| A constant literal value or a unit expression with which to extend the
| expression result Series; can pass None to extend with nulls.
| n
| The number of additional values that will be added.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.extend_constant(99, n=2)
| shape: (5,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| 99
| 99
| ]
|
| fill_nan(self, value: 'int | float | Expr | None') -> 'Series'
| Fill floating point NaN value with a fill value.
|
| Parameters
| ----------
| value
| Value used to fill NaN values.
|
| See Also
| --------
| fill_null
|
| Notes
| -----
| A NaN value is not the same as a null value.
| To fill null values, use :func:`fill_null`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, float("nan")])
| >>> s.fill_nan(0)
| shape: (4,)
| Series: 'a' [f64]
| [
| 1.0
| 2.0
| 3.0
| 0.0
| ]
|
| fill_null(self, value: 'Any | Expr | None' = None, strategy: 'FillNullStrategy | None' = None, limit: 'int | None' = None) -> 'Series'
| Fill null values using the specified value or strategy.
|
| Parameters
| ----------
| value
| Value used to fill null values.
| strategy : {None, 'forward', 'backward', 'min', 'max', 'mean', 'zero', 'one'}
| Strategy used to fill null values.
| limit
| Number of consecutive null values to fill when using the 'forward' or
| 'backward' strategy.
|
| See Also
| --------
| backward_fill
| fill_nan
| forward_fill
|
| Notes
| -----
| A null value is not the same as a NaN value.
| To fill NaN values, use :func:`fill_nan`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, None])
| >>> s.fill_null(strategy="forward")
| shape: (4,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 3
| ]
| >>> s.fill_null(strategy="min")
| shape: (4,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 1
| ]
| >>> s = pl.Series("b", ["x", None, "z"])
| >>> s.fill_null(pl.lit(""))
| shape: (3,)
| Series: 'b' [str]
| [
| "x"
| ""
| "z"
| ]
|
| filter(self, predicate: 'Series | Iterable[bool]') -> 'Self'
| Filter elements by a boolean mask.
|
| The original order of the remaining elements is preserved.
|
| Elements where the filter does not evaluate to True are discarded, including
| nulls.
|
| Parameters
| ----------
| predicate
| Boolean mask.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> mask = pl.Series("", [True, False, True])
| >>> s.filter(mask)
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 3
| ]
|
| first(self, *, ignore_nulls: 'bool' = False) -> 'PythonLiteral | None'
| Get the first element of the Series.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the first non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Returns `None` if the Series is empty.
|
| floor(self) -> 'Series'
| Rounds down to the nearest integer value.
|
| Only works on floating point Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.12345, 2.56789, 3.901234])
| >>> s.floor()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.0
| 2.0
| 3.0
| ]
|
| forward_fill(self, limit: 'int | None' = None) -> 'Series'
| Fill missing values with the last non-null value.
|
| This is an alias of `.fill_null(strategy="forward")`.
|
| Parameters
| ----------
| limit
| The number of consecutive null values to forward fill.
|
| See Also
| --------
| backward_fill
| fill_null
| shift
|
| gather(self, indices: 'int | list_[int] | Expr | Series | np.ndarray[Any, Any]', *, null_on_oob: 'bool' = False) -> 'Series'
| Take values by index.
|
| Parameters
| ----------
| indices
| Index location used for selection.
| null_on_oob
| Behavior if an index is out of bounds:
|
| - True -> set the result to null
| - False -> raise an error
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4])
| >>> s.gather([1, 3])
| shape: (2,)
| Series: 'a' [i64]
| [
| 2
| 4
| ]
|
| Use `null_on_oob=True` to return null for out-of-bounds indices.
|
| >>> s.gather([1, 10], null_on_oob=True)
| shape: (2,)
| Series: 'a' [i64]
| [
| 2
| null
| ]
|
| gather_every(self, n: 'int', offset: 'int' = 0) -> 'Series'
| Take every nth value in the Series and return as new Series.
|
| Parameters
| ----------
| n
| Gather every *n*-th row.
| offset
| Start the row index at this offset.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4])
| >>> s.gather_every(2)
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 3
| ]
| >>> s.gather_every(2, offset=1)
| shape: (2,)
| Series: 'a' [i64]
| [
| 2
| 4
| ]
|
| ge(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series >= other`.
|
| get_chunks(self) -> 'list_[Series]'
| Get the chunks of this Series as a list of Series.
|
| Examples
| --------
| >>> s1 = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("a", [4, 5, 6])
| >>> s = pl.concat([s1, s2], rechunk=False)
| >>> s.get_chunks()
| [shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ], shape: (3,)
| Series: 'a' [i64]
| [
| 4
| 5
| 6
| ]]
|
| gt(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series > other`.
|
| has_nulls(self) -> 'bool'
| Check whether the Series contains one or more null values.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, None])
| >>> s.has_nulls()
| True
| >>> s[:2].has_nulls()
| False
|
| has_validity(self) -> 'bool'
| Check whether the Series contains one or more null values.
|
| .. deprecated:: 0.20.30
| Use the :meth:`has_nulls` method instead.
|
| hash(self, seed: 'int' = 0, seed_1: 'int | None' = None, seed_2: 'int | None' = None, seed_3: 'int | None' = None) -> 'Series'
| Hash the Series.
|
| The hash value is of type `UInt64`.
|
| Parameters
| ----------
| seed
| Random seed parameter. Defaults to 0.
| seed_1
| Random seed parameter. Defaults to `seed` if not set.
| seed_2
| Random seed parameter. Defaults to `seed` if not set.
| seed_3
| Random seed parameter. Defaults to `seed` if not set.
|
| Notes
| -----
| This implementation of `hash` does not guarantee stable results
| across different Polars versions. Its stability is only guaranteed within a
| single version.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.hash(seed=42) # doctest: +IGNORE_RESULT
| shape: (3,)
| Series: 'a' [u64]
| [
| 10734580197236529959
| 3022416320763508302
| 13756996518000038261
| ]
|
| head(self, n: 'int' = 10) -> 'Series'
| Get the first `n` elements.
|
| Parameters
| ----------
| n
| Number of elements to return. If a negative value is passed, return all
| elements except the last `abs(n)`.
|
| See Also
| --------
| tail, slice
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.head(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| Pass a negative value to get all rows `except` the last `abs(n)`.
|
| >>> s.head(-3)
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 2
| ]
|
| hist(self, bins: 'list_[float] | None' = None, *, bin_count: 'int | None' = None, include_category: 'bool' = True, include_breakpoint: 'bool' = True) -> 'DataFrame'
| Bin values into buckets and count their occurrences.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| bins
| Bin edges. If None given, we determine the edges based on the data.
| bin_count
| If `bins` is not provided, `bin_count` uniform bins are created that fully
| encompass the data.
| include_breakpoint
| Include a column that indicates the upper breakpoint.
| include_category
| Include a column that shows the intervals as categories.
|
| Returns
| -------
| DataFrame
|
| Examples
| --------
| >>> a = pl.Series("a", [1, 3, 8, 8, 2, 1, 3])
| >>> a.hist(bin_count=4)
| shape: (4, 3)
| ┌────────────┬─────────────┬───────┐
| │ breakpoint ┆ category ┆ count │
| │ --- ┆ --- ┆ --- │
| │ f64 ┆ cat ┆ u32 │
| ╞════════════╪═════════════╪═══════╡
| │ 2.75 ┆ [1.0, 2.75] ┆ 3 │
| │ 4.5 ┆ (2.75, 4.5] ┆ 2 │
| │ 6.25 ┆ (4.5, 6.25] ┆ 0 │
| │ 8.0 ┆ (6.25, 8.0] ┆ 2 │
| └────────────┴─────────────┴───────┘
|
| implode(self) -> 'Self'
| Aggregate values into a list.
|
| The returned list itself is a scalar value of `list` dtype.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.implode()
| shape: (1,)
| Series: 'a' [list[i64]]
| [
| [1, 2, 3]
| ]
|
| index_of(self, element: 'IntoExpr') -> 'int | None'
| Get the index of the first occurrence of a value, or ``None`` if it's not found.
|
| Parameters
| ----------
| element
| Value to find.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, None, 17])
| >>> s.index_of(17)
| 2
| >>> s.index_of(None) # search for a null
| 1
| >>> s.index_of(55) is None
| True
|
| interpolate(self, method: 'InterpolationMethod' = 'linear') -> 'Series'
| Interpolate intermediate values.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| method : {'linear', 'nearest'}
| Interpolation method.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, None, None, 5])
| >>> s.interpolate()
| shape: (5,)
| Series: 'a' [f64]
| [
| 1.0
| 2.0
| 3.0
| 4.0
| 5.0
| ]
|
| interpolate_by(self, by: 'IntoExpr') -> 'Series'
| Interpolate intermediate values with x-coordinate based on another column.
|
| Nulls at the beginning and end of the series remain null.
|
| Parameters
| ----------
| by
| Column to interpolate values based on.
|
| Examples
| --------
| Fill null values using linear interpolation.
|
| >>> s = pl.Series([1, None, None, 3])
| >>> by = pl.Series([1, 2, 7, 8])
| >>> s.interpolate_by(by)
| shape: (4,)
| Series: '' [f64]
| [
| 1.0
| 1.285714
| 2.714286
| 3.0
| ]
|
| is_between(self, lower_bound: 'IntoExpr', upper_bound: 'IntoExpr', closed: 'ClosedInterval' = 'both') -> 'Series'
| Get a boolean mask of the values that are between the given lower/upper bounds.
|
| Parameters
| ----------
| lower_bound
| Lower bound value. Accepts expression input. Non-expression inputs
| (including strings) are parsed as literals.
| upper_bound
| Upper bound value. Accepts expression input. Non-expression inputs
| (including strings) are parsed as literals.
| closed : {'both', 'left', 'right', 'none'}
| Define which sides of the interval are closed (inclusive).
|
| Notes
| -----
| If the value of the `lower_bound` is greater than that of the `upper_bound`
| then the result will be False, as no value can satisfy the condition.
|
| Examples
| --------
| >>> s = pl.Series("num", [1, 2, 3, 4, 5])
| >>> s.is_between(2, 4)
| shape: (5,)
| Series: 'num' [bool]
| [
| false
| true
| true
| true
| false
| ]
|
| Use the `closed` argument to include or exclude the values at the bounds:
|
| >>> s.is_between(2, 4, closed="left")
| shape: (5,)
| Series: 'num' [bool]
| [
| false
| true
| true
| false
| false
| ]
|
| You can also use strings as well as numeric/temporal values:
|
| >>> s = pl.Series("s", ["a", "b", "c", "d", "e"])
| >>> s.is_between("b", "d", closed="both")
| shape: (5,)
| Series: 's' [bool]
| [
| false
| true
| true
| true
| false
| ]
|
| is_close(self, other: 'IntoExpr', *, abs_tol: 'float' = 0.0, rel_tol: 'float' = 1e-09, nans_equal: 'bool' = False) -> 'Series'
| Get a boolean mask of the values being close to the other values.
|
| Two values `a` and `b` are considered close if the following condition holds:
|
| .. math::
| |a-b| \le max \{ \text{rel_tol} \cdot max \{ |a|, |b| \}, \text{abs_tol} \}
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
| abs_tol
| Absolute tolerance. This is the maximum allowed absolute difference between
| two values. Must be non-negative.
| rel_tol
| Relative tolerance. This is the maximum allowed difference between two
| values, relative to the larger absolute value. Must be non-negative.
| nans_equal
| Whether NaN values should be considered equal.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Notes
| -----
| The implementation of this method is symmetric and mirrors the behavior of
| :meth:`math.isclose`. Specifically note that this behavior is different to
| :meth:`numpy.isclose`.
|
| Examples
| --------
| >>> s = pl.Series("s", [1.0, 1.2, 1.4, 1.45, 1.6])
| >>> s.is_close(1.4, abs_tol=0.1)
| shape: (5,)
| Series: 's' [bool]
| [
| false
| false
| true
| true
| false
| ]
|
| is_duplicated(self) -> 'Series'
| Get mask of all duplicated values.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.is_duplicated()
| shape: (4,)
| Series: 'a' [bool]
| [
| false
| true
| true
| false
| ]
|
| is_empty(self, *, ignore_nulls: 'bool' = False) -> 'bool'
| Check if the Series is empty.
|
| Parameters
| ----------
| ignore_nulls
| If true a series containing only nulls will also be considered empty.
| The default is false.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Examples
| --------
| >>> s = pl.Series("a", [], dtype=pl.Float32)
| >>> s.is_empty()
| True
| >>> s = pl.Series("a", [None], dtype=pl.Float32)
| >>> s.is_empty()
| False
| >>> s.is_empty(ignore_nulls=True)
| True
|
| is_finite(self) -> 'Series'
| Returns a boolean Series indicating which values are finite.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> import numpy as np
| >>> s = pl.Series("a", [1.0, 2.0, np.inf])
| >>> s.is_finite()
| shape: (3,)
| Series: 'a' [bool]
| [
| true
| true
| false
| ]
|
| is_first_distinct(self) -> 'Series'
| Return a boolean mask indicating the first occurrence of each distinct value.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series([1, 1, 2, 3, 2])
| >>> s.is_first_distinct()
| shape: (5,)
| Series: '' [bool]
| [
| true
| false
| true
| true
| false
| ]
|
| is_in(self, other: 'Series | Collection[Any]', *, nulls_equal: 'bool' = False) -> 'Series'
| Check if elements of this Series are in the other Series.
|
| Parameters
| ----------
| other
| A Series or collection to search in.
| nulls_equal : bool, default False
| If True, treat null as a distinct value. Null values will not propagate.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s2 = pl.Series("b", [2, 4, None])
| >>> s2.is_in(s)
| shape: (3,)
| Series: 'b' [bool]
| [
| true
| false
| null
| ]
| >>> # when nulls_equal=True, None is treated as a distinct value
| >>> s2.is_in(s, nulls_equal=True)
| shape: (3,)
| Series: 'b' [bool]
| [
| true
| false
| false
| ]
|
| >>> # check if some values are a member of sublists
| >>> sets = pl.Series("sets", [[1, 2, 3], [1, 2], [9, 10]])
| >>> optional_members = pl.Series("optional_members", [1, 2, 3])
| >>> print(sets)
| shape: (3,)
| Series: 'sets' [list[i64]]
| [
| [1, 2, 3]
| [1, 2]
| [9, 10]
| ]
| >>> print(optional_members)
| shape: (3,)
| Series: 'optional_members' [i64]
| [
| 1
| 2
| 3
| ]
| >>> optional_members.is_in(sets)
| shape: (3,)
| Series: 'optional_members' [bool]
| [
| true
| true
| false
| ]
|
| is_infinite(self) -> 'Series'
| Returns a boolean Series indicating which values are infinite.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> import numpy as np
| >>> s = pl.Series("a", [1.0, 2.0, np.inf])
| >>> s.is_infinite()
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| false
| true
| ]
|
| is_last_distinct(self) -> 'Series'
| Return a boolean mask indicating the last occurrence of each distinct value.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series([1, 1, 2, 3, 2])
| >>> s.is_last_distinct()
| shape: (5,)
| Series: '' [bool]
| [
| false
| true
| false
| true
| true
| ]
|
| is_nan(self) -> 'Series'
| Returns a boolean Series indicating which values are NaN.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> import numpy as np
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, np.nan])
| >>> s.is_nan()
| shape: (4,)
| Series: 'a' [bool]
| [
| false
| false
| false
| true
| ]
|
| is_not_nan(self) -> 'Series'
| Returns a boolean Series indicating which values are not NaN.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> import numpy as np
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, np.nan])
| >>> s.is_not_nan()
| shape: (4,)
| Series: 'a' [bool]
| [
| true
| true
| true
| false
| ]
|
| is_not_null(self) -> 'Series'
| Returns a boolean Series indicating which values are not null.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, None])
| >>> s.is_not_null()
| shape: (4,)
| Series: 'a' [bool]
| [
| true
| true
| true
| false
| ]
|
| is_null(self) -> 'Series'
| Returns a boolean Series indicating which values are null.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, None])
| >>> s.is_null()
| shape: (4,)
| Series: 'a' [bool]
| [
| false
| false
| false
| true
| ]
|
| is_sorted(self, *, descending: 'bool' = False, nulls_last: 'bool' = False) -> 'bool'
| Check if the Series is sorted.
|
| Parameters
| ----------
| descending
| Check if the Series is sorted in descending order
| nulls_last
| Set nulls at the end of the Series in sorted check.
|
| Examples
| --------
| >>> s = pl.Series([1, 3, 2])
| >>> s.is_sorted()
| False
|
| >>> s = pl.Series([3, 2, 1])
| >>> s.is_sorted(descending=True)
| True
|
| is_unique(self) -> 'Series'
| Get mask of all unique values.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.is_unique()
| shape: (4,)
| Series: 'a' [bool]
| [
| true
| false
| false
| true
| ]
|
| item(self, index: 'int | None' = None) -> 'Any'
| Return the Series as a scalar, or return the element at the given index.
|
| If no index is provided, this is equivalent to `s[0]`, with a check
| that the shape is (1,). With an index, this is equivalent to `s[index]`.
|
| Examples
| --------
| >>> s1 = pl.Series("a", [1])
| >>> s1.item()
| 1
| >>> s2 = pl.Series("a", [9, 8, 7])
| >>> s2.cum_sum().item(-1)
| 24
|
| kurtosis(self, *, fisher: 'bool' = True, bias: 'bool' = True) -> 'float | None'
| Compute the kurtosis (Fisher or Pearson) of a dataset.
|
| Kurtosis is the fourth central moment divided by the square of the
| variance. If Fisher's definition is used, then 3.0 is subtracted from
| the result to give 0.0 for a normal distribution.
| If bias is False then the kurtosis is calculated using k statistics to
| eliminate bias coming from biased moment estimators
|
| See scipy.stats for more information
|
| Parameters
| ----------
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Examples
| --------
| >>> s = pl.Series("grades", [66, 79, 54, 97, 96, 70, 69, 85, 93, 75])
| >>> s.kurtosis()
| -1.0522623626787952
| >>> s.kurtosis(fisher=False)
| 1.9477376373212048
| >>> s.kurtosis(fisher=False, bias=False)
| 2.1040361802642717
|
| last(self, *, ignore_nulls: 'bool' = False) -> 'PythonLiteral | None'
| Get the last element of the Series.
|
| Parameters
| ----------
| ignore_nulls
| Ignore null values (default `False`).
| If set to `True`, the last non-null value is returned, otherwise `None` is
| returned if no non-null value exists.
|
| Returns `None` if the Series is empty.
|
| le(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series <= other`.
|
| len(self) -> 'int'
| Return the number of elements in the Series.
|
| Null values count towards the total.
|
| See Also
| --------
| count
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, None])
| >>> s.len()
| 3
|
| limit(self, n: 'int' = 10) -> 'Series'
| Get the first `n` elements.
|
| Alias for :func:`Series.head`.
|
| Parameters
| ----------
| n
| Number of elements to return. If a negative value is passed, return all
| elements except the last `abs(n)`.
|
| See Also
| --------
| head
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.limit(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| Pass a negative value to get all rows `except` the last `abs(n)`.
|
| >>> s.limit(-3)
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 2
| ]
|
| log(self, base: 'float | Series' = 2.718281828459045) -> 'Series'
| Compute the logarithm to a given base.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.log()
| shape: (3,)
| Series: '' [f64]
| [
| 0.0
| 0.693147
| 1.098612
| ]
|
| log10(self) -> 'Series'
| Compute the base 10 logarithm of the input array, element-wise.
|
| Examples
| --------
| >>> s = pl.Series([10, 100, 1000])
| >>> s.log10()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 2.0
| 3.0
| ]
|
| log1p(self) -> 'Series'
| Compute the natural logarithm of the input array plus one, element-wise.
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.log1p()
| shape: (3,)
| Series: '' [f64]
| [
| 0.693147
| 1.098612
| 1.386294
| ]
|
| lower_bound(self) -> 'Self'
| Return the lower bound of this Series' dtype as a unit Series.
|
| See Also
| --------
| upper_bound : return the upper bound of the given Series' dtype.
|
| Examples
| --------
| >>> s = pl.Series("s", [-1, 0, 1], dtype=pl.Int32)
| >>> s.lower_bound()
| shape: (1,)
| Series: 's' [i32]
| [
| -2147483648
| ]
|
| >>> s = pl.Series("s", [1.0, 2.5, 3.0], dtype=pl.Float32)
| >>> s.lower_bound()
| shape: (1,)
| Series: 's' [f32]
| [
| -inf
| ]
|
| lt(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series < other`.
|
| map_elements(self, function: 'Callable[[Any], Any]', return_dtype: 'PolarsDataType | None' = None, *, skip_nulls: 'bool' = True, _disable_inefficient_map_warning: 'bool' = False) -> 'Self'
| Map a custom/user-defined function (UDF) over elements in this Series.
|
| .. warning::
| This method is much slower than the native expressions API.
| Only use it if you cannot implement your logic otherwise.
|
| Suppose that the function is: `x ↦ sqrt(x)`:
|
| - For mapping elements of a series, consider: `s.sqrt()`.
| - For mapping inner elements of lists, consider:
| `s.list.eval(pl.element().sqrt())`.
| - For mapping elements of struct fields, consider:
| `s.struct.field("field_name").sqrt()`.
|
| If the function returns a different datatype, the return_dtype arg should
| be set, otherwise the method will fail.
|
| Implementing logic using a Python function is almost always *significantly*
| slower and more memory intensive than implementing the same logic using
| the native expression API because:
|
| - The native expression engine runs in Rust; UDFs run in Python.
| - Use of Python UDFs forces the DataFrame to be materialized in memory.
| - Polars-native expressions can be parallelised (UDFs typically cannot).
| - Polars-native expressions can be logically optimised (UDFs cannot).
|
| Wherever possible you should strongly prefer the native expression API
| to achieve the best performance.
|
| Parameters
| ----------
| function
| Custom function or lambda.
| return_dtype
| Output datatype.
| If not set, the dtype will be inferred based on the first non-null value
| that is returned by the function.
| skip_nulls
| Nulls will be skipped and not passed to the python function.
| This is faster because python can be skipped and because we call
| more specialized functions.
|
| Warnings
| --------
| If `return_dtype` is not provided, this may lead to unexpected results.
| We allow this, but it is considered a bug in the user's query.
|
| Notes
| -----
| * If your function is expensive and you don't want it to be called more than
| once for a given input, consider applying an `@lru_cache` decorator to it.
| If your data is suitable you may achieve *significant* speedups.
|
| * A UDF passed to `map_elements` must be pure, meaning that it cannot modify
| or depend on state other than its arguments.
|
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.map_elements(lambda x: x + 10, return_dtype=pl.Int64) # doctest: +SKIP
| shape: (3,)
| Series: 'a' [i64]
| [
| 11
| 12
| 13
| ]
|
| Returns
| -------
| Series
|
| max(self) -> 'PythonLiteral | None'
| Get the maximum value in this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.max()
| 3
|
| max_by(self, by: 'IntoExpr') -> 'Expr'
| Get the maximum value in this Series, ordered by an expression.
|
| If the by expression has multiple values equal to the maximum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the largest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> s = pl.Series("a", [-2.0, float("nan"), 1.0])
| >>> s.max_by(pl.col.a.abs())
| -2.0
|
| mean(self) -> 'PythonLiteral | None'
| Reduce this Series to the mean value.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.mean()
| 2.0
|
| median(self) -> 'PythonLiteral | None'
| Get the median of this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.median()
| 2.0
|
| min(self) -> 'PythonLiteral | None'
| Get the minimal value in this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.min()
| 1
|
| min_by(self, by: 'IntoExpr') -> 'Expr'
| Get the minimum value in this Series, ordered by an expression.
|
| If the by expression has multiple values equal to the minimum it is not
| defined which value will be chosen.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| by
| Column used to determine the smallest element.
| Accepts expression input. Strings are parsed as column names.
|
| Examples
| --------
| >>> s = pl.Series("a", [-2.0, float("nan"), 1.0])
| >>> s.min_by(pl.col.a.abs())
| 1.0
|
| mode(self, *, maintain_order: 'bool' = False) -> 'Series'
| Compute the most occurring value(s).
|
| Can return multiple Values.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.mode()
| shape: (1,)
| Series: 'a' [i64]
| [
| 2
| ]
|
| n_chunks(self) -> 'int'
| Get the number of chunks that this Series contains.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.n_chunks()
| 1
| >>> s2 = pl.Series("a", [4, 5, 6])
|
| Concatenate Series with rechunk = True
|
| >>> pl.concat([s, s2], rechunk=True).n_chunks()
| 1
|
| Concatenate Series with rechunk = False
|
| >>> pl.concat([s, s2], rechunk=False).n_chunks()
| 2
|
| n_unique(self) -> 'int'
| Count the number of unique values in this Series.
|
| `null` is considered to be a unique value for the purposes of this operation.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, None])
| >>> s.n_unique()
| 3
|
| nan_max(self) -> 'int | float | date | datetime | timedelta | str_'
| Get maximum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 3, 4])
| >>> s.nan_max()
| 4
|
| >>> s = pl.Series("a", [1.0, float("nan"), 4.0])
| >>> s.nan_max()
| nan
|
| nan_min(self) -> 'int | float | date | datetime | timedelta | str_'
| Get minimum value, but propagate/poison encountered NaN values.
|
| This differs from numpy's `nanmax` as numpy defaults to propagating NaN values,
| whereas polars defaults to ignoring them.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 3, 4])
| >>> s.nan_min()
| 1
|
| >>> s = pl.Series("a", [1.0, float("nan"), 4.0])
| >>> s.nan_min()
| nan
|
| ne(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of operator expression `series != other`.
|
| ne_missing(self, other: 'Any') -> 'Series | Expr'
| Method equivalent of equality operator `series != other` where `None == None`.
|
| This differs from the standard `ne` where null values are propagated.
|
| Parameters
| ----------
| other
| A literal or expression value to compare with.
|
| See Also
| --------
| eq_missing
| ne
|
| Examples
| --------
| >>> s1 = pl.Series("a", [333, 200, None])
| >>> s2 = pl.Series("a", [100, 200, None])
| >>> s1.ne(s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| true
| false
| null
| ]
| >>> s1.ne_missing(s2)
| shape: (3,)
| Series: 'a' [bool]
| [
| true
| false
| false
| ]
|
| new_from_index(self, index: 'int', length: 'int') -> 'Self'
| Create a new Series filled with values from the given index.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.new_from_index(1, 3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 2
| 2
| 2
| ]
|
| not_(self) -> 'Series'
| Negate a boolean Series.
|
| Returns
| -------
| Series
| Series of data type :class:`Boolean`.
|
| Examples
| --------
| >>> s = pl.Series("a", [True, False, False])
| >>> s.not_()
| shape: (3,)
| Series: 'a' [bool]
| [
| false
| true
| true
| ]
|
| null_count(self) -> 'int'
| Count the null values in this Series.
|
| Examples
| --------
| >>> s = pl.Series([1, None, None])
| >>> s.null_count()
| 2
|
| pct_change(self, n: 'int | IntoExprColumn' = 1) -> 'Series'
| Computes percentage change between values.
|
| Percentage change (as fraction) between current element and most-recent
| non-null element at least `n` period(s) before the current element.
|
| Computes the change from the previous row by default.
|
| Parameters
| ----------
| n
| periods to shift for forming percent change.
|
| Notes
| -----
| Null values are preserved. If you're coming from pandas, this matches
| their ``fill_method=None`` behaviour.
|
| Examples
| --------
| >>> pl.Series(range(10)).pct_change()
| shape: (10,)
| Series: '' [f64]
| [
| null
| inf
| 1.0
| 0.5
| 0.333333
| 0.25
| 0.2
| 0.166667
| 0.142857
| 0.125
| ]
|
| >>> pl.Series([1, 2, 4, 8, 16, 32, 64, 128, 256, 512]).pct_change(2)
| shape: (10,)
| Series: '' [f64]
| [
| null
| null
| 3.0
| 3.0
| 3.0
| 3.0
| 3.0
| 3.0
| 3.0
| 3.0
| ]
|
| peak_max(self) -> 'Self'
| Get a boolean mask of the local maximum peaks.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.peak_max()
| shape: (5,)
| Series: 'a' [bool]
| [
| false
| false
| false
| false
| true
| ]
|
| peak_min(self) -> 'Self'
| Get a boolean mask of the local minimum peaks.
|
| Examples
| --------
| >>> s = pl.Series("a", [4, 1, 3, 2, 5])
| >>> s.peak_min()
| shape: (5,)
| Series: 'a' [bool]
| [
| false
| true
| false
| true
| false
| ]
|
| pow(self, exponent: 'int | float | Series') -> 'Series'
| Raise to the power of the given exponent.
|
| If the exponent is float, the result follows the dtype of exponent.
| Otherwise, it follows dtype of base.
|
| Parameters
| ----------
| exponent
| The exponent. Accepts Series input.
|
| Examples
| --------
| Raising integers to positive integers results in integers:
|
| >>> s = pl.Series("foo", [1, 2, 3, 4])
| >>> s.pow(3)
| shape: (4,)
| Series: 'foo' [i64]
| [
| 1
| 8
| 27
| 64
| ]
|
| In order to raise integers to negative integers, you can cast either the
| base or the exponent to float:
|
| >>> s.pow(-3.0)
| shape: (4,)
| Series: 'foo' [f64]
| [
| 1.0
| 0.125
| 0.037037
| 0.015625
| ]
|
| product(self) -> 'int | float'
| Reduce this Series to the product value.
|
| Notes
| -----
| If there are no non-null values, then the output is `1`.
| If you would prefer empty products to return `None`, you can
| use `s.product() if s.count() else None` instead
| of `s.product()`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.product()
| 6
|
| qcut(self, quantiles: 'Sequence[float] | int', *, labels: 'Sequence[str_] | None' = None, left_closed: 'bool' = False, allow_duplicates: 'bool' = False, include_breaks: 'bool' = False) -> 'Series'
| Bin continuous values into discrete categories based on their quantiles.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Parameters
| ----------
| quantiles
| Either a list of quantile probabilities between 0 and 1 or a positive
| integer determining the number of bins with uniform probability.
| labels
| Names of the categories. The number of labels must be equal to the number
| of cut points plus one.
| left_closed
| Set the intervals to be left-closed instead of right-closed.
| allow_duplicates
| If set to `True`, duplicates in the resulting quantiles are dropped,
| rather than raising a `DuplicateError`. This can happen even with unique
| probabilities, depending on the data.
| include_breaks
| Include a column with the right endpoint of the bin each observation falls
| in. This will change the data type of the output from a
| :class:`Categorical` to a :class:`Struct`.
|
| Returns
| -------
| Series
| Series of data type :class:`Categorical` if `include_breaks` is set to
| `False` (default), otherwise a Series of data type :class:`Struct`.
|
| See Also
| --------
| cut
|
| Examples
| --------
| Divide a column into three categories according to pre-defined quantile
| probabilities.
|
| >>> s = pl.Series("foo", [-2, -1, 0, 1, 2])
| >>> s.qcut([0.25, 0.75], labels=["a", "b", "c"])
| shape: (5,)
| Series: 'foo' [cat]
| [
| "a"
| "a"
| "b"
| "b"
| "c"
| ]
|
| Divide a column into two categories using uniform quantile probabilities.
|
| >>> s.qcut(2, labels=["low", "high"], left_closed=True)
| shape: (5,)
| Series: 'foo' [cat]
| [
| "low"
| "low"
| "high"
| "high"
| "high"
| ]
|
| Create a DataFrame with the breakpoint and category for each value.
|
| >>> cut = s.qcut([0.25, 0.75], include_breaks=True).alias("cut")
| >>> s.to_frame().with_columns(cut).unnest("cut")
| shape: (5, 3)
| ┌─────┬────────────┬────────────┐
| │ foo ┆ breakpoint ┆ category │
| │ --- ┆ --- ┆ --- │
| │ i64 ┆ f64 ┆ cat │
| ╞═════╪════════════╪════════════╡
| │ -2 ┆ -1.0 ┆ (-inf, -1] │
| │ -1 ┆ -1.0 ┆ (-inf, -1] │
| │ 0 ┆ 1.0 ┆ (-1, 1] │
| │ 1 ┆ 1.0 ┆ (-1, 1] │
| │ 2 ┆ inf ┆ (1, inf] │
| └─────┴────────────┴────────────┘
|
| quantile(self, quantile: 'float | list_[float]', interpolation: 'QuantileMethod' = 'nearest') -> 'float | None | list_[float] | list_[None]'
| Get the quantile value of this Series.
|
| Parameters
| ----------
| quantile
| Quantile(s) between 0.0 and 1.0. Can be a single float or a list of floats.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
|
| Returns
| -------
| float | None | list[float] | list[None]
| A single quantile value if a float is provided, or a list of quantile values if a list is provided.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.quantile(0.5)
| 2.0
|
| Return a list of quantiles:
|
| >>> s.quantile([0.25, 0.75], interpolation="linear")
| [1.5, 2.5]
|
| radians(self) -> 'Series'
| Convert from degrees to radians.
|
| Examples
| --------
| >>> s = pl.Series("a", [-720, -540, -360, -180, 0, 180, 360, 540, 720])
| >>> s.radians()
| shape: (9,)
| Series: 'a' [f64]
| [
| -12.566371
| -9.424778
| -6.283185
| -3.141593
| 0.0
| 3.141593
| 6.283185
| 9.424778
| 12.566371
| ]
|
| rank(self, method: 'RankMethod' = 'average', *, descending: 'bool' = False, seed: 'int | None' = None) -> 'Series'
| Assign ranks to data, dealing with ties appropriately.
|
| Parameters
| ----------
| method : {'average', 'min', 'max', 'dense', 'ordinal', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'ordinal' : All values are given a distinct rank, corresponding to
| the order that the values occur in the Series.
| - 'random' : Like 'ordinal', but the rank for ties is not dependent
| on the order that the values occur in the Series.
| descending
| Rank in descending order.
| seed
| If `method="random"`, use this as seed.
|
| Notes
| -----
| If you're coming from SQL, you may be expecting null values to be ranked last.
| Polars, however, only ranks non-null values and preserves the null ones.
|
| Examples
| --------
| The 'average' method:
|
| >>> s = pl.Series("a", [3, 6, 1, 1, 6])
| >>> s.rank()
| shape: (5,)
| Series: 'a' [f64]
| [
| 3.0
| 4.5
| 1.5
| 1.5
| 4.5
| ]
|
| The 'ordinal' method:
|
| >>> s = pl.Series("a", [3, 6, 1, 1, 6])
| >>> s.rank("ordinal")
| shape: (5,)
| Series: 'a' [u32]
| [
| 3
| 4
| 1
| 2
| 5
| ]
|
| rechunk(self, *, in_place: 'bool' = False) -> 'Self'
| Create a single chunk of memory for this Series.
|
| Parameters
| ----------
| in_place
| In place or not.
|
| Examples
| --------
| >>> s1 = pl.Series("a", [1, 2, 3])
| >>> s1.n_chunks()
| 1
| >>> s2 = pl.Series("a", [4, 5, 6])
| >>> s = pl.concat([s1, s2], rechunk=False)
| >>> s.n_chunks()
| 2
| >>> s.rechunk(in_place=True)
| shape: (6,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| ]
| >>> s.n_chunks()
| 1
|
| reinterpret(self, *, signed: 'bool | None' = None, dtype: 'type[int | float] | PolarsDataType | None' = None) -> 'Series'
| Reinterpret the underlying bits as a signed/unsigned integer or float.
|
| This operation is only allowed for numeric types of the same size.
| For lower bits numbers, you can safely use the cast operation.
|
| Either `signed` or `dtype` can be specified.
| Defaults to `signed=True` otherwise.
|
| Parameters
| ----------
| signed
| If True, reinterpret as signed integer. Otherwise, reinterpret
| as unsigned integer.
| dtype
| DataType to reinterpret to.
|
| Examples
| --------
| >>> s = pl.Series("a", [-(2**60), -2, 3])
| >>> s
| shape: (3,)
| Series: 'a' [i64]
| [
| -1152921504606846976
| -2
| 3
| ]
| >>> s.reinterpret(signed=False)
| shape: (3,)
| Series: 'a' [u64]
| [
| 17293822569102704640
| 18446744073709551614
| 3
| ]
| >>> s.reinterpret(dtype=pl.Int64)
| shape: (3,)
| Series: 'a' [i64]
| [
| -1152921504606846976
| -2
| 3
| ]
|
| rename(self, name: 'str_') -> 'Series'
| Rename this Series.
|
| Alias for :func:`Series.alias`.
|
| Parameters
| ----------
| name
| New name.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.rename("b")
| shape: (3,)
| Series: 'b' [i64]
| [
| 1
| 2
| 3
| ]
|
| repeat_by(self, by: 'int | IntoExprColumn') -> 'Self'
| Repeat the elements in this Series as specified in the given expression.
|
| The repeated elements are expanded into a List.
|
| Parameters
| ----------
| by
| Numeric column that determines how often the values will be repeated.
| The column will be coerced to UInt32. Give this dtype to make the coercion
| a no-op.
|
| Returns
| -------
| Expr
| Expression of data type List, where the inner data type is equal to the
| original data type.
|
| replace(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | None' = None) -> 'Self'
| Replace values by different values of the same data type.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Length must match the length of `old` or have length 1.
|
| default
| Set values that were not replaced to this value.
| Defaults to keeping the original value.
| Accepts expression input. Non-expression inputs are parsed as literals.
|
| .. deprecated:: 0.20.31
| Use :meth:`replace_strict` instead to set a default while
| replacing values.
|
| return_dtype
| The data type of the resulting expression. If set to `None` (default),
| the data type is determined automatically based on the other inputs.
|
| .. deprecated:: 0.20.31
| Use :meth:`replace_strict` instead to set a return data type while
| replacing values.
|
|
| See Also
| --------
| replace_strict
| str.replace
|
| Examples
| --------
| Replace a single value by another value. Values that were not replaced remain
| unchanged.
|
| >>> s = pl.Series([1, 2, 2, 3])
| >>> s.replace(2, 100)
| shape: (4,)
| Series: '' [i64]
| [
| 1
| 100
| 100
| 3
| ]
|
| Replace multiple values by passing sequences to the `old` and `new` parameters.
|
| >>> s.replace([2, 3], [100, 200])
| shape: (4,)
| Series: '' [i64]
| [
| 1
| 100
| 100
| 200
| ]
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {2: 100, 3: 200}
| >>> s.replace(mapping)
| shape: (4,)
| Series: '' [i64]
| [
| 1
| 100
| 100
| 200
| ]
|
| The original data type is preserved when replacing by values of a different
| data type. Use :meth:`replace_strict` to replace and change the return data
| type.
|
| >>> s = pl.Series(["x", "y", "z"])
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> s.replace(mapping)
| shape: (3,)
| Series: '' [str]
| [
| "1"
| "2"
| "3"
| ]
|
| replace_strict(self, old: 'IntoExpr | Sequence[Any] | Mapping[Any, Any]', new: 'IntoExpr | Sequence[Any] | NoDefault' = <no_default>, *, default: 'IntoExpr | NoDefault' = <no_default>, return_dtype: 'PolarsDataType | None' = None) -> 'Self'
| Replace all values by different values.
|
| Parameters
| ----------
| old
| Value or sequence of values to replace.
| Also accepts a mapping of values to their replacement as syntactic sugar for
| `replace_strict(old=Series(mapping.keys()), new=Series(mapping.values()))`.
| new
| Value or sequence of values to replace by.
| Length must match the length of `old` or have length 1.
| default
| Set values that were not replaced to this value. If no default is specified,
| (default), an error is raised if any values were not replaced.
| Accepts expression input. Non-expression inputs are parsed as literals.
| return_dtype
| The data type of the resulting Series. If set to `None` (default),
| the data type is determined automatically based on the other inputs.
|
| Raises
| ------
| InvalidOperationError
| If any non-null values in the original column were not replaced, and no
| `default` was specified.
|
| See Also
| --------
| replace
| str.replace
|
| Examples
| --------
| Replace values by passing sequences to the `old` and `new` parameters.
|
| >>> s = pl.Series([1, 2, 2, 3])
| >>> s.replace_strict([1, 2, 3], [100, 200, 300])
| shape: (4,)
| Series: '' [i64]
| [
| 100
| 200
| 200
| 300
| ]
|
| Passing a mapping with replacements is also supported as syntactic sugar.
|
| >>> mapping = {1: 100, 2: 200, 3: 300}
| >>> s.replace_strict(mapping)
| shape: (4,)
| Series: '' [i64]
| [
| 100
| 200
| 200
| 300
| ]
|
| By default, an error is raised if any non-null values were not replaced.
| Specify a default to set all values that were not matched.
|
| >>> mapping = {2: 200, 3: 300}
| >>> s.replace_strict(mapping) # doctest: +SKIP
| Traceback (most recent call last):
| ...
| polars.exceptions.InvalidOperationError: incomplete mapping specified for `replace_strict`
| >>> s.replace_strict(mapping, default=-1)
| shape: (4,)
| Series: '' [i64]
| [
| -1
| 200
| 200
| 300
| ]
|
| The default can be another Series.
|
| >>> default = pl.Series([2.5, 5.0, 7.5, 10.0])
| >>> s.replace_strict(2, 200, default=default)
| shape: (4,)
| Series: '' [f64]
| [
| 2.5
| 200.0
| 200.0
| 10.0
| ]
|
| Replacing by values of a different data type sets the return type based on
| a combination of the `new` data type and the `default` data type.
|
| >>> s = pl.Series(["x", "y", "z"])
| >>> mapping = {"x": 1, "y": 2, "z": 3}
| >>> s.replace_strict(mapping)
| shape: (3,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| ]
| >>> s.replace_strict(mapping, default="x")
| shape: (3,)
| Series: '' [str]
| [
| "1"
| "2"
| "3"
| ]
|
| Set the `return_dtype` parameter to control the resulting data type directly.
|
| >>> s.replace_strict(mapping, return_dtype=pl.UInt8)
| shape: (3,)
| Series: '' [u8]
| [
| 1
| 2
| 3
| ]
|
| reshape(self, dimensions: 'tuple[int, ...]') -> 'Series'
| Reshape this Series to a flat Series or an Array Series.
|
| Parameters
| ----------
| dimensions
| Tuple of the dimension sizes. If a -1 is used in any of the dimensions, that
| dimension is inferred.
|
| Returns
| -------
| Series
| If a single dimension is given, results in a Series of the original
| data type.
| If a multiple dimensions are given, results in a Series of data type
| :class:`Array` with shape `dimensions`.
|
| See Also
| --------
| Series.list.explode : Explode a list column.
|
| Examples
| --------
| >>> s = pl.Series("foo", [1, 2, 3, 4, 5, 6, 7, 8, 9])
| >>> square = s.reshape((3, 3))
| >>> square
| shape: (3,)
| Series: 'foo' [array[i64, 3]]
| [
| [1, 2, 3]
| [4, 5, 6]
| [7, 8, 9]
| ]
| >>> square.reshape((9,))
| shape: (9,)
| Series: 'foo' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| 7
| 8
| 9
| ]
|
| reverse(self) -> 'Series'
| Return Series in reverse order.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3], dtype=pl.Int8)
| >>> s.reverse()
| shape: (3,)
| Series: 'a' [i8]
| [
| 3
| 2
| 1
| ]
|
| rle(self) -> 'Series'
| Compress the Series data using run-length encoding.
|
| Run-length encoding (RLE) encodes data by storing each *run* of identical values
| as a single value and its length.
|
| Returns
| -------
| Series
| Series of data type `Struct` with fields `len` of data type `UInt32`
| and `value` of the original data type.
|
| Examples
| --------
| >>> s = pl.Series("s", [1, 1, 2, 1, None, 1, 3, 3])
| >>> s.rle().struct.unnest()
| shape: (6, 2)
| ┌─────┬───────┐
| │ len ┆ value │
| │ --- ┆ --- │
| │ u32 ┆ i64 │
| ╞═════╪═══════╡
| │ 2 ┆ 1 │
| │ 1 ┆ 2 │
| │ 1 ┆ 1 │
| │ 1 ┆ null │
| │ 1 ┆ 1 │
| │ 2 ┆ 3 │
| └─────┴───────┘
|
| rle_id(self) -> 'Series'
| Get a distinct integer ID for each run of identical values.
|
| The ID starts at 0 and increases by one each time the value of the column
| changes.
|
| Returns
| -------
| Series
| Series of data type `UInt32`.
|
| See Also
| --------
| rle
|
| Notes
| -----
| This functionality is especially useful for defining a new group for every time
| a column's value changes, rather than for every distinct value of that column.
|
| Examples
| --------
| >>> s = pl.Series("s", [1, 1, 2, 1, None, 1, 3, 3])
| >>> s.rle_id()
| shape: (8,)
| Series: 's' [u32]
| [
| 0
| 0
| 1
| 2
| 3
| 4
| 5
| 5
| ]
|
| rolling_kurtosis(self, window_size: 'int', *, fisher: 'bool' = True, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling kurtosis.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself, and the `window_size - 1`
| elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| fisher : bool, optional
| If True, Fisher's definition is used (normal ==> 0.0). If False,
| Pearson's definition is used (normal ==> 3.0).
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Series.kurtosis
|
| Examples
| --------
| >>> pl.Series([1, 4, 2, 9]).rolling_kurtosis(3)
| shape: (4,)
| Series: '' [f64]
| [
| null
| null
| -1.5
| -1.5
| ]
|
| rolling_map(self, function: 'Callable[[Series], Any]', window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a custom rolling window function.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| function
| Custom aggregation function.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Warnings
| --------
| Computing custom functions is extremely slow. Use specialized rolling
| functions such as :func:`Series.rolling_sum` if at all possible.
|
| Examples
| --------
| >>> from numpy import nansum
| >>> s = pl.Series([11.0, 2.0, 9.0, float("nan"), 8.0])
| >>> s.rolling_map(nansum, window_size=3)
| shape: (5,)
| Series: '' [f64]
| [
| null
| null
| 22.0
| 11.0
| 17.0
| ]
|
| rolling_max(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Apply a rolling max (moving max) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their max.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [100, 200, 300, 400, 500])
| >>> s.rolling_max(window_size=2)
| shape: (5,)
| Series: 'a' [i64]
| [
| null
| 200
| 300
| 400
| 500
| ]
|
| rolling_max_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling max based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling max with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_max_by(d, "3h")
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| rolling_mean(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Apply a rolling mean (moving mean) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their mean.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [100, 200, 300, 400, 500])
| >>> s.rolling_mean(window_size=2)
| shape: (5,)
| Series: 'a' [f64]
| [
| null
| 150.0
| 250.0
| 350.0
| 450.0
| ]
|
| rolling_mean_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling mean based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling mean with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_mean_by(d, "3h")
| shape: (25,)
| Series: 'index' [f64]
| [
| 0.0
| 0.5
| 1.0
| 2.0
| 3.0
| …
| 19.0
| 20.0
| 21.0
| 22.0
| 23.0
| ]
|
| rolling_median(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling median.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
| >>> s.rolling_median(window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 2.0
| 3.0
| 4.0
| 6.0
| ]
|
| rolling_median_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling median based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling median with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_median_by(d, "3h")
| shape: (25,)
| Series: 'index' [f64]
| [
| 0.0
| 0.5
| 1.0
| 2.0
| 3.0
| …
| 19.0
| 20.0
| 21.0
| 22.0
| 23.0
| ]
|
| rolling_min(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Apply a rolling min (moving min) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their min.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [100, 200, 300, 400, 500])
| >>> s.rolling_min(window_size=3)
| shape: (5,)
| Series: 'a' [i64]
| [
| null
| null
| 100
| 200
| 300
| ]
|
| rolling_min_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling min based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling min with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_min_by(d, "3h")
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 0
| 0
| 1
| 2
| …
| 18
| 19
| 20
| 21
| 22
| ]
|
| rolling_quantile(self, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', window_size: 'int' = 2, weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling quantile.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
| >>> s.rolling_quantile(quantile=0.33, window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 2.0
| 3.0
| 4.0
| 6.0
| ]
| >>> s.rolling_quantile(quantile=0.33, interpolation="linear", window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 1.66
| 2.66
| 3.66
| 5.32
| ]
|
| rolling_quantile_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, quantile: 'float', interpolation: 'QuantileMethod' = 'nearest', min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling quantile based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| quantile
| Quantile between 0.0 and 1.0.
| interpolation : {'nearest', 'higher', 'lower', 'midpoint', 'linear', 'equiprobable'}
| Interpolation method.
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling quantile with the temporal windows from the second series closed on the right:
|
| >>> s.rolling_quantile_by(d, "3h", quantile=0.5)
| shape: (25,)
| Series: 'index' [f64]
| [
| 0.0
| 1.0
| 1.0
| 2.0
| 3.0
| …
| 19.0
| 20.0
| 21.0
| 22.0
| 23.0
| ]
|
| rolling_rank(self, window_size: 'int', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling rank.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| A window of length `window_size` will traverse the array. The values
| that fill this window will be ranked according to the `method`
| parameter. The resulting values will be the rank of the value that is
| at the end of the sliding window.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Returns
| -------
| Series
| A Series of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| Examples
| --------
| >>> pl.Series([1, 4, 4, 1, 9]).rolling_rank(3, method="average")
| shape: (5,)
| Series: '' [f64]
| [
| null
| null
| 2.5
| 1.0
| 3.0
| ]
|
| rolling_rank_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', method: 'RankMethod' = 'average', *, seed: 'int | None' = None, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right') -> 'Series'
| Compute a rolling rank based on another column.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic
| temporal size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| method : {'average', 'min', 'max', 'dense', 'random'}
| The method used to assign ranks to tied elements.
| The following methods are available (default is 'average'):
|
| - 'average' : The average of the ranks that would have been assigned to
| all the tied values is assigned to each value.
| - 'min' : The minimum of the ranks that would have been assigned to all
| the tied values is assigned to each value. (This is also referred to
| as "competition" ranking.)
| - 'max' : The maximum of the ranks that would have been assigned to all
| the tied values is assigned to each value.
| - 'dense' : Like 'min', but the rank of the next highest element is
| assigned the rank immediately after those assigned to the tied
| elements.
| - 'random' : Choose a random rank for each value in a tie.
| seed
| Random seed used when `method='random'`. If set to None (default), a
| random seed is generated for each rolling rank operation.
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Returns
| -------
| Series
| A Series of data :class:`.Float64` if `method` is `"average"` or,
| the index size (see :func:`.get_index_type()`) otherwise.
|
| rolling_skew(self, window_size: 'int', *, bias: 'bool' = True, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Compute a rolling skew.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| The window at a given row includes the row itself and the
| `window_size - 1` elements before it.
|
| Parameters
| ----------
| window_size
| Integer size of the rolling window.
| bias
| If False, the calculations are corrected for statistical bias.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| See Also
| --------
| Series.skew
|
| Examples
| --------
| >>> pl.Series([1, 4, 2, 9]).rolling_skew(3)
| shape: (4,)
| Series: '' [f64]
| [
| null
| null
| 0.381802
| 0.47033
| ]
|
| Note how the values match
|
| >>> pl.Series([1, 4, 2]).skew(), pl.Series([4, 2, 9]).skew()
| (0.38180177416060584, 0.47033046033698594)
|
| rolling_std(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Series'
| Compute a rolling std dev.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their std dev.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
| >>> s.rolling_std(window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 1.0
| 1.0
| 1.527525
| 2.0
| ]
|
| rolling_std_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Self'
| Compute a rolling standard deviation based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling std with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_std_by(d, "3h")
| shape: (25,)
| Series: 'index' [f64]
| [
| null
| 0.707107
| 1.0
| 1.0
| 1.0
| …
| 1.0
| 1.0
| 1.0
| 1.0
| 1.0
| ]
|
| rolling_sum(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False) -> 'Series'
| Apply a rolling sum (moving sum) over the values in this array.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their sum.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.rolling_sum(window_size=2)
| shape: (5,)
| Series: 'a' [i64]
| [
| null
| 3
| 5
| 7
| 9
| ]
|
| rolling_sum_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 0, closed: 'ClosedInterval' = 'right') -> 'Self'
| Compute a rolling sum based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling mean with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_sum_by(d, "3h")
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 3
| 6
| 9
| …
| 57
| 60
| 63
| 66
| 69
| ]
|
| rolling_var(self, window_size: 'int', weights: 'list_[float] | None' = None, *, min_samples: 'int | None' = None, center: 'bool' = False, ddof: 'int' = 1) -> 'Series'
| Compute a rolling variance.
|
| A window of length `window_size` will traverse the array. The values that fill
| this window will (optionally) be multiplied with the weights given by the
| `weight` vector. The resulting values will be aggregated to their variance.
|
| The window at a given row will include the row itself and the `window_size - 1`
| elements before it.
|
| .. versionchanged:: 1.21.0
| The `min_periods` parameter was renamed `min_samples`.
|
| Parameters
| ----------
| window_size
| The length of the window in number of elements.
| weights
| An optional slice with the same length as the window that will be multiplied
| elementwise with the values in the window.
| min_samples
| The number of values in the window that should be non-null before computing
| a result. If set to `None` (default), it will be set equal to `window_size`.
| center
| Set the labels at the center of the window.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
| >>> s.rolling_var(window_size=3)
| shape: (6,)
| Series: 'a' [f64]
| [
| null
| null
| 1.0
| 1.0
| 2.333333
| 4.0
| ]
|
| rolling_var_by(self, by: 'IntoExpr', window_size: 'timedelta | str_', *, min_samples: 'int' = 1, closed: 'ClosedInterval' = 'right', ddof: 'int' = 1) -> 'Self'
| Compute a rolling variance based on another series.
|
| .. warning::
| This functionality is considered **unstable**. It may be changed
| at any point without it being considered a breaking change.
|
| Given a `by` column `<t_0, t_1, ..., t_n>`, then `closed="right"`
| (the default) means the windows will be:
|
| - (t_0 - window_size, t_0]
| - (t_1 - window_size, t_1]
| - ...
| - (t_n - window_size, t_n]
|
| Parameters
| ----------
| by
| Should be ``DateTime``, ``Date``, ``UInt64``, ``UInt32``, ``Int64``,
| or ``Int32`` data type (note that the integral ones require using `'i'`
| in `window size`).
| window_size
| The length of the window. Can be a dynamic temporal
| size indicated by a timedelta or the following string language:
|
| - 1ns (1 nanosecond)
| - 1us (1 microsecond)
| - 1ms (1 millisecond)
| - 1s (1 second)
| - 1m (1 minute)
| - 1h (1 hour)
| - 1d (1 calendar day)
| - 1w (1 calendar week)
| - 1mo (1 calendar month)
| - 1q (1 calendar quarter)
| - 1y (1 calendar year)
| - 1i (1 index count)
|
| By "calendar day", we mean the corresponding time on the next day
| (which may not be 24 hours, due to daylight savings - in cases of ambiguity,
| we follow RFC-5545 and preserve the DST fold of the original datetime).
| Similarly for "calendar week", "calendar month", "calendar quarter", and
| "calendar year".
| min_samples
| The number of values in the window that should be non-null before computing
| a result.
| closed : {'left', 'right', 'both', 'none'}
| Define which sides of the temporal interval are closed (inclusive),
| defaults to `'right'`.
| ddof
| "Delta Degrees of Freedom": The divisor for a length N window is N - ddof
|
| Notes
| -----
| If you want to compute multiple aggregation statistics over the same dynamic
| window, consider using `rolling` - this method can cache the window size
| computation.
|
| Examples
| --------
| Create a series with a row index value
|
| >>> from datetime import timedelta, datetime
| >>> start = datetime(2001, 1, 1)
| >>> stop = datetime(2001, 1, 2)
| >>> s = pl.Series("index", range(25))
| >>> s
| shape: (25,)
| Series: 'index' [i64]
| [
| 0
| 1
| 2
| 3
| 4
| …
| 20
| 21
| 22
| 23
| 24
| ]
|
| Create another series to apply the window mask:
|
| >>> d = pl.Series("date", pl.datetime_range(start, stop, "1h", eager=True))
| >>> d
| shape: (25,)
| Series: 'date' [datetime[μs]]
| [
| 2001-01-01 00:00:00
| 2001-01-01 01:00:00
| 2001-01-01 02:00:00
| 2001-01-01 03:00:00
| 2001-01-01 04:00:00
| …
| 2001-01-01 20:00:00
| 2001-01-01 21:00:00
| 2001-01-01 22:00:00
| 2001-01-01 23:00:00
| 2001-01-02 00:00:00
| ]
|
| Compute the rolling std with the temporal windows
| from the second series closed on the right:
|
| >>> s.rolling_std_by(d, "3h")
| shape: (25,)
| Series: 'index' [f64]
| [
| null
| 0.707107
| 1.0
| 1.0
| 1.0
| …
| 1.0
| 1.0
| 1.0
| 1.0
| 1.0
| ]
|
| round(self, decimals: 'int' = 0, mode: 'RoundMode' = 'half_to_even') -> 'Series'
| Round underlying floating point data by `decimals` digits.
|
| Parameters
| ----------
| decimals
| Number of decimals to round by.
| mode : {'half_to_even', 'half_away_from_zero', 'to_zero'}
| The rounding strategy used. A "rounded value" is a value with at most
| `decimals` decimal places (e.g. integers when ``decimals=0``, multiples
| of 0.1 when ``decimals=1``, 0.01 when ``decimals=2``, and so on).
|
| Strategies that start with ``half_`` round all values to the *nearest*
| rounded value, only using the strategy to break ties when a value falls
| exactly between two rounded values (e.g. 0.5 when ``decimals=0``, 0.05
| when ``decimals=1``). Other rounding strategies specify explicitly
| which rounded value is chosen and always apply (not just for tiebreaks).
|
| * *half_to_even* (default)
| Round to the nearest value; break ties by choosing the nearest
| **even** value. For example, 0.5 rounds to 0, 1.5 rounds to 2,
| 2.5 rounds to 2. Also known as "banker's rounding"; this is the
| default because it tends to minimise cumulative rounding bias.
| * *half_away_from_zero*
| Round to the nearest value; break ties by rounding **away from
| zero**. For example, 0.5 rounds to 1, -0.5 rounds to -1, 2.5
| rounds to 3. Also known as "commercial rounding".
| * *to_zero*
| Always round (truncate) **towards zero**, discarding the fractional
| part beyond `decimals`. For example, 0.9 rounds to 0, -0.9 rounds
| to 0, 1.29 rounds to 1.2 (with ``decimals=1``). Equivalent to the
| :meth:`truncate` method.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.12345, 2.56789, 3.901234])
| >>> s.round(2)
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.12
| 2.57
| 3.9
| ]
|
| >>> s = pl.Series([-3.5, -2.5, -1.5, -0.5, 0.5, 1.5, 2.5, 3.5])
| >>> s.round(mode="half_to_even")
| shape: (8,)
| Series: '' [f64]
| [
| -4.0
| -2.0
| -2.0
| -0.0
| 0.0
| 2.0
| 2.0
| 4.0
| ]
|
| round_sig_figs(self, digits: 'int') -> 'Series'
| Round to a number of significant figures.
|
| Parameters
| ----------
| digits
| Number of significant figures to round to.
|
| Examples
| --------
| >>> s = pl.Series([0.01234, 3.333, 3450.0])
| >>> s.round_sig_figs(2)
| shape: (3,)
| Series: '' [f64]
| [
| 0.012
| 3.3
| 3500.0
| ]
|
| sample(self, n: 'int | None' = None, *, fraction: 'float | None' = None, with_replacement: 'bool' = False, shuffle: 'bool | None' = None, seed: 'int | None' = None) -> 'Series'
| Sample from this Series.
|
| Parameters
| ----------
| n
| Number of items to return. Cannot be used with `fraction`. Defaults to 1 if
| `fraction` is None.
| fraction
| Fraction of items to return. Cannot be used with `n`.
| with_replacement
| Allow values to be sampled more than once.
| shuffle
| Determines the order of the sampled values.
| If True, sampled values are explicitly shuffled.
| If False, the relative order of the sampled values is preserved.
| (i.e. they appear in the same order as the original Series).
| If None (default), no ordering guarantee; uses the most performant
| algorithm.
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated for each sample operation.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.sample(2, shuffle=False, seed=0) # doctest: +IGNORE_RESULT
| shape: (2,)
| Series: 'a' [i64]
| [
| 1
| 5
| ]
|
| scatter(self, indices: 'Series | Iterable[int] | int | np.ndarray[Any, Any]', values: 'Series | Iterable[PythonLiteral] | PythonLiteral | None') -> 'Series'
| Set values at the index locations.
|
| Parameters
| ----------
| indices
| Integers representing the index locations.
| values
| Replacement values.
|
| Notes
| -----
| Use of this function is frequently an anti-pattern, as it can
| block optimization (predicate pushdown, etc). Consider using
| `pl.when(predicate).then(value).otherwise(self)` instead.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.scatter(1, 10)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 10
| 3
| ]
|
| It is better to implement this as follows:
|
| >>> s.to_frame().with_row_index().select(
| ... pl.when(pl.col("index") == 1).then(10).otherwise(pl.col("a"))
| ... )
| shape: (3, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ i64 │
| ╞═════════╡
| │ 1 │
| │ 10 │
| │ 3 │
| └─────────┘
|
| search_sorted(self, element: 'IntoExpr | np.ndarray[Any, Any] | None', side: 'SearchSortedSide' = 'any', *, descending: 'bool' = False) -> 'int | Series'
| Find indices where elements should be inserted to maintain order.
|
| .. math:: a[i-1] < v <= a[i]
|
| Parameters
| ----------
| element
| Expression or scalar value.
| side : {'any', 'left', 'right'}
| If 'any', the index of the first suitable location found is given.
| If 'left', the index of the leftmost suitable location found is given.
| If 'right', return the rightmost suitable location found is given.
| descending
| Boolean indicating whether the values are descending or not (they
| are required to be sorted either way).
|
| Examples
| --------
| >>> s = pl.Series("set", [1, 2, 3, 4, 4, 5, 6, 7])
| >>> s.search_sorted(4)
| 3
| >>> s.search_sorted(4, "left")
| 3
| >>> s.search_sorted(4, "right")
| 5
| >>> s.search_sorted([1, 4, 5])
| shape: (3,)
| Series: 'set' [u32]
| [
| 0
| 3
| 5
| ]
| >>> s.search_sorted([1, 4, 5], "left")
| shape: (3,)
| Series: 'set' [u32]
| [
| 0
| 3
| 5
| ]
| >>> s.search_sorted([1, 4, 5], "right")
| shape: (3,)
| Series: 'set' [u32]
| [
| 1
| 5
| 6
| ]
|
| set(self, filter: 'Series', value: 'Any') -> 'Series'
| Set masked values.
|
| Parameters
| ----------
| filter
| Boolean mask.
| value
| Value with which to replace the masked values.
|
| Notes
| -----
| Use of this function is frequently an anti-pattern, as it can
| block optimisation (predicate pushdown, etc). Consider using
| `pl.when(predicate).then(value).otherwise(self)` instead.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.set(s == 2, 10)
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 10
| 3
| ]
|
| It is better to implement this as follows:
|
| >>> s.to_frame().select(
| ... pl.when(pl.col("a") == 2).then(10).otherwise(pl.col("a"))
| ... )
| shape: (3, 1)
| ┌─────────┐
| │ literal │
| │ --- │
| │ i64 │
| ╞═════════╡
| │ 1 │
| │ 10 │
| │ 3 │
| └─────────┘
|
| set_sorted(self, *, descending: 'bool' = False) -> 'Self'
| Flags the Series as 'sorted'.
|
| Enables downstream code to user fast paths for sorted arrays.
|
| Parameters
| ----------
| descending
| If the `Series` order is descending.
|
| Warnings
| --------
| This can lead to incorrect results if this `Series` is not sorted!!
| Use with care!
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.set_sorted().max()
| 3
|
| shift(self, n: 'int' = 1, *, fill_value: 'IntoExpr | None' = None) -> 'Series'
| Shift values by the given number of indices.
|
| Parameters
| ----------
| n
| Number of indices to shift forward. If a negative value is passed, values
| are shifted in the opposite direction instead.
| fill_value
| Fill the resulting null values with this value. Accepts scalar expression
| input. Non-expression inputs are parsed as literals.
|
| Notes
| -----
| This method is similar to the `LAG` operation in SQL when the value for `n`
| is positive. With a negative value for `n`, it is similar to `LEAD`.
|
| Examples
| --------
| By default, values are shifted forward by one index.
|
| >>> s = pl.Series([1, 2, 3, 4])
| >>> s.shift()
| shape: (4,)
| Series: '' [i64]
| [
| null
| 1
| 2
| 3
| ]
|
| Pass a negative value to shift in the opposite direction instead.
|
| >>> s.shift(-2)
| shape: (4,)
| Series: '' [i64]
| [
| 3
| 4
| null
| null
| ]
|
| Specify `fill_value` to fill the resulting null values.
|
| >>> s.shift(-2, fill_value=100)
| shape: (4,)
| Series: '' [i64]
| [
| 3
| 4
| 100
| 100
| ]
|
| shrink_dtype(self) -> 'Series'
| Shrink numeric columns to the minimal required datatype.
|
| Shrink to the dtype needed to fit the extrema of this [`Series`].
| This can be used to reduce memory pressure.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5, 6])
| >>> s
| shape: (6,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| ]
| >>> s.shrink_dtype()
| shape: (6,)
| Series: 'a' [i8]
| [
| 1
| 2
| 3
| 4
| 5
| 6
| ]
|
| shrink_to_fit(self, *, in_place: 'bool' = False) -> 'Series'
| Shrink Series memory usage.
|
| Shrinks the underlying array capacity to exactly fit the actual data.
| (Note that this function does not change the Series data type).
|
| shuffle(self, seed: 'int | None' = None) -> 'Series'
| Shuffle the contents of this Series.
|
| Parameters
| ----------
| seed
| Seed for the random number generator. If set to None (default), a
| random seed is generated each time the shuffle is called.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.shuffle(seed=1)
| shape: (3,)
| Series: 'a' [i64]
| [
| 2
| 3
| 1
| ]
|
| sign(self) -> 'Series'
| Compute the element-wise sign function on numeric types.
|
| The returned value is computed as follows:
|
| * -1 if x < 0.
| * 1 if x > 0.
| * x otherwise (typically 0, but could be NaN if the input is).
|
| Null values are preserved as-is, and the dtype of the input is preserved.
|
| Examples
| --------
| >>> s = pl.Series("a", [-9.0, -0.0, 0.0, 4.0, float("nan"), None])
| >>> s.sign()
| shape: (6,)
| Series: 'a' [f64]
| [
| -1.0
| -0.0
| 0.0
| 1.0
| NaN
| null
| ]
|
| sin(self) -> 'Series'
| Compute the element-wise value for the sine.
|
| Notes
| -----
| Input values are interpreted as radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Series.radians>`.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [0.0, math.pi / 2.0, math.pi])
| >>> s.sin()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.0
| 1.0
| 1.2246e-16
| ]
|
| sinh(self) -> 'Series'
| Compute the element-wise value for the hyperbolic sine.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.sinh()
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.175201
| 0.0
| -1.175201
| ]
|
| skew(self, *, bias: 'bool' = True) -> 'float | None'
| Compute the sample skewness of a data set.
|
| For normally distributed data, the skewness should be about zero. For
| unimodal continuous distributions, a skewness value greater than zero means
| that there is more weight in the right tail of the distribution. The
| function `skewtest` can be used to determine if the skewness value
| is close enough to zero, statistically speaking.
|
|
| See scipy.stats for more information.
|
| Parameters
| ----------
| bias : bool, optional
| If False, the calculations are corrected for statistical bias.
|
| Notes
| -----
| The sample skewness is computed as the Fisher-Pearson coefficient
| of skewness, i.e.
|
| .. math:: g_1=\frac{m_3}{m_2^{3/2}}
|
| where
|
| .. math:: m_i=\frac{1}{N}\sum_{n=1}^N(x[n]-\bar{x})^i
|
| is the biased sample :math:`i\texttt{th}` central moment, and
| :math:`\bar{x}` is
| the sample mean. If `bias` is False, the calculations are
| corrected for bias and the value computed is the adjusted
| Fisher-Pearson standardized moment coefficient, i.e.
|
| .. math::
| G_1 = \frac{k_3}{k_2^{3/2}} = \frac{\sqrt{N(N-1)}}{N-2}\frac{m_3}{m_2^{3/2}}
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 2, 4, 5])
| >>> s.skew()
| 0.34776706224699483
|
| slice(self, offset: 'int', length: 'int | None' = None) -> 'Series'
| Get a slice of this Series.
|
| Parameters
| ----------
| offset
| Start index. Negative indexing is supported.
| length
| Length of the slice. If set to `None`, all rows starting at the offset
| will be selected.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4])
| >>> s.slice(1, 2)
| shape: (2,)
| Series: 'a' [i64]
| [
| 2
| 3
| ]
|
| sort(self, *, descending: 'bool' = False, nulls_last: 'bool' = False, multithreaded: 'bool' = True, in_place: 'bool' = False) -> 'Self'
| Sort this Series.
|
| Parameters
| ----------
| descending
| Sort in descending order.
| nulls_last
| Place null values last instead of first.
| multithreaded
| Sort using multiple threads.
| in_place
| Sort in-place.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 3, 4, 2])
| >>> s.sort()
| shape: (4,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| 4
| ]
| >>> s.sort(descending=True)
| shape: (4,)
| Series: 'a' [i64]
| [
| 4
| 3
| 2
| 1
| ]
|
| sql(self, query: 'str_', *, table_name: 'str_' = 'self') -> 'DataFrame'
| Execute a SQL query against the Series.
|
| .. versionadded:: 1.37.0
|
| .. warning::
| This functionality is considered **unstable**, although it is close to
| being considered stable. It may be changed at any point without it being
| considered a breaking change.
|
| Parameters
| ----------
| query
| SQL query to execute.
| table_name
| Optionally provide an explicit name for the table that represents the
| calling frame (defaults to "self").
|
| Notes
| -----
| * The calling Series is automatically registered as a table in the SQLContext
| under the name "self". If you want access to the DataFrames, LazyFrames, and
| other Series found in the current globals, use :meth:`pl.sql <polars.sql>`.
| * More control over registration and execution behaviour is available by
| using the :class:`SQLContext` object.
| * The SQL query executes in lazy mode before being collected and returned
| as a DataFrame.
| * It is recommended to name your Series for use with SQL, otherwise the default
| Series name (an empty string) is used; while `""` is valid, it is awkward.
|
| See Also
| --------
| SQLContext
|
| Examples
| --------
| >>> from datetime import date
| >>> s = pl.Series(
| ... name="dt",
| ... values=[date(1999, 12, 31), date(2099, 2, 14), date(2026, 3, 5)],
| ... )
|
| Query the Series using SQL:
|
| >>> s.sql('''
| ... SELECT
| ... EXTRACT('year',dt) AS y,
| ... EXTRACT('month',dt) AS m,
| ... EXTRACT('day',dt) AS d,
| ... FROM self
| ... WHERE dt > '2020-01-01'
| ... ORDER BY dt DESC
| ... ''')
| shape: (2, 3)
| ┌──────┬─────┬─────┐
| │ y ┆ m ┆ d │
| │ --- ┆ --- ┆ --- │
| │ i32 ┆ i8 ┆ i8 │
| ╞══════╪═════╪═════╡
| │ 2099 ┆ 2 ┆ 14 │
| │ 2026 ┆ 3 ┆ 5 │
| └──────┴─────┴─────┘
|
| While you can refer to an unnamed Series column using the default empty
| string, it is not recommended:
|
| >>> s = pl.Series([1, 2, 3])
| >>> s.sql('SELECT "" AS x, "" * 2 AS "2x" FROM self')
| shape: (3, 2)
| ┌─────┬─────┐
| │ x ┆ 2x │
| │ --- ┆ --- │
| │ i64 ┆ i64 │
| ╞═════╪═════╡
| │ 1 ┆ 2 │
| │ 2 ┆ 4 │
| │ 3 ┆ 6 │
| └─────┴─────┘
|
| sqrt(self) -> 'Series'
| Compute the square root of the elements.
|
| Syntactic sugar for
|
| >>> pl.Series([1, 2]) ** 0.5
| shape: (2,)
| Series: '' [f64]
| [
| 1.0
| 1.414214
| ]
|
| Examples
| --------
| >>> s = pl.Series([1, 2, 3])
| >>> s.sqrt()
| shape: (3,)
| Series: '' [f64]
| [
| 1.0
| 1.414214
| 1.732051
| ]
|
| std(self, ddof: 'int' = 1) -> 'float | timedelta | None'
| Get the standard deviation of this Series.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.std()
| 1.0
|
| sum(self) -> 'int | float | PyDecimal'
| Reduce this Series to the sum value.
|
| Notes
| -----
| * Dtypes in {Int8, UInt8, Int16, UInt16} are cast to
| Int64 before summing to prevent overflow issues.
| * If there are no non-null values, then the output is `0`.
| If you would prefer empty sums to return `None`, you can
| use `s.sum() if s.count() else None` instead
| of `s.sum()`.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.sum()
| 6
|
| tail(self, n: 'int' = 10) -> 'Series'
| Get the last `n` elements.
|
| Parameters
| ----------
| n
| Number of elements to return. If a negative value is passed, return all
| elements except the first `abs(n)`.
|
| See Also
| --------
| head, slice
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3, 4, 5])
| >>> s.tail(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 3
| 4
| 5
| ]
|
| Pass a negative value to get all rows `except` the first `abs(n)`.
|
| >>> s.tail(-3)
| shape: (2,)
| Series: 'a' [i64]
| [
| 4
| 5
| ]
|
| tan(self) -> 'Series'
| Compute the element-wise value for the tangent.
|
| Notes
| -----
| Input values are interpreted as radians.
| To convert from degrees to radians,
| call :meth:`.radians() <polars.Series.radians>`.
|
| Examples
| --------
| >>> import math
| >>> s = pl.Series("a", [0.0, math.pi / 2.0, math.pi])
| >>> s.tan()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.0
| 1.6331e16
| -1.2246e-16
| ]
|
| tanh(self) -> 'Series'
| Compute the element-wise value for the hyperbolic tangent.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.0, 0.0, -1.0])
| >>> s.tanh()
| shape: (3,)
| Series: 'a' [f64]
| [
| 0.761594
| 0.0
| -0.761594
| ]
|
| to_arrow(self, *, compat_level: 'CompatLevel | None' = None) -> 'pa.Array'
| Return the underlying Arrow array.
|
| If the Series contains only a single chunk this operation is zero copy.
|
| .. versionchanged:: 1.24
| The `future` parameter was renamed `compat_level`.
|
| Parameters
| ----------
| compat_level
| Use a specific compatibility level
| when exporting Polars' internal data structures.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s = s.to_arrow()
| >>> s
| <pyarrow.lib.Int64Array object at ...>
| [
| 1,
| 2,
| 3
| ]
|
| to_dummies(self, *, separator: 'str_' = '_', drop_first: 'bool' = False, drop_nulls: 'bool' = False) -> 'DataFrame'
| Get dummy/indicator variables.
|
| Parameters
| ----------
| separator
| Separator/delimiter used when generating column names.
| drop_first
| Remove the first category from the variable being encoded.
| drop_nulls
| If there are `None` values in the series, a `null` column is not generated.
| Null values in the input are represented by zero vectors.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.to_dummies()
| shape: (3, 3)
| ┌─────┬─────┬─────┐
| │ a_1 ┆ a_2 ┆ a_3 │
| │ --- ┆ --- ┆ --- │
| │ u8 ┆ u8 ┆ u8 │
| ╞═════╪═════╪═════╡
| │ 1 ┆ 0 ┆ 0 │
| │ 0 ┆ 1 ┆ 0 │
| │ 0 ┆ 0 ┆ 1 │
| └─────┴─────┴─────┘
|
| >>> s.to_dummies(drop_first=True)
| shape: (3, 2)
| ┌─────┬─────┐
| │ a_2 ┆ a_3 │
| │ --- ┆ --- │
| │ u8 ┆ u8 │
| ╞═════╪═════╡
| │ 0 ┆ 0 │
| │ 1 ┆ 0 │
| │ 0 ┆ 1 │
| └─────┴─────┘
|
| >>> s = pl.Series("a", [1, 2, None, 3])
| >>> s.to_dummies(drop_nulls=True, drop_first=True)
| shape: (4, 2)
| ┌─────┬─────┐
| │ a_2 ┆ a_3 │
| │ --- ┆ --- │
| │ u8 ┆ u8 │
| ╞═════╪═════╡
| │ 0 ┆ 0 │
| │ 1 ┆ 0 │
| │ 0 ┆ 0 │
| │ 0 ┆ 1 │
| └─────┴─────┘
|
| to_frame(self, name: 'str_ | None' = None) -> 'DataFrame'
| Cast this Series to a DataFrame.
|
| Parameters
| ----------
| name
| optionally name/rename the Series column in the new DataFrame.
|
| Examples
| --------
| >>> s = pl.Series("a", [123, 456])
| >>> df = s.to_frame()
| >>> df
| shape: (2, 1)
| ┌─────┐
| │ a │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 123 │
| │ 456 │
| └─────┘
|
| >>> df = s.to_frame("xyz")
| >>> df
| shape: (2, 1)
| ┌─────┐
| │ xyz │
| │ --- │
| │ i64 │
| ╞═════╡
| │ 123 │
| │ 456 │
| └─────┘
|
| to_init_repr(self, n: 'int' = 1000) -> 'str_'
| Convert Series to instantiable string representation.
|
| Parameters
| ----------
| n
| Only use first n elements.
|
| See Also
| --------
| polars.Series.to_init_repr
| polars.from_repr
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, None, 4], dtype=pl.Int16)
| >>> print(s.to_init_repr())
| pl.Series('a', [1, 2, None, 4], dtype=pl.Int16)
| >>> s_from_str_repr = eval(s.to_init_repr())
| >>> s_from_str_repr
| shape: (4,)
| Series: 'a' [i16]
| [
| 1
| 2
| null
| 4
| ]
|
| to_jax(self, device: 'jax.Device | str_ | None' = None) -> 'jax.Array'
| Convert this Series to a Jax Array.
|
| .. versionadded:: 0.20.27
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Parameters
| ----------
| device
| Specify the jax `Device` on which the array will be created; can provide
| a string (such as "cpu", "gpu", or "tpu") in which case the device is
| retrieved as `jax.devices(string)[0]`. For more specific control you
| can supply the instantiated `Device` directly. If None, arrays are
| created on the default device.
|
| Examples
| --------
| >>> s = pl.Series("x", [10.5, 0.0, -10.0, 5.5])
| >>> s.to_jax()
| Array([ 10.5, 0. , -10. , 5.5], dtype=float32)
|
| to_list(self) -> 'list_[Any]'
| Convert this Series to a Python list.
|
| This operation copies data.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.to_list()
| [1, 2, 3]
| >>> type(s.to_list())
| <class 'list'>
|
| to_numpy(self, *, writable: 'bool' = False, allow_copy: 'bool' = True, use_pyarrow: 'bool | None' = None, zero_copy_only: 'bool | None' = None) -> 'np.ndarray[Any, Any]'
| Convert this Series to a NumPy ndarray.
|
| This operation copies data only when necessary. The conversion is zero copy when
| all of the following hold:
|
| - The data type is an integer, float, `Datetime`, `Duration`, or `Array`.
| - The Series contains no null values.
| - The Series consists of a single chunk.
| - The `writable` parameter is set to `False` (default).
|
| Parameters
| ----------
| writable
| Ensure the resulting array is writable. This will force a copy of the data
| if the array was created without copy as the underlying Arrow data is
| immutable.
| allow_copy
| Allow memory to be copied to perform the conversion. If set to `False`,
| causes conversions that are not zero-copy to fail.
|
| use_pyarrow
| First convert to PyArrow, then call `pyarrow.Array.to_numpy
| <https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy>`_
| to convert to NumPy. If set to `False`, Polars' own conversion logic is
| used.
|
| .. deprecated:: 0.20.28
| Polars now uses its native engine by default for conversion to NumPy.
| To use PyArrow's engine, call `.to_arrow().to_numpy()` instead.
|
| zero_copy_only
| Raise an exception if the conversion to a NumPy would require copying
| the underlying data. Data copy occurs, for example, when the Series contains
| nulls or non-numeric types.
|
| .. deprecated:: 0.20.10
| Use the `allow_copy` parameter instead, which is the inverse of this
| one.
|
| Examples
| --------
| Numeric data without nulls can be converted without copying data.
| The resulting array will not be writable.
|
| >>> s = pl.Series([1, 2, 3], dtype=pl.Int8)
| >>> arr = s.to_numpy()
| >>> arr
| array([1, 2, 3], dtype=int8)
| >>> arr.flags.writeable
| False
|
| Set `writable=True` to force data copy to make the array writable.
|
| >>> s.to_numpy(writable=True).flags.writeable
| True
|
| Integer Series containing nulls will be cast to a float type with `nan`
| representing a null value. This requires data to be copied.
|
| >>> s = pl.Series([1, 2, None], dtype=pl.UInt16)
| >>> s.to_numpy()
| array([ 1., 2., nan], dtype=float32)
|
| Set `allow_copy=False` to raise an error if data would be copied.
|
| >>> s.to_numpy(allow_copy=False) # doctest: +SKIP
| Traceback (most recent call last):
| ...
| RuntimeError: copy not allowed: cannot convert to a NumPy array without copying data
|
| Series of data type `Array` and `Struct` will result in an array with more than
| one dimension.
|
| >>> s = pl.Series([[1, 2, 3], [4, 5, 6]], dtype=pl.Array(pl.Int64, 3))
| >>> s.to_numpy()
| array([[1, 2, 3],
| [4, 5, 6]])
|
| to_pandas(self, *, use_pyarrow_extension_array: 'bool' = False, **kwargs: 'Any') -> 'pd.Series[Any]'
| Convert this Series to a pandas Series.
|
| This operation copies data if `use_pyarrow_extension_array` is not enabled.
|
| Parameters
| ----------
| use_pyarrow_extension_array
| Use a PyArrow-backed extension array instead of a NumPy array for the pandas
| Series. This allows zero copy operations and preservation of null values.
| Subsequent operations on the resulting pandas Series may trigger conversion
| to NumPy if those operations are not supported by PyArrow compute functions.
| **kwargs
| Additional keyword arguments to be passed to
| :meth:`pyarrow.Array.to_pandas`.
|
| Returns
| -------
| :class:`pandas.Series`
|
| Notes
| -----
| This operation requires that both :mod:`pandas` and :mod:`pyarrow` are
| installed.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.to_pandas()
| 0 1
| 1 2
| 2 3
| Name: a, dtype: int64
|
| Null values are converted to `NaN`.
|
| >>> s = pl.Series("b", [1, 2, None])
| >>> s.to_pandas()
| 0 1.0
| 1 2.0
| 2 NaN
| Name: b, dtype: float64
|
| Pass `use_pyarrow_extension_array=True` to get a pandas Series backed by a
| PyArrow extension array. This will preserve null values.
|
| >>> s.to_pandas(use_pyarrow_extension_array=True)
| 0 1
| 1 2
| 2 <NA>
| Name: b, dtype: int64[pyarrow]
|
| to_physical(self) -> 'Series'
| Cast to physical representation of the logical dtype.
|
| - :func:`polars.datatypes.Date` -> :func:`polars.datatypes.Int32`
| - :func:`polars.datatypes.Datetime` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Time` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Duration` -> :func:`polars.datatypes.Int64`
| - :func:`polars.datatypes.Categorical` -> :func:`polars.datatypes.UInt32`
| - `List(inner)` -> `List(physical of inner)`
| - `Array(inner)` -> `Array(physical of inner)`
| - `Struct(fields)` -> `Struct(physical of fields)`
| - Other data types will be left unchanged.
|
| Warnings
| --------
| The physical representations are an implementation detail
| and not guaranteed to be stable.
|
| Examples
| --------
| Replicating the pandas
| `pd.Series.factorize
| <https://pandas.pydata.org/docs/reference/api/pandas.Series.factorize.html>`_
| method.
|
| >>> s = pl.Series("values", ["a", None, "x", "a"])
| >>> s.cast(pl.Categorical).to_physical()
| shape: (4,)
| Series: 'values' [u32]
| [
| 0
| null
| 1
| 0
| ]
|
| to_torch(self) -> 'torch.Tensor'
| Convert this Series to a PyTorch Tensor.
|
| .. versionadded:: 0.20.23
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| Notes
| -----
| PyTorch tensors do not support UInt16, UInt32, or UInt64; these dtypes
| will be automatically cast to Int32, Int64, and Int64, respectively.
|
| Examples
| --------
| >>> s = pl.Series("x", [1, 0, 1, 2, 0], dtype=pl.UInt8)
| >>> s.to_torch()
| tensor([1, 0, 1, 2, 0], dtype=torch.uint8)
| >>> s = pl.Series("x", [5.5, -10.0, 2.5], dtype=pl.Float32)
| >>> s.to_torch()
| tensor([ 5.5000, -10.0000, 2.5000])
|
| top_k(self, k: 'int' = 5) -> 'Series'
| Return the `k` largest elements.
|
| Non-null elements are always preferred over null elements. The output is
| not guaranteed to be in any particular order, call :func:`sort` after
| this function if you wish the output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n)
|
| Parameters
| ----------
| k
| Number of elements to return.
|
| See Also
| --------
| top_k_by
| bottom_k
| bottom_k_by
|
| Examples
| --------
| >>> s = pl.Series("a", [2, 5, 1, 4, 3])
| >>> s.top_k(3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 5
| 4
| 3
| ]
|
| top_k_by(self, by: 'IntoExpr | Iterable[IntoExpr]', k: 'int' = 5, *, reverse: 'bool | Sequence[bool]' = False) -> 'Series'
| Return the `k` largest elements of the `by` column.
|
| Non-null elements are always preferred over null elements, regardless of
| the value of `reverse`. The output is not guaranteed to be in any
| particular order, call :func:`sort` after this function if you wish the
| output to be sorted.
|
| This has time complexity:
|
| .. math:: O(n \log{n})
|
| Parameters
| ----------
| by
| Column used to determine the largest elements.
| Accepts expression input. Strings are parsed as column names.
| k
| Number of elements to return.
| reverse
| Consider the `k` smallest elements of the `by` column (instead of the `k`
| largest). This can be specified per column by passing a sequence of
| booleans.
|
| See Also
| --------
| top_k
| bottom_k
| bottom_k_by
|
| Examples
| --------
| >>> s = pl.Series("a", [2, 5, 1, 4, 3])
| >>> s.top_k_by("a", 3)
| shape: (3,)
| Series: 'a' [i64]
| [
| 5
| 4
| 3
| ]
|
| truncate(self, decimals: 'int' = 0) -> 'Series'
| Truncate numeric data toward zero to `decimals` number of decimal places.
|
| Parameters
| ----------
| decimals
| Number of decimal places to truncate to.
|
| Notes
| -----
| * Truncation discards the fractional part beyond the given number of decimals.
| For example, when rounding to 0 decimals 0.25, -0.25, 0.99, and -0.99 will
| all round to 0. When rounding to 1 decimal 1.9999 rounds to 1.9 and -1.9999
| rounds to -1.9. There is no tiebreak behaviour at midpoint values as there
| is with :meth:`round` so 0.5 and -0.5 will also round to 0 when decimals=1.
|
| * This method performs numeric truncation. For truncating temporal
| data (dates/datetimes), use :func:`Series.dt.truncate` instead.
|
| See Also
| --------
| round : Round to a given number of decimals.
| floor : Round down to the nearest integer.
| ceil : Round up to the nearest integer.
|
| Examples
| --------
| >>> s = pl.Series("a", [1.12345, 2.56789, 3.991234])
| >>> s.truncate(2)
| shape: (3,)
| Series: 'a' [f64]
| [
| 1.12
| 2.56
| 3.99
| ]
|
| >>> s = pl.Series("a", [-1.78, 2.56, -3.99])
| >>> s.truncate(0)
| shape: (3,)
| Series: 'a' [f64]
| [
| -1.0
| 2.0
| -3.0
| ]
|
| unique(self, *, maintain_order: 'bool' = False) -> 'Series'
| Get unique elements in series.
|
| `null` is considered to be a unique value for the purposes of this operation.
|
| Parameters
| ----------
| maintain_order
| Maintain order of data. This requires more work.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 2, 3])
| >>> s.unique().sort()
| shape: (3,)
| Series: 'a' [i64]
| [
| 1
| 2
| 3
| ]
|
| unique_counts(self) -> 'Series'
| Return a count of the unique values in the order of appearance.
|
| Examples
| --------
| >>> s = pl.Series("id", ["a", "b", "b", "c", "c", "c"])
| >>> s.unique_counts()
| shape: (3,)
| Series: 'id' [u32]
| [
| 1
| 2
| 3
| ]
|
| upper_bound(self) -> 'Self'
| Return the upper bound of this Series' dtype as a unit Series.
|
| See Also
| --------
| lower_bound : return the lower bound of the given Series' dtype.
|
| Examples
| --------
| >>> s = pl.Series("s", [-1, 0, 1], dtype=pl.Int8)
| >>> s.upper_bound()
| shape: (1,)
| Series: 's' [i8]
| [
| 127
| ]
|
| >>> s = pl.Series("s", [1.0, 2.5, 3.0], dtype=pl.Float64)
| >>> s.upper_bound()
| shape: (1,)
| Series: 's' [f64]
| [
| inf
| ]
|
| value_counts(self, *, sort: 'bool' = False, parallel: 'bool' = False, name: 'str_ | None' = None, normalize: 'bool' = False) -> 'DataFrame'
| Count the occurrences of unique values.
|
| Parameters
| ----------
| sort
| Sort the output by count, in descending order.
| If set to `False` (default), the order is non-deterministic.
| parallel
| Execute the computation in parallel.
|
| .. note::
| This option should likely *not* be enabled in a `group_by` context,
| as the computation will already be parallelized per group.
| name
| Give the resulting count column a specific name; if `normalize` is
| True this defaults to "proportion", otherwise defaults to "count".
| normalize
| If True, the count is returned as the relative frequency of unique
| values normalized to 1.0.
|
| Returns
| -------
| DataFrame
| Columns map the unique values to their count (or proportion).
|
| Examples
| --------
| >>> s = pl.Series("color", ["red", "blue", "red", "green", "blue", "blue"])
| >>> s.value_counts() # doctest: +IGNORE_RESULT
| shape: (3, 2)
| ┌───────┬───────┐
| │ color ┆ count │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═══════╡
| │ red ┆ 2 │
| │ green ┆ 1 │
| │ blue ┆ 3 │
| └───────┴───────┘
|
| Sort the output by count and customize the count column name.
|
| >>> s.value_counts(sort=True, name="n")
| shape: (3, 2)
| ┌───────┬─────┐
| │ color ┆ n │
| │ --- ┆ --- │
| │ str ┆ u32 │
| ╞═══════╪═════╡
| │ blue ┆ 3 │
| │ red ┆ 2 │
| │ green ┆ 1 │
| └───────┴─────┘
|
| Return the count as a relative frequency, normalized to 1.0:
|
| >>> s.value_counts(sort=True, normalize=True, name="fraction")
| shape: (3, 2)
| ┌───────┬──────────┐
| │ color ┆ fraction │
| │ --- ┆ --- │
| │ str ┆ f64 │
| ╞═══════╪══════════╡
| │ blue ┆ 0.5 │
| │ red ┆ 0.333333 │
| │ green ┆ 0.166667 │
| └───────┴──────────┘
|
| var(self, ddof: 'int' = 1) -> 'float | timedelta | None'
| Get variance of this Series.
|
| Parameters
| ----------
| ddof
| “Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
| where N represents the number of elements.
| By default ddof is 1.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.var()
| 1.0
|
| zip_with(self, mask: 'Series', other: 'Series') -> 'Self'
| Take values from self or other based on the given mask.
|
| Where mask evaluates true, take values from self. Where mask evaluates false,
| take values from other.
|
| Parameters
| ----------
| mask
| Boolean Series.
| other
| Series of same type.
|
| Returns
| -------
| Series
|
| Examples
| --------
| >>> s1 = pl.Series([1, 2, 3, 4, 5])
| >>> s2 = pl.Series([5, 4, 3, 2, 1])
| >>> s1.zip_with(s1 < s2, s2)
| shape: (5,)
| Series: '' [i64]
| [
| 1
| 2
| 3
| 2
| 1
| ]
| >>> mask = pl.Series([True, False, True, False, True])
| >>> s1.zip_with(mask, s2)
| shape: (5,)
| Series: '' [i64]
| [
| 1
| 4
| 3
| 2
| 5
| ]
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| arr
| Create an object namespace of all array related methods.
|
| bin
| Create an object namespace of all binary related methods.
|
| cat
| Create an object namespace of all categorical related methods.
|
| dt
| Create an object namespace of all datetime related methods.
|
| dtype
| Get the data type of this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.dtype
| Int64
|
| ext
| Create an object namespace of all extension type related methods.
|
| flags
| Get flags that are set on the Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.flags
| {'SORTED_ASC': False, 'SORTED_DESC': False}
|
| list
| Create an object namespace of all list related methods.
|
| name
| Get the name of this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.name
| 'a'
|
| plot
| Create a plot namespace.
|
| .. warning::
| This functionality is currently considered **unstable**. It may be
| changed at any point without it being considered a breaking change.
|
| .. versionchanged:: 1.6.0
| In prior versions of Polars, HvPlot was the plotting backend. If you would
| like to restore the previous plotting functionality, all you need to do
| is add `import hvplot.polars` at the top of your script and replace
| `df.plot` with `df.hvplot`.
|
| Polars does not implement plotting logic itself, but instead defers to
| Altair:
|
| - `s.plot.hist(**kwargs)`
| is shorthand for
| `alt.Chart(s.to_frame()).mark_bar(tooltip=True).encode(x=alt.X(f'{s.name}:Q', bin=True), y='count()', **kwargs).interactive()`
| - `s.plot.kde(**kwargs)`
| is shorthand for
| `alt.Chart(s.to_frame()).transform_density(s.name, as_=[s.name, 'density']).mark_area(tooltip=True).encode(x=s.name, y='density:Q', **kwargs).interactive()`
| - for any other attribute `attr`, `s.plot.attr(**kwargs)`
| is shorthand for
| `alt.Chart(s.to_frame().with_row_index()).mark_attr(tooltip=True).encode(x='index', y=s.name, **kwargs).interactive()`
|
| For configuration, we suggest reading
| `Chart Configuration <https://altair-viz.github.io/altair-tutorial/notebooks/08-Configuration.html>`_.
| For example, you can:
|
| - Change the width/height/title with ``.properties(width=500, height=350, title="My amazing plot")``.
| - Change the x-axis label rotation with ``.configure_axisX(labelAngle=30)``.
| - Change the opacity of the points in your scatter plot with ``.configure_point(opacity=.5)``.
|
| Examples
| --------
| Histogram:
|
| >>> s = pl.Series([1, 4, 4, 6, 2, 4, 3, 5, 5, 7, 1])
| >>> s.plot.hist() # doctest: +SKIP
|
| KDE plot:
|
| >>> s.plot.kde() # doctest: +SKIP
|
| Line plot:
|
| >>> s.plot.line() # doctest: +SKIP
|
| shape
| Shape of this Series.
|
| Examples
| --------
| >>> s = pl.Series("a", [1, 2, 3])
| >>> s.shape
| (3,)
|
| str
| Create an object namespace of all string related methods.
|
| struct
| Create an object namespace of all struct related methods.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_accessors': 'ClassVar[set_[str_]]', '_s': 'PySeri...
|
| __hash__ = None
DATA
__all__ = ['Series']
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\polars\series\__init__.py
polars.set_random_seed
Python Library Documentation: function set_random_seed in module polars.functions.random
set_random_seed(seed: 'int') -> 'None'
Set the global random seed for Polars.
This random seed is used to determine things such as shuffle ordering.
Parameters
----------
seed
A non-negative integer < 2\ :sup:`64` used to seed the internal global
random number generator.
polars.show_versions
Python Library Documentation: function show_versions in module polars.meta.versions
show_versions() -> 'None'
Print out the version of Polars and its optional dependencies.
Examples
--------
>>> pl.show_versions() # doctest: +SKIP
--------Version info---------
Polars: 0.20.22
Index type: UInt32
Platform: macOS-14.4.1-arm64-arm-64bit
Python: 3.11.8 (main, Feb 6 2024, 21:21:21) [Clang 15.0.0 (clang-1500.1.0.2.5)]
LTS CPU: False
----Optional dependencies----
adbc_driver_manager: 0.11.0
altair: 5.4.0
cloudpickle: 3.0.0
connectorx: 0.3.2
deltalake: 0.17.1
fastexcel: 0.10.4
fsspec: 2023.12.2
gevent: 24.2.1
matplotlib: 3.8.4
numpy: 1.26.4
openpyxl: 3.1.2
pandas: 2.2.2
pyarrow: 16.0.0
pydantic: 2.7.1
pyiceberg: 0.7.1
sqlalchemy: 2.0.29
torch: 2.2.2
xlsx2csv: 0.8.2
xlsxwriter: 3.2.0
polars.sql
Python Library Documentation: function sql in module polars.sql.functions
sql(query: 'str', *, eager: 'bool' = False) -> 'DataFrame | LazyFrame'
Execute a SQL query against frames in the global namespace.
.. versionadded:: 0.20.31
Parameters
----------
query
SQL query to execute.
eager
Automatically collect the result and return a DataFrame instead of a LazyFrame.
Notes
-----
* The Polars SQL engine can operate against Polars DataFrame, LazyFrame, and Series
objects, as well as Pandas DataFrame and Series, PyArrow Table and RecordBatch.
* Additional control over registration and execution behaviour is available
with the :class:`SQLContext` object.
See Also
--------
SQLContext
Examples
--------
>>> lf1 = pl.LazyFrame({"a": [1, 2, 3], "b": [6, 7, 8], "c": ["z", "y", "x"]})
>>> lf2 = pl.LazyFrame({"a": [3, 2, 1], "d": [125, -654, 888]})
Query the LazyFrame using SQL:
>>> lf1.sql("SELECT c, b FROM self WHERE a > 1").collect()
shape: (2, 2)
┌─────┬─────┐
│ c ┆ b │
│ --- ┆ --- │
│ str ┆ i64 │
╞═════╪═════╡
│ y ┆ 7 │
│ x ┆ 8 │
└─────┴─────┘
Join two LazyFrames:
>>> pl.sql(
... '''
... SELECT lf1.*, d
... FROM lf1
... INNER JOIN lf2 USING (a)
... WHERE a > 1 AND b < 8
... '''
... ).collect()
shape: (1, 4)
┌─────┬─────┬─────┬──────┐
│ a ┆ b ┆ c ┆ d │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ i64 │
╞═════╪═════╪═════╪══════╡
│ 2 ┆ 7 ┆ y ┆ -654 │
└─────┴─────┴─────┴──────┘
Apply SQL transforms and subsequently filter natively (you can freely mix SQL and
native operations):
>>> pl.sql(
... query='''
... SELECT
... a,
... (a % 2 == 0) AS a_is_even,
... (b::float4 / 2) AS "b/2",
... CONCAT_WS(':', c, c, c) AS c_c_c
... FROM lf1
... ORDER BY a
... ''',
... ).filter(~pl.col("c_c_c").str.starts_with("x")).collect()
shape: (2, 4)
┌─────┬───────────┬─────┬───────┐
│ a ┆ a_is_even ┆ b/2 ┆ c_c_c │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ bool ┆ f32 ┆ str │
╞═════╪═══════════╪═════╪═══════╡
│ 1 ┆ false ┆ 3.0 ┆ z:z:z │
│ 2 ┆ true ┆ 3.5 ┆ y:y:y │
└─────┴───────────┴─────┴───────┘
Join polars LazyFrame with a pandas DataFrame and a pyarrow Table:
>>> import pandas as pd
>>> import pyarrow as pa
>>> pl_frame = lf1
>>> pd_frame = pd.DataFrame({"a": [2, 3, 4], "d": [-0.5, 0.0, 0.5]})
>>> pa_table = pa.Table.from_arrays(
... [pa.array([1, 2, 3]), pa.array(["x", "y", "z"])],
... names=["a", "e"],
... )
>>> pl.sql(
... query='''
... SELECT pl_frame.*, d, e
... FROM pl_frame
... JOIN pd_frame USING(a)
... JOIN pa_table USING(a)
... ''',
... ).collect()
shape: (2, 5)
┌─────┬─────┬─────┬──────┬─────┐
│ a ┆ b ┆ c ┆ d ┆ e │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ f64 ┆ str │
╞═════╪═════╪═════╪══════╪═════╡
│ 2 ┆ 7 ┆ y ┆ -0.5 ┆ y │
│ 3 ┆ 8 ┆ x ┆ 0.0 ┆ z │
└─────┴─────┴─────┴──────┴─────┘
polars.sql_expr
Python Library Documentation: function sql_expr in module polars.functions.lazy
sql_expr(sql: 'str | Sequence[str]') -> 'Expr | list[Expr]'
Parse one or more SQL expressions to Polars expression(s).
Parameters
----------
sql
One or more SQL expressions.
Examples
--------
Parse a single SQL expression:
>>> df = pl.DataFrame({"a": [2, 1]})
>>> expr = pl.sql_expr("MAX(a)")
>>> df.select(expr)
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 2 │
└─────┘
Parse multiple SQL expressions:
>>> df.with_columns(
... *pl.sql_expr(["POWER(a,a) AS a_a", "CAST(a AS TEXT) AS a_txt"]),
... )
shape: (2, 3)
┌─────┬─────┬───────┐
│ a ┆ a_a ┆ a_txt │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str │
╞═════╪═════╪═══════╡
│ 2 ┆ 4 ┆ 2 │
│ 1 ┆ 1 ┆ 1 │
└─────┴─────┴───────┘
polars.std
Python Library Documentation: function std in module polars.functions.lazy
std(column: 'str', ddof: 'int' = 1) -> 'Expr'
Get the standard deviation.
This function is syntactic sugar for `pl.col(column).std(ddof)`.
Parameters
----------
column
Column name.
ddof
“Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
where N represents the number of elements.
By default ddof is 1.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.std("a"))
shape: (1, 1)
┌──────────┐
│ a │
│ --- │
│ f64 │
╞══════════╡
│ 3.605551 │
└──────────┘
>>> df["a"].std()
3.605551275463989
polars.string_cache
Error rendering doc: unsupported operand type(s) for +: 'NoneType' and 'str'
polars.struct
Python Library Documentation: function struct in module polars.functions.as_datatype
struct(*exprs: 'IntoExpr | Iterable[IntoExpr]', schema: 'SchemaDict | None' = None, eager: 'bool' = False, **named_exprs: 'IntoExpr') -> 'Expr | Series'
Collect columns into a struct column.
Parameters
----------
*exprs
Column(s) to collect into a struct column, specified as positional arguments.
Accepts expression input. Strings are parsed as column names,
other non-expression inputs are parsed as literals.
schema
Optional schema that explicitly defines the struct field dtypes. If no columns
or expressions are provided, schema keys are used to define columns.
eager
Evaluate immediately and return a `Series`. If set to `False` (default),
return an expression instead.
**named_exprs
Additional columns to collect into the struct column, specified as keyword
arguments. The columns will be renamed to the keyword used.
Examples
--------
Collect all columns of a dataframe into a struct by passing `pl.all()`.
>>> df = pl.DataFrame(
... {
... "int": [1, 2],
... "str": ["a", "b"],
... "bool": [True, None],
... "list": [[1, 2], [3]],
... }
... )
>>> df.select(pl.struct(pl.all()).alias("my_struct"))
shape: (2, 1)
┌─────────────────────┐
│ my_struct │
│ --- │
│ struct[4] │
╞═════════════════════╡
│ {1,"a",true,[1, 2]} │
│ {2,"b",null,[3]} │
└─────────────────────┘
Collect selected columns into a struct by either passing a list of columns, or by
specifying each column as a positional argument.
>>> df.select(pl.struct("int", False).alias("my_struct"))
shape: (2, 1)
┌───────────┐
│ my_struct │
│ --- │
│ struct[2] │
╞═══════════╡
│ {1,false} │
│ {2,false} │
└───────────┘
Use keyword arguments to easily name each struct field.
>>> df.select(pl.struct(p="int", q="bool").alias("my_struct")).schema
Schema({'my_struct': Struct({'p': Int64, 'q': Boolean})})
polars.struct_with_fields
Python Library Documentation: function struct_with_fields in module polars.functions.datatype
struct_with_fields(mapping: 'Mapping[str, PolarsDataType | pl.DataTypeExpr]') -> 'pl.DataTypeExpr'
Create a new datatype expression that represents a Struct datatype.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
polars.sum
Python Library Documentation: function sum in module polars.functions.aggregation.vertical
sum(*names: 'str') -> 'Expr'
Sum all values.
Syntactic sugar for `col(name).sum()`.
Parameters
----------
*names
Name(s) of the columns to use in the aggregation.
Notes
-----
If there are no non-null values, then the output is `0`.
If you would prefer empty sums to return `None`, you can
use `pl.when(pl.col(name).count()>0).then(pl.sum(name))` instead
of `pl.sum(name)`.
See Also
--------
sum_horizontal
Examples
--------
Sum a column.
>>> df = pl.DataFrame(
... {
... "a": [1, 2],
... "b": [3, 4],
... "c": [5, 6],
... }
... )
>>> df.select(pl.sum("a"))
shape: (1, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 3 │
└─────┘
Sum multiple columns.
>>> df.select(pl.sum("a", "c"))
shape: (1, 2)
┌─────┬─────┐
│ a ┆ c │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 3 ┆ 11 │
└─────┴─────┘
>>> df.select(pl.sum("^.*[bc]$"))
shape: (1, 2)
┌─────┬─────┐
│ b ┆ c │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 7 ┆ 11 │
└─────┴─────┘
polars.sum_horizontal
Python Library Documentation: function sum_horizontal in module polars.functions.aggregation.horizontal
sum_horizontal(*exprs: 'IntoExpr | Iterable[IntoExpr]', ignore_nulls: 'bool' = True) -> 'Expr'
Sum all values horizontally across columns.
Parameters
----------
*exprs
Column(s) to use in the aggregation. Accepts expression input. Strings are
parsed as column names, other non-expression inputs are parsed as literals.
ignore_nulls
Ignore null values (default).
If set to `False`, any null value in the input will lead to a null output.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, None],
... "c": ["x", "y", "z"],
... }
... )
>>> df.with_columns(sum=pl.sum_horizontal("a", "b"))
shape: (3, 4)
┌─────┬──────┬─────┬─────┐
│ a ┆ b ┆ c ┆ sum │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ i64 │
╞═════╪══════╪═════╪═════╡
│ 1 ┆ 4 ┆ x ┆ 5 │
│ 8 ┆ 5 ┆ y ┆ 13 │
│ 3 ┆ null ┆ z ┆ 3 │
└─────┴──────┴─────┴─────┘
polars.tail
Python Library Documentation: function tail in module polars.functions.lazy
tail(column: 'str', n: 'int' = 10) -> 'Expr'
Get the last `n` rows.
This function is syntactic sugar for `pl.col(column).tail(n)`.
Parameters
----------
column
Column name.
n
Number of rows to return.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... }
... )
>>> df.select(pl.tail("a"))
shape: (3, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 1 │
│ 8 │
│ 3 │
└─────┘
>>> df.select(pl.tail("a", 2))
shape: (2, 1)
┌─────┐
│ a │
│ --- │
│ i64 │
╞═════╡
│ 8 │
│ 3 │
└─────┘
polars.thread_pool_size
Python Library Documentation: function thread_pool_size in module polars.meta.thread_pool
thread_pool_size() -> 'int'
Return the number of threads in the Polars thread pool.
Notes
-----
The thread pool size can be overridden by setting the `POLARS_MAX_THREADS`
environment variable before process start. The thread pool is not behind a
lock, so it cannot be modified once set. A reasonable use case for this might
be temporarily limiting the number of threads before importing Polars in a
PySpark UDF or similar context. Otherwise, it is strongly recommended not to
override this value as it will be set automatically by the engine.
Examples
--------
>>> pl.thread_pool_size() # doctest: +SKIP
16
polars.threadpool_size
Python Library Documentation: function threadpool_size in module polars.meta.thread_pool
threadpool_size() -> 'int'
Return the number of threads in the Polars thread pool.
.. deprecated:: 0.20.7
This function has been renamed to :func:`thread_pool_size`.
polars.time
Python Library Documentation: function time_ in module polars.functions.as_datatype
time_(hour: 'Expr | str | int | None' = None, minute: 'Expr | str | int | None' = None, second: 'Expr | str | int | None' = None, microsecond: 'Expr | str | int | None' = None) -> 'Expr'
Create a Polars literal expression of type Time.
Parameters
----------
hour
column or literal, ranging from 0-23.
minute
column or literal, ranging from 0-59.
second
column or literal, ranging from 0-59.
microsecond
column or literal, ranging from 0-999999.
Returns
-------
Expr
Expression of data type :class:`Date`.
Examples
--------
>>> df = pl.DataFrame(
... {
... "hour": [12, 13, 14],
... "minute": [15, 30, 45],
... }
... )
>>> df.with_columns(pl.time(pl.col("hour"), pl.col("minute")))
shape: (3, 3)
┌──────┬────────┬──────────┐
│ hour ┆ minute ┆ time │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ time │
╞══════╪════════╪══════════╡
│ 12 ┆ 15 ┆ 12:15:00 │
│ 13 ┆ 30 ┆ 13:30:00 │
│ 14 ┆ 45 ┆ 14:45:00 │
└──────┴────────┴──────────┘
polars.time_range
Python Library Documentation: function time_range in module polars.functions.range.time_range
time_range(start: 'time | IntoExprColumn | None' = None, end: 'time | IntoExprColumn | None' = None, interval: 'str | timedelta' = '1h', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Series | Expr'
Generate a time range.
Parameters
----------
start
Lower bound of the time range.
If omitted, defaults to `time(0,0,0,0)`.
end
Upper bound of the time range.
If omitted, defaults to `time(23,59,59,999999)`.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type `:class:Time`.
Notes
-----
`interval` is created according to the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
See Also
--------
time_ranges : Create a column of time ranges.
Examples
--------
>>> from datetime import time, timedelta
>>> pl.time_range(
... start=time(14, 0),
... interval=timedelta(hours=3, minutes=15),
... eager=True,
... ).alias("time")
shape: (4,)
Series: 'time' [time]
[
14:00:00
17:15:00
20:30:00
23:45:00
]
polars.time_ranges
Python Library Documentation: function time_ranges in module polars.functions.range.time_range
time_ranges(start: 'time | IntoExprColumn | None' = None, end: 'time | IntoExprColumn | None' = None, interval: 'str | timedelta' = '1h', *, closed: 'ClosedInterval' = 'both', eager: 'bool' = False) -> 'Series | Expr'
Create a column of time ranges.
Parameters
----------
start
Lower bound of the time range.
If omitted, defaults to `time(0, 0, 0, 0)`.
end
Upper bound of the time range.
If omitted, defaults to `time(23, 59, 59, 999999)`.
interval
Interval of the range periods, specified as a Python `timedelta` object
or using the Polars duration string language (see "Notes" section below).
closed : {'both', 'left', 'right', 'none'}
Define which sides of the range are closed (inclusive).
eager
Evaluate immediately and return a `Series`.
If set to `False` (default), return an expression instead.
Returns
-------
Expr or Series
Column of data type `List(Time)`.
Notes
-----
`interval` is created according to the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 calendar day)
- 1w (1 calendar week)
- 1mo (1 calendar month)
- 1q (1 calendar quarter)
- 1y (1 calendar year)
Or combine them:
"3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
By "calendar day", we mean the corresponding time on the next day (which may
not be 24 hours, due to daylight savings). Similarly for "calendar week",
"calendar month", "calendar quarter", and "calendar year".
See Also
--------
time_range : Generate a single time range.
Examples
--------
>>> from datetime import time
>>> df = pl.DataFrame(
... {
... "start": [time(9, 0), time(10, 0)],
... "end": time(11, 0),
... }
... )
>>> df.with_columns(time_range=pl.time_ranges("start", "end"))
shape: (2, 3)
┌──────────┬──────────┬────────────────────────────────┐
│ start ┆ end ┆ time_range │
│ --- ┆ --- ┆ --- │
│ time ┆ time ┆ list[time] │
╞══════════╪══════════╪════════════════════════════════╡
│ 09:00:00 ┆ 11:00:00 ┆ [09:00:00, 10:00:00, 11:00:00] │
│ 10:00:00 ┆ 11:00:00 ┆ [10:00:00, 11:00:00] │
└──────────┴──────────┴────────────────────────────────┘
polars.union
Python Library Documentation: function union in module polars.functions.eager
union(items: 'Iterable[PolarsType]', *, how: 'ConcatMethod' = 'vertical', strict: 'bool | None' = None) -> 'PolarsType'
Combine multiple DataFrames, LazyFrames, or Series into a single object.
.. warning::
This function does not guarantee any specific ordering of rows in the result.
If you need predictable row ordering, use `pl.concat()` instead.
Parameters
----------
items
DataFrames, LazyFrames, or Series to concatenate.
how : {'vertical', 'vertical_relaxed', 'diagonal', 'diagonal_relaxed', 'horizontal', 'horizontal_extend', 'align', 'align_full', 'align_inner', 'align_left', 'align_right'}
Note that `Series` only support the `vertical` strategy.
* vertical: Applies multiple `vstack` operations.
* vertical_relaxed: Same as `vertical`, but additionally coerces columns to
their common supertype *if* they are mismatched (eg: Int32 → Int64).
* diagonal: Finds a union between the column schemas and fills missing column
values with `null`.
* diagonal_relaxed: Same as `diagonal`, but additionally coerces columns to
their common supertype *if* they are mismatched (eg: Int32 → Int64).
* horizontal: Stacks Series from DataFrames horizontally. All input frames
must have the same height; raises a `ShapeError` otherwise.
* horizontal_extend: Same as `horizontal`, but pads shorter frames with `null`
if the lengths don't match.
* align, align_full, align_left, align_right: Combines frames horizontally,
auto-determining the common key columns and aligning rows using the same
logic as `align_frames` (note that "align" is an alias for "align_full").
The "align" strategy determines the type of join used to align the frames,
equivalent to the "how" parameter on `align_frames`. Note that the common
join columns are automatically coalesced, but other column collisions
will raise an error (if you need more control over this you should use
a suitable `join` method directly).
strict
When how=`horizontal`, require all DataFrames to be the same height, raising an error if not.
.. deprecated:: 1.42.1
Use `how='horizontal'` (equal heights) or
`how='horizontal_extend'` (pad with null) instead.
Examples
--------
>>> df1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df2 = pl.DataFrame({"a": [2], "b": [4]})
>>> pl.union([df1, df2]) # default is 'vertical' strategy
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 3 │
│ 2 ┆ 4 │
└─────┴─────┘
>>> df1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df2 = pl.DataFrame({"a": [2.5], "b": [4]})
>>> pl.union([df1, df2], how="vertical_relaxed") # 'a' coerced into f64
shape: (2, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ f64 ┆ i64 │
╞═════╪═════╡
│ 1.0 ┆ 3 │
│ 2.5 ┆ 4 │
└─────┴─────┘
>>> df_h1 = pl.DataFrame({"l1": [1, 2], "l2": [3, 4]})
>>> df_h2 = pl.DataFrame({"r1": [5, 6], "r2": [7, 8], "r3": [9, 10]})
>>> pl.union([df_h1, df_h2], how="horizontal_extend")
shape: (2, 5)
┌─────┬─────┬─────┬─────┬─────┐
│ l1 ┆ l2 ┆ r1 ┆ r2 ┆ r3 │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 5 ┆ 7 ┆ 9 │
│ 2 ┆ 4 ┆ 6 ┆ 8 ┆ 10 │
└─────┴─────┴─────┴─────┴─────┘
The "diagonal" strategy allows for some frames to have missing columns,
the values for which are filled with `null`:
>>> df_d1 = pl.DataFrame({"a": [1], "b": [3]})
>>> df_d2 = pl.DataFrame({"a": [2], "c": [4]})
>>> pl.union([df_d1, df_d2], how="diagonal")
shape: (2, 3)
┌─────┬──────┬──────┐
│ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╡
│ 1 ┆ 3 ┆ null │
│ 2 ┆ null ┆ 4 │
└─────┴──────┴──────┘
The "align" strategies require at least one common column to align on:
>>> df_a1 = pl.DataFrame({"id": [1, 2], "x": [3, 4]})
>>> df_a2 = pl.DataFrame({"id": [2, 3], "y": [5, 6]})
>>> df_a3 = pl.DataFrame({"id": [1, 3], "z": [7, 8]})
>>> pl.union([df_a1, df_a2, df_a3], how="align") # equivalent to "align_full"
shape: (3, 4)
┌─────┬──────┬──────┬──────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╪══════╡
│ 1 ┆ 3 ┆ null ┆ 7 │
│ 2 ┆ 4 ┆ 5 ┆ null │
│ 3 ┆ null ┆ 6 ┆ 8 │
└─────┴──────┴──────┴──────┘
>>> pl.union([df_a1, df_a2, df_a3], how="align_left")
shape: (2, 4)
┌─────┬─────┬──────┬──────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪══════╪══════╡
│ 1 ┆ 3 ┆ null ┆ 7 │
│ 2 ┆ 4 ┆ 5 ┆ null │
└─────┴─────┴──────┴──────┘
>>> pl.union([df_a1, df_a2, df_a3], how="align_right")
shape: (2, 4)
┌─────┬──────┬──────┬─────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪══════╪══════╪═════╡
│ 1 ┆ null ┆ null ┆ 7 │
│ 3 ┆ null ┆ 6 ┆ 8 │
└─────┴──────┴──────┴─────┘
>>> pl.union([df_a1, df_a2, df_a3], how="align_inner")
shape: (0, 4)
┌─────┬─────┬─────┬─────┐
│ id ┆ x ┆ y ┆ z │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
└─────┴─────┴─────┴─────┘
polars.unregister_extension_type
Python Library Documentation: function unregister_extension_type in module polars.datatypes.extension
unregister_extension_type(ext_name: 'str') -> 'None'
Unregister the extension type for the given extension name.
.. warning::
This functionality is currently considered **unstable**. It may be
changed at any point without it being considered a breaking change.
polars.using_string_cache
Python Library Documentation: function using_string_cache in module polars.string_cache
using_string_cache() -> 'bool'
Always returns true.
.. deprecated:: 1.41.0
The string cache was used to maintain the mapping for the Categorical
dtype, this is now done through ``pl.Categories``.
polars.var
Python Library Documentation: function var in module polars.functions.lazy
var(column: 'str', ddof: 'int' = 1) -> 'Expr'
Get the variance.
This function is syntactic sugar for `pl.col(column).var(ddof)`.
Parameters
----------
column
Column name.
ddof
“Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof,
where N represents the number of elements.
By default ddof is 1.
Examples
--------
>>> df = pl.DataFrame(
... {
... "a": [1, 8, 3],
... "b": [4, 5, 2],
... "c": ["foo", "bar", "foo"],
... },
... )
>>> df.select(pl.var("a"))
shape: (1, 1)
┌──────┐
│ a │
│ --- │
│ f64 │
╞══════╡
│ 13.0 │
└──────┘
>>> df["a"].var()
13.0
polars.when
Python Library Documentation: function when in module polars.functions.whenthen
when(*predicates: 'IntoExprColumn | Iterable[IntoExprColumn] | bool', **constraints: 'Any') -> 'pl.When'
Start a `when-then-otherwise` expression.
Always initiated by a `pl.when().then()`., and optionally followed by chaining one
or more `.when().then()` statements.
An optional `.otherwise()` can be appended at the end. If not declared, a default
of `.otherwise(None)` is used.
Similar to :func:`coalesce`, the value from the first condition that
evaluates to True will be picked.
If all conditions are False, the `otherwise` value is picked.
Parameters
----------
predicates
Condition(s) that must be met in order to apply the subsequent statement.
Accepts one or more boolean expressions, which are implicitly combined with
`&`.
constraints
Apply conditions as `col_name = value` keyword arguments that are treated as
equality matches, such as `x = 123`. As with the predicates parameter, multiple
conditions are implicitly combined using `&`.
Warnings
--------
Polars computes all expressions passed to `when-then-otherwise` in parallel and
filters afterwards. This means each expression must be valid on its own, regardless
of the conditions in the `when-then-otherwise` chain.
Notes
-----
* String inputs e.g. `when("string")`, `then("string")` or `otherwise("string")`
are parsed as column names. :func:`lit` can be used to create string values.
* The expression output name is taken from the first `then` statement. It is
not affected by `predicates`, nor by `constraints`.
Examples
--------
Below we add a column with the value 1, where column "foo" > 2 and the value
1 + column "bar" where it isn't.
>>> df = pl.DataFrame({"foo": [1, 3, 4], "bar": [3, 4, 0]})
>>> df.with_columns(
... pl.when(pl.col.foo > 2).then(1).otherwise(1 + pl.col.bar).alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 4 │
│ 3 ┆ 4 ┆ 1 │
│ 4 ┆ 0 ┆ 1 │
└─────┴─────┴─────┘
Note that `when-then` always executes all expressions.
The results are folded left to right, picking the `then` value from the first `when`
condition that is True.
If no `when` condition is True the `otherwise` value is picked.
>>> df.with_columns(
... when = pl.col.foo > 2,
... then = 1,
... otherwise = 1 + pl.col.bar
... ).with_columns(
... pl.when("when").then("then").otherwise("otherwise").alias("val")
... )
shape: (3, 6)
┌─────┬─────┬───────┬──────┬───────────┬─────┐
│ foo ┆ bar ┆ when ┆ then ┆ otherwise ┆ val │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool ┆ i32 ┆ i64 ┆ i64 │
╞═════╪═════╪═══════╪══════╪═══════════╪═════╡
│ 1 ┆ 3 ┆ false ┆ 1 ┆ 4 ┆ 4 │
│ 3 ┆ 4 ┆ true ┆ 1 ┆ 5 ┆ 1 │
│ 4 ┆ 0 ┆ true ┆ 1 ┆ 1 ┆ 1 │
└─────┴─────┴───────┴──────┴───────────┴─────┘
Note that in regular Polars usage, a single string is parsed as a column name.
>>> df.with_columns(
... when = pl.col.foo > 2,
... then = "foo",
... otherwise = "bar"
... )
shape: (3, 5)
┌─────┬─────┬───────┬──────┬───────────┐
│ foo ┆ bar ┆ when ┆ then ┆ otherwise │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool ┆ i64 ┆ i64 │
╞═════╪═════╪═══════╪══════╪═══════════╡
│ 1 ┆ 3 ┆ false ┆ 1 ┆ 3 │
│ 3 ┆ 4 ┆ true ┆ 3 ┆ 4 │
│ 4 ┆ 0 ┆ true ┆ 4 ┆ 0 │
└─────┴─────┴───────┴──────┴───────────┘
For consistency, `when-then` behaves in the same way.
>>> df.with_columns(
... pl.when(pl.col.foo > 2).then("foo").otherwise("bar").alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 3 │
│ 3 ┆ 4 ┆ 3 │
│ 4 ┆ 0 ┆ 4 │
└─────┴─────┴─────┘
:func:`lit` can be used to create string values.
>>> df.with_columns(
... pl.when(pl.col.foo > 2)
... .then(pl.lit("foo"))
... .otherwise(pl.lit("bar"))
... .alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ bar │
│ 3 ┆ 4 ┆ foo │
│ 4 ┆ 0 ┆ foo │
└─────┴─────┴─────┘
Multiple `when-then` statements can be chained.
>>> df.with_columns(
... pl.when(pl.col.foo > 2)
... .then(1)
... .when(pl.col.bar > 2)
... .then(4)
... .otherwise(-1)
... .alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i32 │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ 4 │
│ 3 ┆ 4 ┆ 1 │
│ 4 ┆ 0 ┆ 1 │
└─────┴─────┴─────┘
In the case of `foo=3` and `bar=4`, both conditions are True but the first value
(i.e. 1) is picked.
>>> df.with_columns(
... when1 = pl.col.foo > 2,
... then1 = 1,
... when2 = pl.col.bar > 2,
... then2 = 4,
... otherwise = -1
... )
shape: (3, 7)
┌─────┬─────┬───────┬───────┬───────┬───────┬───────────┐
│ foo ┆ bar ┆ when1 ┆ then1 ┆ when2 ┆ then2 ┆ otherwise │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool ┆ i32 ┆ bool ┆ i32 ┆ i32 │
╞═════╪═════╪═══════╪═══════╪═══════╪═══════╪═══════════╡
│ 1 ┆ 3 ┆ false ┆ 1 ┆ true ┆ 4 ┆ -1 │
│ 3 ┆ 4 ┆ true ┆ 1 ┆ true ┆ 4 ┆ -1 │
│ 4 ┆ 0 ┆ true ┆ 1 ┆ false ┆ 4 ┆ -1 │
└─────┴─────┴───────┴───────┴───────┴───────┴───────────┘
The `otherwise` statement is optional and defaults to `.otherwise(None)`
if not given.
This idiom is commonly used to null out values.
>>> df.with_columns(pl.when(pl.col.foo == 3).then("bar"))
shape: (3, 2)
┌─────┬──────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪══════╡
│ 1 ┆ null │
│ 3 ┆ 4 │
│ 4 ┆ null │
└─────┴──────┘
`when` accepts keyword arguments as shorthand for equality conditions.
>>> df.with_columns(pl.when(foo=3).then("bar"))
shape: (3, 2)
┌─────┬──────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪══════╡
│ 1 ┆ null │
│ 3 ┆ 4 │
│ 4 ┆ null │
└─────┴──────┘
Multiple predicates passed to `when` are combined with `&`
>>> df.with_columns(
... pl.when(pl.col.foo > 2, pl.col.bar < 3) # when((pred1) & (pred2))
... .then(pl.lit("Yes"))
... .otherwise(pl.lit("No"))
... .alias("val")
... )
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str │
╞═════╪═════╪═════╡
│ 1 ┆ 3 ┆ No │
│ 3 ┆ 4 ┆ No │
│ 4 ┆ 0 ┆ Yes │
└─────┴─────┴─────┘
It could also be thought of as an implicit :func:`all_horizontal` being present.
>>> df.with_columns(
... when = pl.all_horizontal(pl.col.foo > 2, pl.col.bar < 3)
... )
shape: (3, 3)
┌─────┬─────┬───────┐
│ foo ┆ bar ┆ when │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool │
╞═════╪═════╪═══════╡
│ 1 ┆ 3 ┆ false │
│ 3 ┆ 4 ┆ false │
│ 4 ┆ 0 ┆ true │
└─────┴─────┴───────┘
Structs can be used as a way to return multiple values.
Here we swap the "foo" and "bar" values when "foo" is greater than 2.
>>> df.with_columns(
... pl.when(pl.col.foo > 2)
... .then(pl.struct(foo="bar", bar="foo"))
... .otherwise(pl.struct("foo", "bar"))
... .struct.unnest()
... )
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1 ┆ 3 │
│ 4 ┆ 3 │
│ 0 ┆ 4 │
└─────┴─────┘
The struct fields are given the same name as the target columns, which are then
unnested.
>>> df.with_columns(
... when = pl.col.foo > 2,
... then = pl.struct(foo="bar", bar="foo"),
... otherwise = pl.struct("foo", "bar")
... )
shape: (3, 5)
┌─────┬─────┬───────┬───────────┬───────────┐
│ foo ┆ bar ┆ when ┆ then ┆ otherwise │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool ┆ struct[2] ┆ struct[2] │
╞═════╪═════╪═══════╪═══════════╪═══════════╡
│ 1 ┆ 3 ┆ false ┆ {3,1} ┆ {1,3} │
│ 3 ┆ 4 ┆ true ┆ {4,3} ┆ {3,4} │
│ 4 ┆ 0 ┆ true ┆ {0,4} ┆ {4,0} │
└─────┴─────┴───────┴───────────┴───────────┘
The output name of a `when-then` expression comes from the first `then` branch.
Here we try to set all columns to 0 if any column contains a value less than 2.
>>> df.with_columns( # doctest: +SKIP
... pl.when(pl.any_horizontal(pl.all() < 2))
... .then(0)
... .otherwise(pl.all())
... )
# ComputeError: the name 'literal' passed to `LazyFrame.with_columns` is duplicate
:meth:`.name.keep` could be used to give preference to the column expression.
>>> df.with_columns(
... pl.when(pl.any_horizontal(pl.all() < 2))
... .then(0)
... .otherwise(pl.all())
... .name.keep()
... )
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 0 ┆ 0 │
│ 3 ┆ 4 │
│ 0 ┆ 0 │
└─────┴─────┘
The logic could also be changed to move the column expression inside `then`.
>>> df.with_columns(
... pl.when(pl.any_horizontal(pl.all() < 2).not_())
... .then(pl.all())
... .otherwise(0)
... )
shape: (3, 2)
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 0 ┆ 0 │
│ 3 ┆ 4 │
│ 0 ┆ 0 │
└─────┴─────┘
polars.wrap_df
Python Library Documentation: function wrap_df in module polars._utils.wrap wrap_df(df: 'PyDataFrame') -> 'DataFrame'
polars.wrap_s
Python Library Documentation: function wrap_s in module polars._utils.wrap wrap_s(s: 'PySeries') -> 'Series'
polars.zeros
Python Library Documentation: function zeros in module polars.functions.repeat
zeros(n: 'int | Expr', dtype: 'PolarsDataType' = Float64, *, eager: 'bool' = False) -> 'Expr | Series'
Construct a column of length `n` filled with zeros.
This is syntactic sugar for the `repeat` function.
Parameters
----------
n
Length of the resulting column.
dtype
Data type of the resulting column. Defaults to Float64.
eager
Evaluate immediately and return a `Series`. If set to `False`,
return an expression instead.
Notes
-----
If you want to construct a column in lazy mode and do not need a pre-determined
length, use :func:`lit` instead.
See Also
--------
repeat
lit
Examples
--------
>>> pl.zeros(3, pl.Int8, eager=True)
shape: (3,)
Series: 'zeros' [i8]
[
0
0
0
]