datasets API Reference
Full documentation for datasets (5.0.1)
datasets.Array2D
Python Library Documentation: class Array2D in module datasets.features.features
class Array2D(_ArrayXD)
| Array2D(shape: tuple, dtype: str, id: Optional[str] = None) -> None
|
| Create a two-dimensional array.
|
| Args:
| shape (`tuple`):
| Size of each dimension.
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'x': Array2D(shape=(1, 3), dtype='int32')})
| ```
|
| Method resolution order:
| Array2D
| _ArrayXD
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, shape: tuple, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('shape', 'dtype', 'id')
|
| id = None
|
| ----------------------------------------------------------------------
| Methods inherited from _ArrayXD:
|
| __call__(self)
| Call self as a function.
|
| __post_init__(self)
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _ArrayXD:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
datasets.Array3D
Python Library Documentation: class Array3D in module datasets.features.features
class Array3D(_ArrayXD)
| Array3D(shape: tuple, dtype: str, id: Optional[str] = None) -> None
|
| Create a three-dimensional array.
|
| Args:
| shape (`tuple`):
| Size of each dimension.
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'x': Array3D(shape=(1, 2, 3), dtype='int32')})
| ```
|
| Method resolution order:
| Array3D
| _ArrayXD
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, shape: tuple, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('shape', 'dtype', 'id')
|
| id = None
|
| ----------------------------------------------------------------------
| Methods inherited from _ArrayXD:
|
| __call__(self)
| Call self as a function.
|
| __post_init__(self)
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _ArrayXD:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
datasets.Array4D
Python Library Documentation: class Array4D in module datasets.features.features
class Array4D(_ArrayXD)
| Array4D(shape: tuple, dtype: str, id: Optional[str] = None) -> None
|
| Create a four-dimensional array.
|
| Args:
| shape (`tuple`):
| Size of each dimension.
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'x': Array4D(shape=(1, 2, 2, 3), dtype='int32')})
| ```
|
| Method resolution order:
| Array4D
| _ArrayXD
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, shape: tuple, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('shape', 'dtype', 'id')
|
| id = None
|
| ----------------------------------------------------------------------
| Methods inherited from _ArrayXD:
|
| __call__(self)
| Call self as a function.
|
| __post_init__(self)
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _ArrayXD:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
datasets.Array5D
Python Library Documentation: class Array5D in module datasets.features.features
class Array5D(_ArrayXD)
| Array5D(shape: tuple, dtype: str, id: Optional[str] = None) -> None
|
| Create a five-dimensional array.
|
| Args:
| shape (`tuple`):
| Size of each dimension.
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'x': Array5D(shape=(1, 2, 2, 3, 3), dtype='int32')})
| ```
|
| Method resolution order:
| Array5D
| _ArrayXD
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, shape: tuple, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('shape', 'dtype', 'id')
|
| id = None
|
| ----------------------------------------------------------------------
| Methods inherited from _ArrayXD:
|
| __call__(self)
| Call self as a function.
|
| __post_init__(self)
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _ArrayXD:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
datasets.ArrowBasedBuilder
Python Library Documentation: class ArrowBasedBuilder in module datasets.builder
class ArrowBasedBuilder(DatasetBuilder)
| ArrowBasedBuilder(cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
|
| Base class for datasets with data generation based on Arrow loading functions (CSV/JSON/Parquet).
|
| Method resolution order:
| ArrowBasedBuilder
| DatasetBuilder
| builtins.object
|
| Methods inherited from DatasetBuilder:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setstate__(self, d)
|
| as_dataset(self, split: Union[str, datasets.splits.Split, list[str], list[datasets.splits.Split], NoneType] = None, in_memory=False) -> Union[datasets.arrow_dataset.Dataset, datasets.dataset_dict.DatasetDict]
| Return a Dataset for the specified split.
|
| Args:
| split (`datasets.Split`):
| Which subset of the data to return.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the
| downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
|
| Returns:
| datasets.Dataset
|
| Example:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')
| >>> builder.download_and_prepare()
| >>> ds = builder.as_dataset(split='train')
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 8530
| })
| ```
|
| as_streaming_dataset(self, split: Optional[str] = None, base_path: Optional[str] = None) -> Union[dict[str, datasets.iterable_dataset.IterableDataset], datasets.iterable_dataset.IterableDataset]
|
| download_and_prepare(self, output_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, verification_mode: Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None, dl_manager: Optional[datasets.download.download_manager.DownloadManager] = None, base_path: Optional[str] = None, file_format: str = 'arrow', max_shard_size: Union[str, int, NoneType] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **download_and_prepare_kwargs)
| Downloads and prepares dataset for reading.
|
| Args:
| output_dir (`str`, *optional*):
| Output directory for the dataset.
| Default to this builder's `cache_dir`, which is inside `~/.cache/huggingface/datasets` by default.
|
| <Added version="2.5.0"/>
| download_config (`DownloadConfig`, *optional*):
| Specific download configuration parameters.
| download_mode ([`DownloadMode`] or `str`, *optional*):
| Select the download/generate mode, default to `REUSE_DATASET_IF_EXISTS`.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| dl_manager (`DownloadManager`, *optional*):
| Specific `DownloadManger` to use.
| base_path (`str`, *optional*):
| Base path for relative paths that are used to download files. This can be a remote url.
| If not specified, the value of the `base_path` attribute (`self.base_path`) will be used instead.
| file_format (`str`, *optional*):
| Format of the data files in which the dataset will be written.
| Supported formats: "arrow", "parquet". Default to "arrow" format.
| If the format is "parquet", then image and audio data are embedded into the Parquet files instead of pointing to local files.
|
| <Added version="2.5.0"/>
| max_shard_size (`Union[str, int]`, *optional*):
| Maximum number of bytes written per shard, default is "500MB".
| The size is based on uncompressed data size, so in practice your shard files may be smaller than
| `max_shard_size` thanks to Parquet compression for example.
|
| <Added version="2.5.0"/>
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.7.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the caching file-system backend, if any.
|
| <Added version="2.5.0"/>
| **download_and_prepare_kwargs (additional keyword arguments): Keyword arguments.
|
| Example:
|
| Download and prepare the dataset as Arrow files that can be loaded as a Dataset using `builder.as_dataset()`:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare()
| ```
|
| Download and prepare the dataset as sharded Parquet files locally:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("./output_dir", file_format="parquet")
| ```
|
| Download and prepare the dataset as sharded Parquet files in a cloud storage:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> storage_options = {"key": aws_access_key_id, "secret": aws_secret_access_key}
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("s3://my-bucket/my_rotten_tomatoes", storage_options=storage_options, file_format="parquet")
| ```
|
| ----------------------------------------------------------------------
| Class methods inherited from DatasetBuilder:
|
| get_imported_module_dir()
| Return the path of the module of this class or subclass.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from DatasetBuilder:
|
| builder_configs
| Dictionary of pre-defined configurations for this builder class.
|
| cache_dir
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DatasetBuilder:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from DatasetBuilder:
|
| BUILDER_CONFIGS = []
|
| BUILDER_CONFIG_CLASS = <class 'datasets.builder.BuilderConfig'>
| Base class for `DatasetBuilder` data configuration.
|
| `DatasetBuilder` subclasses with data configuration options should subclass
| `BuilderConfig` and add their own properties.
|
| Attributes:
| name (`str`, defaults to `default`):
| The name of the configuration.
| version (`Version` or `str`, defaults to `0.0.0`):
| The version of the configuration.
| data_dir (`str`, *optional*):
| Path to the directory containing the source data.
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| description (`str`, *optional*):
| A human description of the configuration.
|
|
| DEFAULT_CONFIG_NAME = None
|
| DEFAULT_WRITER_BATCH_SIZE = None
|
| SLEEP_ON_THREADS_SHUTDOWNS = False
|
| VERSION = None
datasets.Audio
Python Library Documentation: class Audio in module datasets.features.audio
class Audio(builtins.object)
| Audio(sampling_rate: Optional[int] = None, decode: bool = True, num_channels: Optional[int] = None, stream_index: Optional[int] = None, id: Optional[str] = None) -> None
|
| Audio [`Feature`] to extract audio data from an audio file.
|
| Input: The Audio feature accepts as input:
| - A `str`: Absolute path to the audio file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the audio file (i.e. random access is allowed).
| - A `dict` with the keys:
|
| - `path`: String with relative path of the audio file to the archive file.
| - `bytes`: Bytes content of the audio file.
|
| This is useful for parquet or webdataset files which embed audio files.
|
| - A `dict` with the keys:
|
| - `array`: Array containing the audio sample
| - `sampling_rate`: Integer corresponding to the sampling rate of the audio sample.
|
| - A `torchcodec.decoders.AudioDecoder`: torchcodec audio decoder object.
|
| Output: The Audio features output data as `torchcodec.decoders.AudioDecoder` objects, with additional keys:
|
| - `array`: Array containing the audio sample
| - `sampling_rate`: Integer corresponding to the sampling rate of the audio sample.
|
| Args:
| sampling_rate (`int`, *optional*):
| Target sampling rate. If `None`, the native sampling rate is used.
| num_channels (`int`, *optional*):
| The desired number of channels of the samples. By default, the number of channels of the source is used.
| Audio decoding will return samples with shape (num_channels, num_samples)
| Currently `None` (number of channels of the source, default), `1` (mono) or `2` (stereo) channels are supported.
| The `num_channels` argument is passed to `torchcodec.decoders.AudioDecoder`.
|
| <Added version="4.4.0"/>
| decode (`bool`, defaults to `True`):
| Whether to decode the audio data. If `False`,
| returns the underlying dictionary in the format `{"path": audio_path, "bytes": audio_bytes}`.
| stream_index (`int`, *optional*):
| The streaming index to use from the file. If `None` defaults to the "best" index.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, Audio
| >>> ds = load_dataset("PolyAI/minds14", name="en-US", split="train")
| >>> ds = ds.cast_column("audio", Audio(sampling_rate=44100, num_channels=2))
| >>> ds[0]["audio"]
| <datasets.features._torchcodec.AudioDecoder object at 0x11642b6a0>
| >>> audio = ds[0]["audio"]
| >>> audio.get_samples_played_in_range(0, 10)
| AudioSamples:
| data (shape): torch.Size([2, 110592])
| pts_seconds: 0.0
| duration_seconds: 2.507755102040816
| sample_rate: 44100
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, sampling_rate: Optional[int] = None, decode: bool = True, num_channels: Optional[int] = None, stream_index: Optional[int] = None, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Audio arrow storage type.
| The Arrow types that can be converted to the Audio pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.binary()` - it must contain the audio bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Audio arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`
|
| decode_example(self, value: dict, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None) -> 'AudioDecoder'
| Decode example audio file into audio data.
|
| Args:
| value (`dict`):
| A dictionary with keys:
|
| - `path`: String with relative audio file path.
| - `bytes`: Bytes of the audio file.
| token_per_repo_id (`dict`, *optional*):
| To access and decode
| audio files from private repositories on the Hub, you can pass
| a dictionary repo_id (`str`) -> token (`bool` or `str`)
|
| Returns:
| `torchcodec.decoders.AudioDecoder`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed audio files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the Audio arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, ForwardRef('AudioDecoder')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `bytes`,`bytearray`,`dict`, `AudioDecoder`):
| Data passed as input to Audio feature.
|
| Returns:
| `dict`
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| If in the decodable state, raise an error, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('sampling_rate', 'decode', 'num_channels', 'stream_i...
|
| decode = True
|
| dtype = 'dict'
|
| id = None
|
| num_channels = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
|
| sampling_rate = None
|
| stream_index = None
datasets.BuilderConfig
Python Library Documentation: class BuilderConfig in module datasets.builder
class BuilderConfig(builtins.object)
| BuilderConfig(name: str = 'default', version: Union[datasets.utils.version.Version, str, NoneType] = 0.0.0, data_dir: Optional[str] = None, data_files: Union[datasets.data_files.DataFilesDict, datasets.data_files.DataFilesPatternsDict, NoneType] = None, description: Optional[str] = None) -> None
|
| Base class for `DatasetBuilder` data configuration.
|
| `DatasetBuilder` subclasses with data configuration options should subclass
| `BuilderConfig` and add their own properties.
|
| Attributes:
| name (`str`, defaults to `default`):
| The name of the configuration.
| version (`Version` or `str`, defaults to `0.0.0`):
| The version of the configuration.
| data_dir (`str`, *optional*):
| Path to the directory containing the source data.
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| description (`str`, *optional*):
| A human description of the configuration.
|
| Methods defined here:
|
| __eq__(self, o)
| Return self==value.
|
| __init__(self, name: str = 'default', version: Union[datasets.utils.version.Version, str, NoneType] = 0.0.0, data_dir: Optional[str] = None, data_files: Union[datasets.data_files.DataFilesDict, datasets.data_files.DataFilesPatternsDict, NoneType] = None, description: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| create_config_id(self, config_kwargs: dict, custom_features: Optional[datasets.features.features.Features] = None) -> str
| The config id is used to build the cache directory.
| By default it is equal to the config name.
| However the name of a config is not sufficient to have a unique identifier for the dataset being generated
| since it doesn't take into account:
| - the config kwargs that can be used to overwrite attributes
| - the custom features used to write the dataset
| - the data_files for json/text/csv/pandas datasets
|
| Therefore the config id is just the config name with an optional suffix based on these.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'data_dir': typing.Optional[str], 'data_files': typ...
|
| __dataclass_fields__ = {'data_dir': Field(name='data_dir',type=typing....
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('name', 'version', 'data_dir', 'data_files', 'descri...
|
| data_dir = None
|
| data_files = None
|
| description = None
|
| name = 'default'
|
| version = 0.0.0
datasets.ClassLabel
Python Library Documentation: class ClassLabel in module datasets.features.features
class ClassLabel(builtins.object)
| ClassLabel(num_classes: dataclasses.InitVar[typing.Optional[int]] = None, names: list[str] = None, names_file: dataclasses.InitVar[typing.Optional[str]] = None, id: Optional[str] = None) -> None
|
| Feature type for integer class labels.
|
| There are 3 ways to define a `ClassLabel`, which correspond to the 3 arguments:
|
| * `num_classes`: Create 0 to (num_classes-1) labels.
| * `names`: List of label strings.
| * `names_file`: File containing the list of labels.
|
| Under the hood the labels are stored as integers.
| You can use -1 to represent unknown/missing labels.
|
| Args:
| num_classes (`int`, *optional*):
| Number of classes. All labels must be < `num_classes`.
| names (`list` of `str`, *optional*):
| String names for the integer classes.
| The order in which the names are provided is kept.
| names_file (`str`, *optional*):
| Path to a file with names for the integer classes, one per line.
|
| Example:
|
| ```py
| >>> from datasets import Features, ClassLabel
| >>> features = Features({'label': ClassLabel(num_classes=3, names=['bad', 'ok', 'good'])})
| >>> features
| {'label': ClassLabel(names=['bad', 'ok', 'good'])}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, num_classes: dataclasses.InitVar[typing.Optional[int]] = None, names: list[str] = None, names_file: dataclasses.InitVar[typing.Optional[str]] = None, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self, num_classes, names_file)
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.IntegerArray]) -> pyarrow.lib.Int64Array
| Cast an Arrow array to the `ClassLabel` arrow storage type.
| The Arrow types that can be converted to the `ClassLabel` pyarrow storage type are:
|
| - `pa.string()`
| - `pa.int()`
|
| Args:
| storage (`Union[pa.StringArray, pa.IntegerArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.Int64Array`: Array in the `ClassLabel` arrow storage type.
|
| encode_example(self, example_data)
|
| int2str(self, values: Union[int, collections.abc.Iterable]) -> Union[str, collections.abc.Iterable]
| Conversion `integer` => class name `string`.
|
| Regarding unknown/missing labels: passing negative integers raises `ValueError`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> ds.features["label"].int2str(0)
| 'neg'
| ```
|
| str2int(self, values: Union[str, collections.abc.Iterable]) -> Union[int, collections.abc.Iterable]
| Conversion class name `string` => `integer`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> ds.features["label"].str2int('neg')
| 0
| ```
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_int2str': typing.ClassVar[dict[int, int]], '_str2...
|
| __dataclass_fields__ = {'_int2str': Field(name='_int2str',type=typing....
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('num_classes', 'names', 'names_file', 'id')
|
| dtype = 'int64'
|
| id = None
|
| names = None
|
| names_file = None
|
| num_classes = None
|
| pa_type = DataType(int64)
datasets.Column
Python Library Documentation: class Column in module datasets.arrow_dataset
class Column(collections.abc.Sequence)
| Column(source: Union[ForwardRef('Dataset'), ForwardRef('Column')], column_name: str)
|
| An iterable for a specific column of a [`Dataset`].
|
| Example:
|
| Iterate on the texts of the "text" column of a dataset:
|
| ```python
| for text in dataset["text"]:
| ...
| ```
|
| It also works with nested columns:
|
| ```python
| for source in dataset["metadata"]["source"]:
| ...
| ```
|
| Method resolution order:
| Column
| collections.abc.Sequence
| collections.abc.Reversible
| collections.abc.Collection
| collections.abc.Sized
| collections.abc.Iterable
| collections.abc.Container
| builtins.object
|
| Methods defined here:
|
| __eq__(self, value)
| Return self==value.
|
| __getitem__(self, key: Union[int, str, list[int]]) -> Any
|
| __init__(self, source: Union[ForwardRef('Dataset'), ForwardRef('Column')], column_name: str)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> collections.abc.Iterator[typing.Any]
|
| __len__(self) -> int
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __hash__ = None
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.Sequence:
|
| __contains__(self, value)
|
| __reversed__(self)
|
| count(self, value)
| S.count(value) -> integer -- return number of occurrences of value
|
| index(self, value, start=0, stop=None)
| S.index(value, [start, [stop]]) -> integer -- return first index of value.
| Raises ValueError if the value is not present.
|
| Supporting start and stop arguments is optional, but
| recommended.
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Reversible:
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
datasets.Dataset
Python Library Documentation: class Dataset in module datasets.arrow_dataset
class Dataset(DatasetInfoMixin, datasets.search.IndexableMixin, TensorflowDatasetMixin)
| Dataset(arrow_table: datasets.table.Table, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, indices_table: Optional[datasets.table.Table] = None, fingerprint: Optional[str] = None)
|
| A Dataset backed by an Arrow table.
|
| Method resolution order:
| Dataset
| DatasetInfoMixin
| datasets.search.IndexableMixin
| TensorflowDatasetMixin
| builtins.object
|
| Methods defined here:
|
| __del__(self)
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| __getitem__(self, key)
| Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools).
|
| __getitems__(self, keys: list) -> list
| Can be used to get a batch using a list of integers indices.
|
| __init__(self, arrow_table: datasets.table.Table, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, indices_table: Optional[datasets.table.Table] = None, fingerprint: Optional[str] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| Iterate through the examples.
|
| If a formatting is set with [`Dataset.set_format`] rows will be returned with the
| selected format.
|
| __len__(self)
| Number of rows in the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.__len__
| <bound method Dataset.__len__ of Dataset({
| features: ['text', 'label'],
| num_rows: 1066
| })>
| ```
|
| __repr__(self)
| Return repr(self).
|
| __setstate__(self, state)
|
| add_column(self, name: str, column: Union[list, numpy.ndarray], new_fingerprint: Optional[str] = None, feature: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None)
| Add column to Dataset.
|
| <Added version="1.7"/>
|
| Args:
| name (`str`):
| Column name.
| column (`list` or `np.array`):
| Column data to be added.
| feature (`FeatureType` or `None`, defaults to `None`):
| Column datatype.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> more_text = ds["text"]
| >>> ds = ds.add_column(name="text_2", column=more_text)
| >>> ds
| Dataset({
| features: ['text', 'label', 'text_2'],
| num_rows: 1066
| })
| ```
|
| add_elasticsearch_index(self, column: str, index_name: Optional[str] = None, host: Optional[str] = None, port: Optional[int] = None, es_client: Optional[ForwardRef('elasticsearch.Elasticsearch')] = None, es_index_name: Optional[str] = None, es_index_config: Optional[dict] = None)
| Add a text index using ElasticSearch for fast retrieval. This is done in-place.
|
| Args:
| column (`str`):
| The column of the documents to add to the index.
| index_name (`str`, *optional*):
| The `index_name`/identifier of the index.
| This is the index name that is used to call [`~Dataset.get_nearest_examples`] or [`~Dataset.search`].
| By default it corresponds to `column`.
| host (`str`, *optional*, defaults to `localhost`):
| Host of where ElasticSearch is running.
| port (`str`, *optional*, defaults to `9200`):
| Port of where ElasticSearch is running.
| es_client (`elasticsearch.Elasticsearch`, *optional*):
| The elasticsearch client used to create the index if host and port are `None`.
| es_index_name (`str`, *optional*):
| The elasticsearch index name used to create the index.
| es_index_config (`dict`, *optional*):
| The configuration of the elasticsearch index.
| Default config is:
| ```
| {
| "settings": {
| "number_of_shards": 1,
| "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},
| },
| "mappings": {
| "properties": {
| "text": {
| "type": "text",
| "analyzer": "standard",
| "similarity": "BM25"
| },
| }
| },
| }
| ```
| Example:
|
| ```python
| >>> es_client = elasticsearch.Elasticsearch()
| >>> ds = datasets.load_dataset('community-datasets/crime_and_punish', split='train')
| >>> ds.add_elasticsearch_index(column='line', es_client=es_client, es_index_name="my_es_index")
| >>> scores, retrieved_examples = ds.get_nearest_examples('line', 'my new query', k=10)
| ```
|
| add_faiss_index(self, column: str, index_name: Optional[str] = None, device: Optional[int] = None, string_factory: Optional[str] = None, metric_type: Optional[int] = None, custom_index: Optional[ForwardRef('faiss.Index')] = None, batch_size: int = 1000, train_size: Optional[int] = None, faiss_verbose: bool = False, dtype=<class 'numpy.float32'>)
| Add a dense index using Faiss for fast retrieval.
| By default the index is done over the vectors of the specified column.
| You can specify `device` if you want to run it on GPU (`device` must be the GPU index).
| You can find more information about Faiss here:
|
| - For [string factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)
|
| Args:
| column (`str`):
| The column of the vectors to add to the index.
| index_name (`str`, *optional*):
| The `index_name`/identifier of the index.
| This is the `index_name` that is used to call [`~datasets.Dataset.get_nearest_examples`] or [`~datasets.Dataset.search`].
| By default it corresponds to `column`.
| device (`Union[int, List[int]]`, *optional*):
| If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs.
| If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU.
| string_factory (`str`, *optional*):
| This is passed to the index factory of Faiss to create the index.
| Default index class is `IndexFlat`.
| metric_type (`int`, *optional*):
| Type of metric. Ex: `faiss.METRIC_INNER_PRODUCT` or `faiss.METRIC_L2`.
| custom_index (`faiss.Index`, *optional*):
| Custom Faiss index that you already have instantiated and configured for your needs.
| batch_size (`int`):
| Size of the batch to use while adding vectors to the `FaissIndex`. Default value is `1000`.
| <Added version="2.4.0"/>
| train_size (`int`, *optional*):
| If the index needs a training step, specifies how many vectors will be used to train the index.
| faiss_verbose (`bool`, defaults to `False`):
| Enable the verbosity of the Faiss index.
| dtype (`data-type`):
| The dtype of the numpy arrays that are indexed.
| Default is `np.float32`.
|
| Example:
|
| ```python
| >>> ds = datasets.load_dataset('community-datasets/crime_and_punish', split='train')
| >>> ds_with_embeddings = ds.map(lambda example: {'embeddings': embed(example['line']}))
| >>> ds_with_embeddings.add_faiss_index(column='embeddings')
| >>> # query
| >>> scores, retrieved_examples = ds_with_embeddings.get_nearest_examples('embeddings', embed('my new query'), k=10)
| >>> # save index
| >>> ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')
|
| >>> ds = datasets.load_dataset('community-datasets/crime_and_punish', split='train')
| >>> # load index
| >>> ds.load_faiss_index('embeddings', 'my_index.faiss')
| >>> # query
| >>> scores, retrieved_examples = ds.get_nearest_examples('embeddings', embed('my new query'), k=10)
| ```
|
| add_faiss_index_from_external_arrays(self, external_arrays: <built-in function array>, index_name: str, device: Optional[int] = None, string_factory: Optional[str] = None, metric_type: Optional[int] = None, custom_index: Optional[ForwardRef('faiss.Index')] = None, batch_size: int = 1000, train_size: Optional[int] = None, faiss_verbose: bool = False, dtype=<class 'numpy.float32'>)
| Add a dense index using Faiss for fast retrieval.
| The index is created using the vectors of `external_arrays`.
| You can specify `device` if you want to run it on GPU (`device` must be the GPU index).
| You can find more information about Faiss here:
|
| - For [string factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)
|
| Args:
| external_arrays (`np.array`):
| If you want to use arrays from outside the lib for the index, you can set `external_arrays`.
| It will use `external_arrays` to create the Faiss index instead of the arrays in the given `column`.
| index_name (`str`):
| The `index_name`/identifier of the index.
| This is the `index_name` that is used to call [`~datasets.Dataset.get_nearest_examples`] or [`~datasets.Dataset.search`].
| device (Optional `Union[int, List[int]]`, *optional*):
| If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs.
| If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU.
| string_factory (`str`, *optional*):
| This is passed to the index factory of Faiss to create the index.
| Default index class is `IndexFlat`.
| metric_type (`int`, *optional*):
| Type of metric. Ex: `faiss.faiss.METRIC_INNER_PRODUCT` or `faiss.METRIC_L2`.
| custom_index (`faiss.Index`, *optional*):
| Custom Faiss index that you already have instantiated and configured for your needs.
| batch_size (`int`, *optional*):
| Size of the batch to use while adding vectors to the FaissIndex. Default value is 1000.
| <Added version="2.4.0"/>
| train_size (`int`, *optional*):
| If the index needs a training step, specifies how many vectors will be used to train the index.
| faiss_verbose (`bool`, defaults to False):
| Enable the verbosity of the Faiss index.
| dtype (`numpy.dtype`):
| The dtype of the numpy arrays that are indexed. Default is np.float32.
|
| add_item(self, item: dict, new_fingerprint: Optional[str] = None)
| Add item to Dataset.
|
| <Added version="1.7"/>
|
| Args:
| item (`dict`):
| Item data to be added.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> new_review = {'label': 0, 'text': 'this movie is the absolute worst thing I have ever seen'}
| >>> ds = ds.add_item(new_review)
| >>> ds[-1]
| {'label': 0, 'text': 'this movie is the absolute worst thing I have ever seen'}
| ```
|
| align_labels_with_mapping(self, label2id: dict, label_column: str) -> 'Dataset'
| Align the dataset's label ID and label name mapping to match an input `label2id` mapping.
| This is useful when you want to ensure that a model's predicted labels are aligned with the dataset.
| The alignment in done using the lowercase label names.
|
| Args:
| label2id (`dict`):
| The label name to ID mapping to align the dataset with.
| label_column (`str`):
| The column name of labels to align on.
|
| Example:
|
| ```python
| >>> # dataset with mapping {'entailment': 0, 'neutral': 1, 'contradiction': 2}
| >>> ds = load_dataset("nyu-mll/glue", "mnli", split="train")
| >>> # mapping to align with
| >>> label2id = {'CONTRADICTION': 0, 'NEUTRAL': 1, 'ENTAILMENT': 2}
| >>> ds_aligned = ds.align_labels_with_mapping(label2id, "label")
| ```
|
| batch(self, batch_size: Optional[int] = None, by_column: Union[str, list[str], NoneType] = None, drop_last_batch: bool = False, num_proc: Optional[int] = None, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Group samples from the dataset into batches.
|
| Specify either `batch_size` to make batches of `batch_size` examples.
|
| Or use `by_column=column_name` to make batches of successive examples that share
| the same value in column `column_name`.
|
| Args:
| batch_size (`int`, optional):
| The number of samples in each batch.
| by_column (`Union[str, list[str]`, optional):
| The column used to batch examples together.
| Successive examples with the same value for that column are in grouped the same batch.
| This can also be a list of columns if you want to batch by multiple columns.
| If batching by column, the batch_size is only used to control the size of the batches
| to group together or slice during acculumation.
|
| <Added version="4.9.0"/>
| drop_last_batch (`bool`, defaults to `False`):
| Whether to drop the last incomplete batch.
| num_proc (`int`, *optional*, defaults to `None`):
| Max number of processes when generating cache. Already cached shards are loaded sequentially.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A new Dataset where each item is a batch of multiple samples from the original dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> batched_ds = ds.batch(batch_size=4)
| >>> batched_ds[0]
| {'text': ['compassionately explores the seemingly irreconcilable situation...', ...], # 4 items
| 'label': [1, 1, 1, 1]}
| ```
|
| cast(self, features: datasets.features.features.Features, batch_size: Optional[int] = 1000, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, num_proc: Optional[int] = None) -> 'Dataset'
| Cast the dataset to a new set of features.
|
| Args:
| features ([`Features`]):
| New features to cast the dataset to.
| The name of the fields in the features must match the current column names.
| The type of the data must also be convertible from one type to the other.
| For non-trivial conversion, e.g. `str` <-> `ClassLabel` you should use [`~datasets.Dataset.map`] to update the Dataset.
| batch_size (`int`, defaults to `1000`):
| Number of examples per batch provided to cast.
| If `batch_size <= 0` or `batch_size == None` then provide the full dataset as a single batch to cast.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| load_from_cache_file (`bool`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_name (`str`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running [`~datasets.Dataset.map`].
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes for multiprocessing. By default it doesn't
| use multiprocessing.
|
| Returns:
| [`Dataset`]: A copy of the dataset with casted features.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel, Value
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> new_features = ds.features.copy()
| >>> new_features['label'] = ClassLabel(names=['bad', 'good'])
| >>> new_features['text'] = Value('large_string')
| >>> ds = ds.cast(new_features)
| >>> ds.features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('large_string')}
| ```
|
| cast_column(self, column: str, feature: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti], new_fingerprint: Optional[str] = None) -> 'Dataset'
| Cast column to feature for decoding.
|
| Args:
| column (`str`):
| Column name.
| feature (`FeatureType`):
| Target feature.
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> ds = ds.cast_column('label', ClassLabel(names=['bad', 'good']))
| >>> ds.features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('string')}
| ```
|
| class_encode_column(self, column: str, include_nulls: bool = False) -> 'Dataset'
| Casts the given column as [`~datasets.features.ClassLabel`] and updates the table.
|
| Args:
| column (`str`):
| The name of the column to cast (list all the column names with [`~datasets.Dataset.column_names`])
| include_nulls (`bool`, defaults to `False`):
| Whether to include null values in the class labels. If `True`, the null values will be encoded as the `"None"` class label.
|
| <Added version="1.14.2"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("google/boolq", split="validation")
| >>> ds.features
| {'answer': Value('bool'),
| 'passage': Value('string'),
| 'question': Value('string')}
| >>> ds = ds.class_encode_column('answer')
| >>> ds.features
| {'answer': ClassLabel(num_classes=2, names=['False', 'True']),
| 'passage': Value('string'),
| 'question': Value('string')}
| ```
|
| cleanup_cache_files(self) -> int
| Clean up all cache files in the dataset cache directory, excepted the currently used cache file if there is
| one.
|
| Be careful when running this command that no other process is currently using other cache files.
|
| Returns:
| `int`: Number of removed files.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.cleanup_cache_files()
| 10
| ```
|
| filter(self, function: Optional[Callable] = None, with_indices: bool = False, with_rank: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, suffix_template: str = '_{rank:05d}_of_{num_proc:05d}', new_fingerprint: Optional[str] = None, desc: Optional[str] = None) -> 'Dataset'
| Apply a filter function to all the elements in the table in batches
| and update the table so that the dataset only includes examples according to the filter function.
|
| If the function is asynchronous, then `filter` will run your function in parallel, with up to one thousand simultaneous calls (configurable).
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`): Callable with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> bool` if `batched=False` and `with_indices=False` and `with_rank=False`
| - `function(example: Dict[str, Any], *extra_args) -> bool` if `batched=False` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
| - `function(batch: Dict[str, List]) -> List[bool]` if `batched=True` and `with_indices=False` and `with_rank=False`
| - `function(batch: Dict[str, List], *extra_args) -> List[bool]` if `batched=True` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
|
| If the function is asynchronous, then `filter` will run your function in parallel.
| If no function is provided, defaults to an always `True` function: `lambda x: True`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the
| signature of `function` should be `def function(example, idx[, rank]): ...`.
| with_rank (`bool`, defaults to `False`):
| Provide process rank to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], rank): ...`.
| input_columns (`str` or `List[str]`, *optional*):
| The columns to be passed into `function` as
| positional arguments. If `None`, a `dict` mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if
| `batched = True`. If `batched = False`, one example per batch is passed to `function`.
| If `batch_size <= 0` or `batch_size == None`, provide the full dataset as a single batch to `function`.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| fn_kwargs (`dict`, *optional*):
| Keyword arguments to be passed to `function`.
| num_proc (`int`, *optional*, defaults to `None`):
| The number of processes to use for multiprocessing.
| - If `None` or `0`, no multiprocessing is used and the operation runs in the main process.
| - If greater than `1`, one or multiple worker processes are used to process data in parallel.
| Note: The function passed to `map()` must be picklable for multiprocessing to work correctly
| (i.e., prefer functions defined at the top level of a module, not inside another function or class).
| suffix_template (`str`):
| If `cache_file_name` is specified, then this suffix will be added at the end of the base name of each.
| For example, if `cache_file_name` is `"processed.arrow"`, then for `rank = 1` and `num_proc = 4`,
| the resulting file would be `"processed_00001_of_00004.arrow"` for the default suffix (default
| `_{rank:05d}_of_{num_proc:05d}`).
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
| desc (`str`, *optional*, defaults to `None`):
| Meaningful description to be displayed alongside with the progress bar while filtering examples.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.filter(lambda x: x["label"] == 1)
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 533
| })
| ```
|
| flatten(self, new_fingerprint: Optional[str] = None, max_depth=16) -> 'Dataset'
| Flatten the table.
| Each column with a struct type is flattened into one column per struct field.
| Other columns are left unchanged.
|
| Args:
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset with flattened columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("rajpurkar/squad", split="train")
| >>> ds.features
| {'id': Value('string'),
| 'title': Value('string'),
| 'context': Value('string'),
| 'question': Value('string'),
| 'answers': {'text': List(Value('string')),
| 'answer_start': List(Value('int32'))}}
| >>> ds = ds.flatten()
| >>> ds
| Dataset({
| features: ['id', 'title', 'context', 'question', 'answers.text', 'answers.answer_start'],
| num_rows: 87599
| })
| ```
|
| flatten_indices(self, keep_in_memory: bool = False, cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, features: Optional[datasets.features.features.Features] = None, disable_nullable: bool = False, num_proc: Optional[int] = None, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Create and cache a new Dataset by flattening the indices mapping.
|
| Args:
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| cache_file_name (`str`, *optional*, default `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| features (`Optional[datasets.Features]`, defaults to `None`):
| Use a specific [`Features`] to store the cache file
| instead of the automatically generated one.
| disable_nullable (`bool`, defaults to `False`):
| Allow null values in the table.
| num_proc (`int`, optional, default `None`):
| Max number of processes when generating cache. Already cached shards are loaded sequentially
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
|
| formatted_as(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| To be used in a `with` statement. Set `__getitem__` return format (type and columns).
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__`` returns python objects (default).
| columns (`List[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| iter(self, batch_size: int, drop_last_batch: bool = False)
| Iterate through the batches of size `batch_size`.
|
| If a formatting is set with [`~datasets.Dataset.set_format`] rows will be returned with the
| selected format.
|
| Args:
| batch_size (:obj:`int`): size of each batch to yield.
| drop_last_batch (:obj:`bool`, default `False`): Whether a last batch smaller than the batch_size should be
| dropped
|
| map(self, function: Optional[Callable] = None, with_indices: bool = False, with_rank: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, drop_last_batch: bool = False, remove_columns: Union[str, list[str], NoneType] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, features: Optional[datasets.features.features.Features] = None, disable_nullable: bool = False, fn_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, suffix_template: str = '_{rank:05d}_of_{num_proc:05d}', new_fingerprint: Optional[str] = None, desc: Optional[str] = None, try_original_type: Optional[bool] = True, on_mixed_types: Optional[Literal['use_json']] = 'use_json') -> 'Dataset'
| Apply a function to all the examples in the table (individually or in batches) and update the table.
| If your function returns a column that already exists, then it overwrites it.
|
| You can specify whether the function should be batched or not with the `batched` parameter:
|
| - If batched is `False`, then the function takes 1 example in and should return 1 example.
| An example is a dictionary, e.g. `{"text": "Hello there !"}`.
| - If batched is `True` and `batch_size` is 1, then the function takes a batch of 1 example as input and can return a batch with 1 or more examples.
| A batch is a dictionary, e.g. a batch of 1 example is `{"text": ["Hello there !"]}`.
| - If batched is `True` and `batch_size` is `n > 1`, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.
| Note that the last batch may have less than `n` examples.
| A batch is a dictionary, e.g. a batch of `n` examples is `{"text": ["Hello there !"] * n}`.
|
| If the function is asynchronous, then `map` will run your function in parallel, with up to one thousand simultaneous calls.
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`): Function with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> Dict[str, Any]` if `batched=False` and `with_indices=False` and `with_rank=False`
| - `function(example: Dict[str, Any], *extra_args) -> Dict[str, Any]` if `batched=False` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
| - `function(batch: Dict[str, List]) -> Dict[str, List]` if `batched=True` and `with_indices=False` and `with_rank=False`
| - `function(batch: Dict[str, List], *extra_args) -> Dict[str, List]` if `batched=True` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
|
| For advanced usage, the function can also return a `pyarrow.Table`.
| If the function is asynchronous, then `map` will run your function in parallel.
| Moreover if your function returns nothing (`None`), then `map` will run your function and return the dataset unchanged.
| If no function is provided, default to identity function: `lambda x: x`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the
| signature of `function` should be `def function(example, idx[, rank]): ...`.
| with_rank (`bool`, defaults to `False`):
| Provide process rank to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], rank): ...`.
| input_columns (`Optional[Union[str, List[str]]]`, defaults to `None`):
| The columns to be passed into `function`
| as positional arguments. If `None`, a `dict` mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| If `batch_size <= 0` or `batch_size == None`, provide the full dataset as a single batch to `function`.
| drop_last_batch (`bool`, defaults to `False`):
| Whether a last batch smaller than the batch_size should be
| dropped instead of being processed by the function.
| remove_columns (`Optional[Union[str, List[str]]]`, defaults to `None`):
| Remove a selection of columns while doing the mapping.
| Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding
| columns with names in `remove_columns`, these columns will be kept.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_name (`str`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| features (`Optional[datasets.Features]`, defaults to `None`):
| Use a specific Features to store the cache file
| instead of the automatically generated one.
| disable_nullable (`bool`, defaults to `False`):
| Disallow null values in the table.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`.
| num_proc (`int`, *optional*, defaults to `None`):
| The number of processes to use for multiprocessing.
| - If `None` or `0`, no multiprocessing is used and the operation runs in the main process.
| - If greater than `1`, one or multiple worker processes are used to process data in parallel.
| Note: The function passed to `map()` must be picklable for multiprocessing to work correctly
| (i.e., prefer functions defined at the top level of a module, not inside another function or class).
| suffix_template (`str`):
| If `cache_file_name` is specified, then this suffix
| will be added at the end of the base name of each. Defaults to `"_{rank:05d}_of_{num_proc:05d}"`. For example, if `cache_file_name` is "processed.arrow", then for
| `rank=1` and `num_proc=4`, the resulting file would be `"processed_00001_of_00004.arrow"` for the default suffix.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
| desc (`str`, *optional*, defaults to `None`):
| Meaningful description to be displayed alongside with the progress bar while mapping examples.
| try_original_type (`Optional[bool]`, defaults to `True`):
| Try to keep the types of the original columns (e.g. int32 -> int32).
| Set to False if you want to always infer new types.
| on_mixed_types (`Literal["use_json"]`, *optional*, defaults to `None`):
| If "use_json", use the Json() type for mixed-types fields,
| i.e. unstructured fields that contain data without a predefined schema.
| In this case, a field with mixed type is set to Json().
|
| This allow loading lists with a mix of strings/integers/floats
| for example, or dictionaries with arbitrary value types.
|
| <Added version="4.7.0"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> def add_prefix(example):
| ... example["text"] = "Review: " + example["text"]
| ... return example
| >>> ds = ds.map(add_prefix)
| >>> ds[0:3]["text"]
| ['Review: compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'Review: the soundtrack alone is worth the price of admission .',
| 'Review: rodriguez does a splendid job of racial profiling hollywood style--casting excellent latin actors of all ages--a trend long overdue .']
|
| # process a batch of examples
| >>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
| # set number of processors
| >>> ds = ds.map(add_prefix, num_proc=4)
| ```
|
| push_to_hub(self, repo_id: str, config_name: str = 'default', set_default: Optional[bool] = None, split: Optional[str] = None, data_dir: Optional[str] = None, commit_message: Optional[str] = None, commit_description: Optional[str] = None, private: Optional[bool] = None, token: Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = False, max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[int] = None, embed_external_files: bool = True, num_proc: Optional[int] = None) -> huggingface_hub.hf_api.CommitInfo
| Pushes the dataset to the hub as a Parquet dataset.
| The dataset is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
|
| The resulting Parquet files are self-contained by default. If your dataset contains [`Image`], [`Audio`] or [`Video`]
| data, the Parquet files will store the bytes of your images or audio files.
| You can disable this by setting `embed_external_files` to `False`.
|
| Args:
| repo_id (`str`):
| The ID of the repository to push to in the following format: `<user>/<dataset_name>` or
| `<org>/<dataset_name>`. Also accepts `<dataset_name>`, which will default to the namespace
| of the logged-in user.
|
| It could also be a location inside a bucket, e.g. `buckets/<user_or_org>/<bucket_name>/...`
| config_name (`str`, defaults to "default"):
| The configuration name (or subset) of a dataset. Defaults to "default".
| set_default (`bool`, *optional*):
| Whether to set this configuration as the default one. Otherwise, the default configuration is the one
| named "default".
| split (`str`, *optional*):
| The name of the split that will be given to that dataset. Defaults to `self.split`.
| data_dir (`str`, *optional*):
| Directory name that will contain the uploaded data files. Defaults to the `config_name` if different
| from "default", else "data".
|
| <Added version="2.17.0"/>
| commit_message (`str`, *optional*):
| Message to commit while pushing. Will default to `"Upload dataset"`.
| commit_description (`str`, *optional*):
| Description of the commit that will be created.
| Additionally, description of the PR if a PR is created (`create_pr` is True).
|
| <Added version="2.16.0"/>
| private (`bool`, *optional*):
| Whether to make the repo private. If `None` (default), the repo will be public unless the
| organization's default is private. This value is ignored if the repo already exists.
| token (`str`, *optional*):
| An optional authentication token for the Hugging Face Hub. If no token is passed, will default
| to the token saved locally when logging in with `huggingface-cli login`. Will raise an error
| if no token is passed and the user is not logged-in.
| revision (`str`, *optional*):
| Branch to push the uploaded files to. Defaults to the `"main"` branch.
|
| <Added version="2.15.0"/>
| create_pr (`bool`, *optional*, defaults to `False`):
| Whether to create a PR with the uploaded files or directly commit.
|
| <Added version="2.15.0"/>
| max_shard_size (`int` or `str`, *optional*, defaults to `"500MB"`):
| The maximum size of the dataset shards to be uploaded to the hub. If expressed as a string, needs to be digits followed by
| a unit (like `"5MB"`).
| num_shards (`int`, *optional*):
| Number of shards to write. By default, the number of shards depends on `max_shard_size`.
|
| <Added version="2.8.0"/>
| embed_external_files (`bool`, defaults to `True`):
| Whether to embed file bytes in the shards.
| In particular, this will do the following before the push for the fields of type:
|
| - [`Audio`] and [`Image`]: remove local path information and embed file content in the Parquet files.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when preparing and uploading the dataset.
| This is helpful if the dataset is made of many samples or media files to embed.
| I uses "spawn" context to work with hf_xet, the rust client for fast uploads to HF.
| Multiprocessing is disabled by default.
|
| <Added version="4.0.0"/>
|
| Return:
| huggingface_hub.CommitInfo
|
| Example:
|
| ```python
| >>> dataset.push_to_hub("<organization>/<dataset_id>")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", private=True)
| >>> dataset.push_to_hub("<organization>/<dataset_id>", max_shard_size="1GB")
| >>> dataset.push_to_hub("<organization>/<dataset_id>", num_shards=1024)
| ```
|
| If your dataset has multiple splits (e.g. train/validation/test):
|
| ```python
| >>> train_dataset.push_to_hub("<organization>/<dataset_id>", split="train")
| >>> val_dataset.push_to_hub("<organization>/<dataset_id>", split="validation")
| >>> # later
| >>> dataset = load_dataset("<organization>/<dataset_id>")
| >>> train_dataset = dataset["train"]
| >>> val_dataset = dataset["validation"]
| ```
|
| If you want to add a new configuration (or subset) to a dataset (e.g. if the dataset has multiple tasks/versions/languages):
|
| ```python
| >>> english_dataset.push_to_hub("<organization>/<dataset_id>", "en")
| >>> french_dataset.push_to_hub("<organization>/<dataset_id>", "fr")
| >>> # later
| >>> english_dataset = load_dataset("<organization>/<dataset_id>", "en")
| >>> french_dataset = load_dataset("<organization>/<dataset_id>", "fr")
| ```
|
| remove_columns(self, column_names: Union[str, list[str]], new_fingerprint: Optional[str] = None) -> 'Dataset'
| Remove one or several column(s) in the dataset and the features associated to them.
|
| You can also remove a column using [`~datasets.Dataset.map`] with `remove_columns` but the present method
| doesn't copy the data of the remaining columns and is thus faster.
|
| Args:
| column_names (`Union[str, List[str]]`):
| Name of the column(s) to remove.
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset object without the columns to remove.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.remove_columns('label')
| Dataset({
| features: ['text'],
| num_rows: 1066
| })
| >>> ds = ds.remove_columns(column_names=ds.column_names) # Removing all the columns returns an empty dataset with the `num_rows` property set to 0
| Dataset({
| features: [],
| num_rows: 0
| })
| ```
|
| rename_column(self, original_column_name: str, new_column_name: str, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Rename a column in the dataset, and move the features associated to the original column under the new column
| name.
|
| Args:
| original_column_name (`str`):
| Name of the column to rename.
| new_column_name (`str`):
| New name for the column.
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset with a renamed column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.rename_column('label', 'label_new')
| Dataset({
| features: ['text', 'label_new'],
| num_rows: 1066
| })
| ```
|
| rename_columns(self, column_mapping: dict[str, str], new_fingerprint: Optional[str] = None) -> 'Dataset'
| Rename several columns in the dataset, and move the features associated to the original columns under
| the new column names.
|
| Args:
| column_mapping (`Dict[str, str]`):
| A mapping of columns to rename to their new names
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset with renamed columns
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.rename_columns({'text': 'text_new', 'label': 'label_new'})
| Dataset({
| features: ['text_new', 'label_new'],
| num_rows: 1066
| })
| ```
|
| repeat(self, num_times: int) -> 'Dataset'
| Create a new [`Dataset`] that repeats the underlying dataset `num_times` times.
|
| Like itertools.repeat, repeating once just returns the full dataset.
|
| Args:
| num_times (`int`):
| Number of times to repeat the dataset.
|
| Example:
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> ds = ds.take(2).repeat(2)
| >>> list(ds)
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'},
| {'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| ```
|
| reset_format(self)
| Reset `__getitem__` return format to python objects and all columns.
|
| Same as `self.set_format()`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds.set_format(type='numpy', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
| >>> ds.format
| {'columns': ['input_ids', 'token_type_ids', 'attention_mask', 'label'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'numpy'}
| >>> ds.reset_format()
| >>> ds.format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': None}
| ```
|
| save_to_disk(self, dataset_path: Union[str, bytes, os.PathLike], max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[int] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None)
| Saves a dataset to a dataset directory, or in a filesystem using any implementation of `fsspec.spec.AbstractFileSystem`.
|
| For [`Image`], [`Audio`] and [`Video`] data:
|
| All the Image(), Audio() and Video() data are stored in the arrow files.
| If you want to store paths or urls, please use the Value("string") type.
|
| Args:
| dataset_path (`path-like`):
| Path (e.g. `dataset/train`) or remote URI (e.g. `s3://my-bucket/dataset/train`)
| of the dataset directory where the dataset will be saved to.
| max_shard_size (`int` or `str`, *optional*, defaults to `"500MB"`):
| The maximum size of the dataset shards to be saved to the filesystem. If expressed as a string, needs to be digits followed by a unit
| (like `"50MB"`).
| num_shards (`int`, *optional*):
| Number of shards to write. By default the number of shards depends on `max_shard_size` and `num_proc`.
|
| <Added version="2.8.0"/>
| num_proc (`int`, *optional*):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.8.0"/>
|
| Example:
|
| ```py
| >>> ds.save_to_disk("path/to/dataset/directory")
| >>> ds.save_to_disk("path/to/dataset/directory", max_shard_size="1GB")
| >>> ds.save_to_disk("path/to/dataset/directory", num_shards=1024)
| ```
|
| select(self, indices: collections.abc.Iterable, keep_in_memory: bool = False, indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Create a new dataset with rows selected following the list/array of indices.
|
| Args:
| indices (`range`, `list`, `iterable`, `ndarray` or `Series`):
| Range, list or 1D-array of integer indices for indexing.
| If the indices correspond to a contiguous range, the Arrow table is simply sliced.
| However passing a list of indices that are not contiguous creates indices mapping, which is much less efficient,
| but still faster than recreating an Arrow table made of the requested rows.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the indices mapping in memory instead of writing it to a cache file.
| indices_cache_file_name (`str`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| indices mapping instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.select(range(4))
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 4
| })
| ```
|
| select_columns(self, column_names: Union[str, list[str]], new_fingerprint: Optional[str] = None) -> 'Dataset'
| Select one or several column(s) in the dataset and the features
| associated to them.
|
| Args:
| column_names (`Union[str, List[str]]`):
| Name of the column(s) to keep.
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform. If `None`,
| the new fingerprint is computed using a hash of the previous
| fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset object which only consists of
| selected columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.select_columns(['text'])
| >>> ds
| Dataset({
| features: ['text'],
| num_rows: 1066
| })
| ```
|
| set_format(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| Set `__getitem__` return format (type and columns). The data formatting is applied on-the-fly.
| The format `type` (for example "numpy") is used to format batches when using `__getitem__`.
| It's also possible to use custom transforms for formatting using [`~datasets.Dataset.set_transform`].
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`List[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| It is possible to call [`~datasets.Dataset.map`] after calling `set_format`. Since `map` may add new columns, then the list of formatted columns
| gets updated. In this case, if you apply `map` on a dataset to add a new column, then this column will be formatted as:
|
| ```
| new formatted columns = (all columns - previously unformatted columns)
| ```
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds.set_format(type='numpy', columns=['text', 'label'])
| >>> ds.format
| {'type': 'numpy',
| 'format_kwargs': {},
| 'columns': ['text', 'label'],
| 'output_all_columns': False}
| ```
|
| set_transform(self, transform: Optional[Callable], columns: Optional[list] = None, output_all_columns: bool = False)
| Set `__getitem__` return format using this transform. The transform is applied on-the-fly on batches when `__getitem__` is called.
| As [`~datasets.Dataset.set_format`], this can be reset using [`~datasets.Dataset.reset_format`].
|
| Args:
| transform (`Callable`, *optional*):
| User-defined formatting transform, replaces the format defined by [`~datasets.Dataset.set_format`].
| A formatting function is a callable that takes a batch (as a `dict`) as input and returns a batch.
| This function is applied right before returning the objects in `__getitem__`.
| columns (`List[str]`, *optional*):
| Columns to format in the output.
| If specified, then the input batch of the transform only contains those columns.
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| If set to True, then the other un-formatted columns are kept with the output of the transform.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
| >>> def encode(batch):
| ... return tokenizer(batch['text'], padding=True, truncation=True, return_tensors='pt')
| >>> ds.set_transform(encode)
| >>> ds[0]
| {'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1]),
| 'input_ids': tensor([ 101, 29353, 2135, 15102, 1996, 9428, 20868, 2890, 8663, 6895,
| 20470, 2571, 3663, 2090, 4603, 3017, 3008, 1998, 2037, 24211,
| 5637, 1998, 11690, 2336, 1012, 102]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0])}
| ```
|
| shard(self, num_shards: int, index: int, contiguous: bool = True, keep_in_memory: bool = False, indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000) -> 'Dataset'
| Return the `index`-nth shard from dataset split into `num_shards` pieces.
|
| This shards deterministically. `dataset.shard(n, i)` splits the dataset into contiguous chunks,
| so it can be easily concatenated back together after processing. If `len(dataset) % n == l`, then the
| first `l` dataset each have length `(len(dataset) // n) + 1`, and the remaining dataset have length `(len(dataset) // n)`.
| `datasets.concatenate_datasets([dset.shard(n, i) for i in range(n)])` returns a dataset with the same order as the original.
|
| Note: n should be less or equal to the number of elements in the dataset `len(dataset)`.
|
| On the other hand, `dataset.shard(n, i, contiguous=False)` contains all elements of the dataset whose index mod `n = i`.
|
| Be sure to shard before using any randomizing operator (such as `shuffle`).
| It is best if the shard operator is used early in the dataset pipeline.
|
| Args:
| num_shards (`int`):
| How many shards to split the dataset into.
| index (`int`):
| Which shard to select and return.
| contiguous: (`bool`, defaults to `True`):
| Whether to select contiguous blocks of indices for shards.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| indices_cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| indices of each shard instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| This only concerns the indices mapping.
| Number of indices per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 1066
| })
| >>> ds = ds.shard(num_shards=2, index=0)
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 533
| })
| ```
|
| shuffle(self, seed: Optional[int] = None, generator: Optional[numpy.random._generator.Generator] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Create a new Dataset where the rows are shuffled.
|
| Currently shuffling uses numpy random generators.
| You can either supply a NumPy BitGenerator to use, or a seed to initiate NumPy's default random generator (PCG64).
|
| Shuffling takes the list of indices `[0:len(my_dataset)]` and shuffles it to create an indices mapping.
| However as soon as your [`Dataset`] has an indices mapping, the speed can become 10x slower.
| This is because there is an extra step to get the row index to read using the indices mapping, and most importantly, you aren't reading contiguous chunks of data anymore.
| To restore the speed, you'd need to rewrite the entire dataset on your disk again using [`Dataset.flatten_indices`], which removes the indices mapping.
| This may take a lot of time depending of the size of your dataset though:
|
| ```python
| my_dataset[0] # fast
| my_dataset = my_dataset.shuffle(seed=42)
| my_dataset[0] # up to 10x slower
| my_dataset = my_dataset.flatten_indices() # rewrite the shuffled dataset on disk as contiguous chunks of data
| my_dataset[0] # fast again
| ```
|
| In this case, we recommend switching to an [`IterableDataset`] and leveraging its fast approximate shuffling method [`IterableDataset.shuffle`].
| It only shuffles the shards order and adds a shuffle buffer to your dataset, which keeps the speed of your dataset optimal:
|
| ```python
| my_iterable_dataset = my_dataset.to_iterable_dataset(num_shards=128)
| for example in enumerate(my_iterable_dataset): # fast
| pass
|
| shuffled_iterable_dataset = my_iterable_dataset.shuffle(seed=42, buffer_size=100)
|
| for example in enumerate(shuffled_iterable_dataset): # as fast as before
| pass
| ```
|
| Args:
| seed (`int`, *optional*):
| A seed to initialize the default BitGenerator if `generator=None`.
| If `None`, then fresh, unpredictable entropy will be pulled from the OS.
| If an `int` or `array_like[ints]` is passed, then it will be passed to SeedSequence to derive the initial BitGenerator state.
| generator (`numpy.random.Generator`, *optional*):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| keep_in_memory (`bool`, default `False`):
| Keep the shuffled indices in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the shuffled indices
| can be identified, use it instead of recomputing.
| indices_cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| shuffled indices instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
|
| # set a seed
| >>> shuffled_ds = ds.shuffle(seed=42)
| >>> shuffled_ds['label'][:10]
| [1, 0, 1, 1, 0, 0, 0, 0, 0, 0]
| ```
|
| skip(self, n: int) -> 'Dataset'
| Create a new [`Dataset`] that skips the first `n` elements.
|
| Args:
| n (`int`):
| Number of elements to skip.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| >>> ds = ds.skip(1)
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'},
| {'label': 1,
| 'text': 'if you sometimes like to go to the movies to have fun , wasabi is a good place to start .'}]
| ```
|
| sort(self, column_names: Union[str, collections.abc.Sequence[str]], reverse: Union[bool, collections.abc.Sequence[bool]] = False, null_placement: str = 'at_end', keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Create a new dataset sorted according to a single or multiple columns.
|
| Args:
| column_names (`Union[str, Sequence[str]]`):
| Column name(s) to sort by.
| reverse (`Union[bool, Sequence[bool]]`, defaults to `False`):
| If `True`, sort by descending order rather than ascending. If a single bool is provided,
| the value is applied to the sorting of all column names. Otherwise a list of bools with the
| same length and order as column_names must be provided.
| null_placement (`str`, defaults to `at_end`):
| Put `None` values at the beginning if `at_start` or `first` or at the end if `at_end` or `last`
|
| <Added version="1.14.2"/>
| keep_in_memory (`bool`, defaults to `False`):
| Keep the sorted indices in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the sorted indices
| can be identified, use it instead of recomputing.
| indices_cache_file_name (`str`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| sorted indices instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| Higher value gives smaller cache files, lower value consume less temporary memory.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset('cornell-movie-review-data/rotten_tomatoes', split='validation')
| >>> ds['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
| >>> sorted_ds = ds.sort('label')
| >>> sorted_ds['label'][:10]
| [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
| >>> another_sorted_ds = ds.sort(['label', 'text'], reverse=[True, False])
| >>> another_sorted_ds['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
| ```
|
| take(self, n: int) -> 'Dataset'
| Create a new [`Dataset`] with only the first `n` elements.
|
| Args:
| n (`int`):
| Number of elements to take.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> small_ds = ds.take(2)
| >>> list(small_ds)
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}]
| ```
|
| to_csv(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **to_csv_kwargs) -> int
| Exports the dataset to csv
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.csv`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.csv`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| num_proc (`int`, *optional*):
| Number of processes for multiprocessing. By default it doesn't
| use multiprocessing. `batch_size` in this case defaults to
| `datasets.config.DEFAULT_MAX_BATCH_SIZE` but feel free to make it 5x or 10x of the default
| value if you have sufficient compute power.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.19.0"/>
| **to_csv_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_csv`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html).
|
| <Changed version="2.10.0">
|
| Now, `index` defaults to `False` if not specified.
|
| If you would like to write the index, pass `index=True` and also set a name for the index column by
| passing `index_label`.
|
| </Changed>
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_csv("path/to/dataset/directory")
| ```
|
| to_dict(self, batch_size: Optional[int] = None, batched: bool = False) -> Union[dict, collections.abc.Iterator[dict]]
| Returns the dataset as a Python dict. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*): The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
|
| Returns:
| `dict` or `Iterator[dict]`
|
| Example:
|
| ```py
| >>> ds.to_dict()
| ```
|
| to_iterable_dataset(self, num_shards: Optional[int] = 1) -> 'IterableDataset'
| Get an [`datasets.IterableDataset`] from a map-style [`datasets.Dataset`].
| This is equivalent to loading a dataset in streaming mode with [`datasets.load_dataset`], but much faster since the data is streamed from local files.
|
| Contrary to map-style datasets, iterable datasets are lazy and can only be iterated over (e.g. using a for loop).
| Since they are read sequentially in training loops, iterable datasets are much faster than map-style datasets.
| All the transformations applied to iterable datasets like filtering or processing are done on-the-fly when you start iterating over the dataset.
|
| Still, it is possible to shuffle an iterable dataset using [`datasets.IterableDataset.shuffle`].
| This is a fast approximate shuffling that works best if you have multiple shards and if you specify a buffer size that is big enough.
|
| To get the best speed performance, make sure your dataset doesn't have an indices mapping.
| If this is the case, the data are not read contiguously, which can be slow sometimes.
| You can use `ds = ds.flatten_indices()` to write your dataset in contiguous chunks of data and have optimal speed before switching to an iterable dataset.
|
| Args:
| num_shards (`int`, default to `1`):
| Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly,
| and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example.
| Shards are defined using [`datasets.Dataset.shard`]: it simply slices the data without writing anything on disk.
|
| Returns:
| [`datasets.IterableDataset`]
|
| Example:
|
| Basic usage:
| ```python
| >>> ids = ds.to_iterable_dataset()
| >>> for example in ids:
| ... pass
| ```
|
| With lazy filtering and processing:
| ```python
| >>> ids = ds.to_iterable_dataset()
| >>> ids = ids.filter(filter_fn).map(process_fn) # will filter and process on-the-fly when you start iterating over the iterable dataset
| >>> for example in ids:
| ... pass
| ```
|
| With sharding to enable efficient shuffling:
| ```python
| >>> ids = ds.to_iterable_dataset(num_shards=64) # the dataset is split into 64 shards to be iterated over
| >>> ids = ids.shuffle(buffer_size=10_000) # will shuffle the shards order and use a shuffle buffer for fast approximate shuffling when you start iterating
| >>> for example in ids:
| ... pass
| ```
|
| With a PyTorch DataLoader:
| ```python
| >>> import torch
| >>> ids = ds.to_iterable_dataset(num_shards=64)
| >>> ids = ids.filter(filter_fn).map(process_fn)
| >>> dataloader = torch.utils.data.DataLoader(ids, num_workers=4) # will assign 64 / 4 = 16 shards to each worker to load, filter and process when you start iterating
| >>> for example in ids:
| ... pass
| ```
|
| With a PyTorch DataLoader and shuffling:
| ```python
| >>> import torch
| >>> ids = ds.to_iterable_dataset(num_shards=64)
| >>> ids = ids.shuffle(buffer_size=10_000) # will shuffle the shards order and use a shuffle buffer when you start iterating
| >>> dataloader = torch.utils.data.DataLoader(ids, num_workers=4) # will assign 64 / 4 = 16 shards from the shuffled list of shards to each worker when you start iterating
| >>> for example in ids:
| ... pass
| ```
|
| In a distributed setup like PyTorch DDP with a PyTorch DataLoader and shuffling
| ```python
| >>> from datasets.distributed import split_dataset_by_node
| >>> ids = ds.to_iterable_dataset(num_shards=512)
| >>> ids = ids.shuffle(buffer_size=10_000, seed=42) # will shuffle the shards order and use a shuffle buffer when you start iterating
| >>> ids = split_dataset_by_node(ds, world_size=8, rank=0) # will keep only 512 / 8 = 64 shards from the shuffled lists of shards when you start iterating
| >>> dataloader = torch.utils.data.DataLoader(ids, num_workers=4) # will assign 64 / 4 = 16 shards from this node's list of shards to each worker when you start iterating
| >>> for example in ids:
| ... pass
| ```
|
| With shuffling and multiple epochs:
| ```python
| >>> ids = ds.to_iterable_dataset(num_shards=64)
| >>> ids = ids.shuffle(buffer_size=10_000, seed=42) # will shuffle the shards order and use a shuffle buffer when you start iterating
| >>> for epoch in range(n_epochs):
| ... ids.set_epoch(epoch) # will use effective_seed = seed + epoch to shuffle the shards and for the shuffle buffer when you start iterating
| ... for example in ids:
| ... pass
| ```
| Feel free to also use [`IterableDataset.set_epoch`] when using a PyTorch DataLoader or in distributed setups.
|
| to_json(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **to_json_kwargs) -> int
| Export the dataset to JSON Lines or JSON.
|
| The default output format is [JSON Lines](https://jsonlines.org/).
| To export to [JSON](https://www.json.org), pass `lines=False` argument and the desired `orient`.
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.json`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.json`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| num_proc (`int`, *optional*):
| Number of processes for multiprocessing. By default, it doesn't
| use multiprocessing. `batch_size` in this case defaults to
| `datasets.config.DEFAULT_MAX_BATCH_SIZE` but feel free to make it 5x or 10x of the default
| value if you have sufficient compute power.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.19.0"/>
| **to_json_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_json`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html).
| Default arguments are `lines=True` and `orient="records".
|
| <Changed version="2.11.0">
|
| The parameter `index` defaults to `False` if `orient` is `"split"` or `"table"`.
|
| If you would like to write the index, pass `index=True`.
|
| </Changed>
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_json("path/to/dataset/directory/filename.jsonl")
| ```
|
| to_list(self) -> list
| Returns the dataset as a Python list.
|
| Returns:
| `list`
|
| Example:
|
| ```py
| >>> ds.to_list()
| ```
|
| to_pandas(self, batch_size: Optional[int] = None, batched: bool = False) -> Union[pandas.core.frame.DataFrame, collections.abc.Iterator[pandas.core.frame.DataFrame]]
| Returns the dataset as a `pandas.DataFrame`. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*):
| The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
|
| Returns:
| `pandas.DataFrame` or `Iterator[pandas.DataFrame]`
|
| Example:
|
| ```py
| >>> ds.to_pandas()
| ```
|
| to_parquet(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, storage_options: Optional[dict] = None, **parquet_writer_kwargs) -> int
| Exports the dataset to parquet
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.parquet`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.parquet`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| By default it aims for row groups with maximum uncompressed byte size of "100MB",
| defined by `datasets.config.MAX_ROW_GROUP_SIZE`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.19.0"/>
| **parquet_writer_kwargs (additional keyword arguments):
| Parameters to pass to PyArrow's `pyarrow.parquet.ParquetWriter`.
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_parquet("path/to/dataset/directory")
| ```
|
| to_polars(self, batch_size: Optional[int] = None, batched: bool = False, schema_overrides: Optional[dict] = None, rechunk: bool = True) -> Union[ForwardRef('pl.DataFrame'), collections.abc.Iterator['pl.DataFrame']]
| Returns the dataset as a `polars.DataFrame`. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*):
| The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
| schema_overrides (`dict`, *optional*):
| Support type specification or override of one or more columns; note that
| any dtypes inferred from the schema param will be overridden.
| rechunk (`bool`):
| Make sure that all data is in contiguous memory. Defaults to `True`.
| Returns:
| `polars.DataFrame` or `Iterator[polars.DataFrame]`
|
| Example:
|
| ```py
| >>> ds.to_polars()
| ```
|
| to_sql(self, name: str, con: Union[str, ForwardRef('sqlalchemy.engine.Connection'), ForwardRef('sqlalchemy.engine.Engine'), ForwardRef('sqlite3.Connection')], batch_size: Optional[int] = None, num_proc: Optional[int] = None, **sql_writer_kwargs) -> int
| Exports the dataset to a SQL database.
|
| Args:
| name (`str`):
| Name of SQL table.
| con (`str` or `sqlite3.Connection` or `sqlalchemy.engine.Connection` or `sqlalchemy.engine.Connection`):
| A [URI string](https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls) or a SQLite3/SQLAlchemy connection object used to write to a database.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| num_proc (`int`, *optional*):
| Number of processes for multiprocessing. By default, it doesn't
| use multiprocessing. `batch_size` in this case defaults to
| `datasets.config.DEFAULT_MAX_BATCH_SIZE` but feel free to make it 5x or 10x of the default
| value if you have sufficient compute power.
| **sql_writer_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_sql`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_sql.html).
|
| <Changed version="2.11.0">
|
| Now, `index` defaults to `False` if not specified.
|
| If you would like to write the index, pass `index=True` and also set a name for the index column by
| passing `index_label`.
|
| </Changed>
|
| Returns:
| `int`: The number of records written.
|
| Example:
|
| ```py
| >>> # con provided as a connection URI string
| >>> ds.to_sql("data", "sqlite:///my_own_db.sql")
| >>> # con provided as a sqlite3 connection object
| >>> import sqlite3
| >>> con = sqlite3.connect("my_own_db.sql")
| >>> with con:
| ... ds.to_sql("data", con)
| ```
|
| train_test_split(self, test_size: Union[float, int, NoneType] = None, train_size: Union[float, int, NoneType] = None, shuffle: bool = True, stratify_by_column: Optional[str] = None, seed: Optional[int] = None, generator: Optional[numpy.random._generator.Generator] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, train_indices_cache_file_name: Optional[str] = None, test_indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, train_new_fingerprint: Optional[str] = None, test_new_fingerprint: Optional[str] = None) -> 'DatasetDict'
| Return a dictionary ([`datasets.DatasetDict`]) with two random train and test subsets (`train` and `test` `Dataset` splits).
| Splits are created from the dataset according to `test_size`, `train_size` and `shuffle`.
|
| This method is similar to scikit-learn `train_test_split`.
|
| Args:
| test_size (`Union[float, int, None]`, *optional*):
| Size of the test split
| If `float`, should be between `0.0` and `1.0` and represent the proportion of the dataset to include in the test split.
| If `int`, represents the absolute number of test samples.
| If `None`, the value is set to the complement of the train size.
| If `train_size` is also `None`, it will be set to `0.25`.
| train_size (`Union[float, int, None]`, *optional*):
| Size of the train split
| If `float`, should be between `0.0` and `1.0` and represent the proportion of the dataset to include in the train split.
| If `int`, represents the absolute number of train samples.
| If `None`, the value is automatically set to the complement of the test size.
| shuffle (`bool`, *optional*, defaults to `True`):
| Whether or not to shuffle the data before splitting.
| stratify_by_column (`str`, *optional*, defaults to `None`):
| The column name of labels to be used to perform stratified split of data.
| seed (`int`, *optional*):
| A seed to initialize the default BitGenerator if `generator=None`.
| If `None`, then fresh, unpredictable entropy will be pulled from the OS.
| If an `int` or `array_like[ints]` is passed, then it will be passed to SeedSequence to derive the initial BitGenerator state.
| generator (`numpy.random.Generator`, *optional*):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| keep_in_memory (`bool`, defaults to `False`):
| Keep the splits indices in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the splits indices
| can be identified, use it instead of recomputing.
| train_cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| train split indices instead of the automatically generated cache file name.
| test_cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| test split indices instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| train_new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the train set after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
| test_new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the test set after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.train_test_split(test_size=0.2, shuffle=True)
| DatasetDict({
| train: Dataset({
| features: ['text', 'label'],
| num_rows: 852
| })
| test: Dataset({
| features: ['text', 'label'],
| num_rows: 214
| })
| })
|
| # set a seed
| >>> ds = ds.train_test_split(test_size=0.2, seed=42)
|
| # stratified split
| >>> ds = load_dataset("stanfordnlp/imdb",split="train")
| Dataset({
| features: ['text', 'label'],
| num_rows: 25000
| })
| >>> ds = ds.train_test_split(test_size=0.2, stratify_by_column="label")
| DatasetDict({
| train: Dataset({
| features: ['text', 'label'],
| num_rows: 20000
| })
| test: Dataset({
| features: ['text', 'label'],
| num_rows: 5000
| })
| })
| ```
|
| unique(self, column: str) -> list
| Return a list of the unique elements in a column.
|
| This is implemented in the low-level backend and as such, very fast.
|
| Args:
| column (`str`):
| Column name (list all the column names with [`~datasets.Dataset.column_names`]).
|
| Returns:
| `list`: List of unique elements in the given column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.unique('label')
| [1, 0]
| ```
|
| with_format(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| Set `__getitem__` return format (type and columns). The data formatting is applied on-the-fly.
| The format `type` (for example "numpy") is used to format batches when using `__getitem__`.
|
| It's also possible to use custom transforms for formatting using [`~datasets.Dataset.with_transform`].
|
| Contrary to [`~datasets.Dataset.set_format`], `with_format` returns a new [`Dataset`] object.
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`List[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds.format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': None}
| >>> ds = ds.with_format("torch")
| >>> ds.format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'torch'}
| >>> ds[0]
| {'text': 'compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'label': tensor(1),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
| 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| with_transform(self, transform: Optional[Callable], columns: Optional[list] = None, output_all_columns: bool = False)
| Set `__getitem__` return format using this transform. The transform is applied on-the-fly on batches when `__getitem__` is called.
|
| As [`~datasets.Dataset.set_format`], this can be reset using [`~datasets.Dataset.reset_format`].
|
| Contrary to [`~datasets.Dataset.set_transform`], `with_transform` returns a new [`Dataset`] object.
|
| Args:
| transform (`Callable`, `optional`):
| User-defined formatting transform, replaces the format defined by [`~datasets.Dataset.set_format`].
| A formatting function is a callable that takes a batch (as a `dict`) as input and returns a batch.
| This function is applied right before returning the objects in `__getitem__`.
| columns (`List[str]`, `optional`):
| Columns to format in the output.
| If specified, then the input batch of the transform only contains those columns.
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| If set to `True`, then the other un-formatted columns are kept with the output of the transform.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> def encode(example):
| ... return tokenizer(example["text"], padding=True, truncation=True, return_tensors='pt')
| >>> ds = ds.with_transform(encode)
| >>> ds[0]
| {'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1]),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0])}
| ```
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_buffer(buffer: pyarrow.lib.Buffer, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, indices_buffer: Optional[pyarrow.lib.Buffer] = None) -> 'Dataset'
| Instantiate a Dataset backed by an Arrow buffer.
|
| Args:
| buffer (`pyarrow.Buffer`):
| Arrow buffer.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| indices_buffer (`pyarrow.Buffer`, *optional*):
| Indices Arrow buffer.
|
| Returns:
| [`Dataset`]
|
| from_dict(mapping: dict, features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, on_mixed_types: Optional[Literal['use_json']] = None) -> 'Dataset'
| Convert `dict` to a `pyarrow.Table` to create a [`Dataset`].
|
| Important: a dataset created with from_dict() lives in memory
| and therefore doesn't have an associated cache directory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. save_to_disk / load_from_disk.
|
| Args:
| mapping (`Mapping`):
| Mapping of strings to Arrays or Python lists.
| features ([`Features`], *optional*):
| Dataset features.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| on_mixed_types (`Literal["use_json"]`, *optional*, defaults to `None`):
| If "use_json", use the Json() type for mixed-types fields,
| i.e. unstructured fields that contain data without a predefined schema.
| In this case, a field with mixed type is set to Json().
|
| This allow loading lists with a mix of strings/integers/floats
| for example, or dictionaries with arbitrary value types.
|
| <Added version="4.7.0"/>
|
| Returns:
| [`Dataset`]
|
| Examples:
|
| Get a Dataset from a dictionary containing one list per column:
|
| ```py
| >>> ds = Dataset.from_dict({"text": ["hello there !", "general kenobi !"]})
| ```
|
| Pass features to set the column types, e.g. for an image dataset:
|
| ```py
| >>> features = Features({"image": Image()})
| >>> ds = Dataset.from_dict({"image": ["path/to/image.png"]}, features=features)
| ```
|
| Datasets are based on Arrow which is a columnar format, and therefore they expect every example to have the same
| type and subtypes, and dictionaries to have the same keys and values types.
| Loading a dataset errors out when fields have mismatching types, and fills missing fields in dictionaries with None so all dictionaries have the same keys and value types.
|
| To avoid this and allow mixed-types without errors, you can use `on_mixed_types="use_json"` or specify `features=` with a [`Json`] type:
|
| ```py
| >>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]})
| Traceback (most recent call last):
| ...
| File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
| pyarrow.lib.ArrowInvalid: Could not convert 'foo' with type str: tried to convert to int64
|
| >>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]}, on_mixed_types="use_json")
| >>> ds.features
| {'a': Json()}
| >>> list(ds["a"])
| [0, "foo", {"subfield": "bar"}]
|
| >>> features = Features({"a": Json()})
| >>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]}, features=features)
| >>> ds.features
| {'a': Json()}
| >>> list(ds["a"])
| [0, "foo", {"subfield": "bar"}]
| ```
|
| This is also useful for lists of dictionaries with arbitrary keys and values, to avoid filling missing fields with None:
|
| ```py
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]})
| >>> ds.features
| {'a': List({'b': Value('int64'), 'c': Value('int64')})}
| >>> list(ds["a"])
| [[{'b': 0, 'c': None}, {'b': None, 'c': 0}]] # missing fields are filled with None
|
| >>> features = Features({"a": List(Json())})
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, features=features)
| >>> ds.features
| {'a': List(Json())}
| >>> list(ds["a"])
| [[{'b': 0}, {'c': 0}]] # OK
|
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, on_mixed_types="use_json")
| >>> ds.features
| {'a': List(Json())}
| >>> list(ds["a"])
| [[{'b': 0}, {'c': 0}]] # OK
| ```
|
| Another example with tool calling data:
|
| ```py
| >>> messages = [
| ... {"role": "user", "content": "Turn on the living room lights and play my electronic music playlist."},
| ... {"role": "assistant", "tool_calls": [
| ... {"type": "function", "function": {
| ... "name": "control_light",
| ... "arguments": {"room": "living room", "state": "on"}
| ... }},
| ... {"type": "function", "function": {
| ... "name": "play_music",
| ... "arguments": {"playlist": "electronic"} # mixed-type here since keys ["playlist"] and ["room", "state"] are different
| ... }}]
| ... },
| ... {"role": "tool", "name": "control_light", "content": "The lights in the living room are now on."},
| ... {"role": "tool", "name": "play_music", "content": "The music is now playing."},
| ... {"role": "assistant", "content": "Done!"}
| ... ]
| >>> ds = Dataset.from_dict({"messages": [messages]}, on_mixed_types="use_json")
| >>> ds.features
| {'messages': List({'role': Value('string'), 'content': Value('string'), 'tool_calls': List(Json()), 'name': Value('string')})}
| >>> ds[0]["messages"][1]["tool_calls"][0]["function"]["arguments"]
| {"room": "living room", "state": "on"}
| ```
|
| from_file(filename: str, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, indices_filename: Optional[str] = None, in_memory: bool = False) -> 'Dataset'
| Instantiate a Dataset backed by an Arrow table at filename.
|
| Args:
| filename (`str`):
| File name of the dataset.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| indices_filename (`str`, *optional*):
| File names of the indices.
| in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
|
| Returns:
| [`Dataset`]
|
| from_list(mapping: list[dict], features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, on_mixed_types: Optional[Literal['use_json']] = None) -> 'Dataset'
| Convert a list of dicts to a `pyarrow.Table` to create a [`Dataset`]`.
|
| Note that the keys of the first entry will be used to determine the dataset columns,
| regardless of what is passed to features.
|
| Important: a dataset created with from_list() lives in memory
| and therefore doesn't have an associated cache directory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. save_to_disk / load_from_disk.
|
| Args:
| mapping (`List[dict]`): A list of mappings of strings to row values.
| features (`Features`, optional): Dataset features.
| info (`DatasetInfo`, optional): Dataset information, like description, citation, etc.
| split (`NamedSplit`, optional): Name of the dataset split.
| on_mixed_types (`Literal["use_json"]`, *optional*, defaults to `None`):
| If "use_json", use the Json() type for mixed-types fields,
| i.e. unstructured fields that contain data without a predefined schema.
| In this case, a field with mixed type is set to Json().
|
| This allow loading lists with a mix of strings/integers/floats
| for example, or dictionaries with arbitrary value types.
|
| <Added version="4.7.0"/>
|
| Returns:
| [`Dataset`]
|
| Examples:
|
| Get a Dataset from a list containing the examples:
|
| ```py
| >>> ds = Dataset.from_list([{"text": "hello there !"}, {"text": "general kenobi !"}]})
| ```
|
| Pass features to set the column types, e.g. for an image dataset:
|
| ```py
| >>> features = Features({"image": Image()})
| >>> ds = Dataset.from_list([{"image": "path/to/image.png"}], features=features)
| ```
|
| Datasets are based on Arrow which is a columnar format, and therefore they expect every example to have the same
| type and subtypes, and dictionaries to have the same keys and values types.
| Loading a dataset errors out when fields have mismatching types, and fills missing fields in dictionaries with None so all dictionaries have the same keys and value types.
|
| To avoid this and allow mixed-types without errors, you can use `on_mixed_types="use_json"` or specify `features=` with a [`Json`] type:
|
| ```py
| >>> ds = Dataset.from_list([{"a": 0}, {"a": "foo"}, {"a": {"subfield": "bar"}}])
| Traceback (most recent call last):
| ...
| File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
| pyarrow.lib.ArrowInvalid: Could not convert 'foo' with type str: tried to convert to int64
|
| >>> ds = Dataset.from_list([{"a": 0}, {"a": "foo"}, {"a": {"subfield": "bar"}}], on_mixed_types="use_json")
| >>> ds.features
| {'a': Json()}
| >>> list(ds["a"])
| [0, "foo", {"subfield": "bar"}]
|
| >>> features = Features({"a": Json()})
| >>> ds = Dataset.from_list([{"a": 0}, {"a": "foo"}, {"a": {"subfield": "bar"}}], features=features)
| >>> ds.features
| {'a': Json()}
| >>> list(ds["a"])
| [0, "foo", {"subfield": "bar"}]
| ```
|
| This is also useful for lists of dictionaries with arbitrary keys and values, to avoid filling missing fields with None:
|
| ```py
| >>> ds = Dataset.from_list([{"a": [{"b": 0}, {"c": 0}]}])
| >>> ds.features
| {'a': List({'b': Value('int64'), 'c': Value('int64')})}
| >>> list(ds["a"])
| [[{'b': 0, 'c': None}, {'b': None, 'c': 0}]] # missing fields are filled with None
|
| >>> features = Features({"a": List(Json())})
| >>> ds = Dataset.from_list([{"a": [{"b": 0}, {"c": 0}]}], features=features)
| >>> ds.features
| {'a': List(Json())}
| >>> list(ds["a"])
| [[{'b': 0}, {'c': 0}]] # OK
|
| >>> ds = Dataset.from_list([{"a": [{"b": 0}, {"c": 0}]}], on_mixed_types="use_json")
| >>> ds.features
| {'a': List(Json())}
| >>> list(ds["a"])
| [[{'b': 0}, {'c': 0}]] # OK
| ```
|
| Another example with tool calling data:
|
| ```py
| >>> messages = [
| ... {"role": "user", "content": "Turn on the living room lights and play my electronic music playlist."},
| ... {"role": "assistant", "tool_calls": [
| ... {"type": "function", "function": {
| ... "name": "control_light",
| ... "arguments": {"room": "living room", "state": "on"}
| ... }},
| ... {"type": "function", "function": {
| ... "name": "play_music",
| ... "arguments": {"playlist": "electronic"} # mixed-type here since keys ["playlist"] and ["room", "state"] are different
| ... }}]
| ... },
| ... {"role": "tool", "name": "control_light", "content": "The lights in the living room are now on."},
| ... {"role": "tool", "name": "play_music", "content": "The music is now playing."},
| ... {"role": "assistant", "content": "Done!"}
| ... ]
| >>> ds = Dataset.from_list([{"messages": messages}], on_mixed_types="use_json")
| >>> ds.features
| {'messages': List({'role': Value('string'), 'content': Value('string'), 'tool_calls': List(Json()), 'name': Value('string')})}
| >>> ds[0]["messages"][1]["tool_calls"][0]["function"]["arguments"]
| {"room": "living room", "state": "on"}
| ```
|
| from_pandas(df: pandas.core.frame.DataFrame, features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, preserve_index: Optional[bool] = None) -> 'Dataset'
| Convert `pandas.DataFrame` to a `pyarrow.Table` to create a [`Dataset`].
|
| The column types in the resulting Arrow Table are inferred from the dtypes of the `pandas.Series` in the
| DataFrame. In the case of non-object Series, the NumPy dtype is translated to its Arrow equivalent. In the
| case of `object`, we need to guess the datatype by looking at the Python objects in this Series.
|
| Be aware that Series of the `object` dtype don't carry enough information to always lead to a meaningful Arrow
| type. In the case that we cannot infer a type, e.g. because the DataFrame is of length 0 or the Series only
| contains `None/nan` objects, the type is set to `null`. This behavior can be avoided by constructing explicit
| features and passing it to this function.
|
| Important: a dataset created with from_pandas() lives in memory
| and therefore doesn't have an associated cache directory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. save_to_disk / load_from_disk.
|
| Args:
| df (`pandas.DataFrame`):
| Dataframe that contains the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| preserve_index (`bool`, *optional*):
| Whether to store the index as an additional column in the resulting Dataset.
| The default of `None` will store the index as a column, except for `RangeIndex` which is stored as metadata only.
| Use `preserve_index=True` to force it to be stored as a column.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_pandas(df)
| ```
|
| from_polars(df: Union[ForwardRef('pl.DataFrame'), ForwardRef('pl.LazyFrame')], features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None) -> 'Dataset'
| Collect the underlying arrow arrays in an Arrow Table.
|
| This operation is mostly zero copy.
|
| Data types that do copy:
| * CategoricalType
|
| Args:
| df (`polars.DataFrame`): DataFrame to convert to Arrow Table
| features (`Features`, optional): Dataset features.
| info (`DatasetInfo`, optional): Dataset information, like description, citation, etc.
| split (`NamedSplit`, optional): Name of the dataset split.
|
| Examples:
| ```py
| >>> ds = Dataset.from_polars(df)
| ```
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| from_csv(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, num_proc: Optional[int] = None, **kwargs) -> 'Dataset'
| Create Dataset from CSV file(s).
|
| Read the CSV files, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the CSV file(s).
| split ([`NamedSplit`], *optional*):
| Split name to be assigned to the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`pandas.read_csv`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_csv('path/to/dataset.csv')
| ```
|
| from_generator(generator: Callable, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, gen_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, split: datasets.splits.NamedSplit = NamedSplit('train'), fingerprint: Optional[str] = None, **kwargs) -> 'Dataset'
| Create a Dataset from a generator.
|
| Load the data from the generator, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| generator (:`Callable`):
| A generator function that `yields` examples.
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| gen_kwargs(`dict`, *optional*):
| Keyword arguments to be passed to the `generator` callable.
| You can define a sharded dataset by passing the list of shards in `gen_kwargs` and setting `num_proc` greater than 1.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
| If `num_proc` is greater than one, then all list values in `gen_kwargs` must be the same length. These values will be split between calls to the generator. The number of shards will be the minimum of the shortest list in `gen_kwargs` and `num_proc`.
|
| <Added version="2.7.0"/>
| split ([`NamedSplit`], defaults to `Split.TRAIN`):
| Split name to be assigned to the dataset.
|
| <Added version="2.21.0"/>
| fingerprint (`str`, *optional*):
| Fingerprint that will be used to generate dataset ID.
| By default `fingerprint` is generated by hashing the generator function and all the args which can be slow
| if it uses large objects like AI models.
|
| <Added version="4.3.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to :[`GeneratorConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> def gen():
| ... yield {"text": "Good", "label": 0}
| ... yield {"text": "Bad", "label": 1}
| ...
| >>> ds = Dataset.from_generator(gen)
| ```
|
| ```py
| >>> def gen(shards):
| ... for shard in shards:
| ... with open(shard) as f:
| ... for line in f:
| ... yield {"line": line}
| ...
| >>> shards = [f"data{i}.txt" for i in range(32)]
| >>> ds = Dataset.from_generator(gen, gen_kwargs={"shards": shards})
| ```
|
| from_json(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, field: Optional[str] = None, num_proc: Optional[int] = None, **kwargs) -> 'Dataset'
| Create Dataset from JSON or JSON Lines file(s).
|
| Read the JSON files, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the JSON or JSON Lines file(s).
| split ([`NamedSplit`], *optional*):
| Split name to be assigned to the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| field (`str`, *optional*):
| Field name of the JSON file where the dataset is contained in.
| num_proc (`int`, *optional* defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`JsonConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_json('path/to/dataset.json')
| ```
|
| from_parquet(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, columns: Optional[list[str]] = None, num_proc: Optional[int] = None, filters: Union[pyarrow._compute.Expression, list[tuple], list[list[tuple]], NoneType] = None, fragment_scan_options: Optional[pyarrow._dataset_parquet.ParquetFragmentScanOptions] = None, on_bad_files: Literal['error', 'warn', 'skip'] = 'error', **kwargs) -> 'Dataset'
| Create Dataset from Parquet file(s).
|
| Read the Parquet files, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the Parquet file(s).
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| columns (`List[str]`, *optional*):
| If not `None`, only these columns will be read from the file.
| A column name may be a prefix of a nested field, e.g. 'a' will select
| 'a.b', 'a.c', and 'a.d.e'.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| filters (`Union[pyarrow.dataset.Expression, list[tuple], list[list[tuple]]]`, *optional*):
| Return only the rows matching the filter.
| If possible the predicate will be pushed down to exploit the partition information
| or internal metadata found in the data source, e.g. Parquet statistics.
| Otherwise filters the loaded RecordBatches before yielding them.
| fragment_scan_options (`pyarrow.dataset.ParquetFragmentScanOptions`, *optional*)
| Scan-specific options for Parquet fragments.
| This is especially useful to configure buffering and caching.
|
| <Added version="4.2.0"/>
| on_bad_files (`Literal["error", "warn", "skip"]`, *optional*, defaults to "error")
| Specify what to do upon encountering a bad file (a file that can't be read). Allowed values are :
| * 'error', raise an Exception when a bad file is encountered.
| * 'warn', raise a warning when a bad file is encountered and skip that file.
| * 'skip', skip bad files without raising or warning when they are encountered.
|
| <Added version="4.2.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`ParquetConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_parquet('path/to/dataset.parquet')
| ```
|
| Load a subset of columns:
|
| ```python
| >>> ds = Dataset.from_parquet('path/to/dataset.parquet', columns=["col_0", "col_1"])
| ```
|
| Efficiently filter data, possibly skipping entire files or row groups:
|
| ```python
| >>> filters = [("col_0", "==", 0)]
| >>> ds = Dataset.from_parquet(parquet_files_list, filters=filters)
| ```
|
| from_spark(df: 'pyspark.sql.DataFrame', split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, cache_dir: str = None, working_dir: str = None, load_from_cache_file: bool = True, **kwargs) -> 'Dataset'
| Create a Dataset from Spark DataFrame. Dataset downloading is distributed over Spark workers.
|
| Read the Spark DataFrame, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| df (`pyspark.sql.DataFrame`):
| The DataFrame containing the desired data.
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data. When using a multi-node Spark cluster, the cache_dir must be accessible to both
| workers and the driver.
| keep_in_memory (`bool`):
| Whether to copy the data in-memory.
| working_dir (`str`, *optional*)
| Intermediate directory for each Spark worker to write data to before moving it to `cache_dir`. Setting
| a non-NFS intermediate directory may improve performance.
| load_from_cache_file (`bool`):
| Whether to load the dataset from the cache if possible.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> df = spark.createDataFrame(
| >>> data=[[1, "Elia"], [2, "Teo"], [3, "Fang"]],
| >>> columns=["id", "name"],
| >>> )
| >>> ds = Dataset.from_spark(df)
| ```
|
| from_sql(sql: Union[str, ForwardRef('sqlalchemy.sql.Selectable')], con: Union[str, ForwardRef('sqlalchemy.engine.Connection'), ForwardRef('sqlalchemy.engine.Engine'), ForwardRef('sqlite3.Connection')], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, **kwargs) -> 'Dataset'
| Create Dataset from SQL query or database table.
|
| Query the SQL database, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| sql (`str` or `sqlalchemy.sql.Selectable`):
| SQL query to be executed or a table name.
| con (`str` or `sqlite3.Connection` or `sqlalchemy.engine.Connection` or `sqlalchemy.engine.Connection`):
| A [URI string](https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls) used to instantiate a database connection or a SQLite3/SQLAlchemy connection object.
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`SqlConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> # Fetch a database table
| >>> ds = Dataset.from_sql("test_data", "postgres:///db_name")
| >>> # Execute a SQL query on the table
| >>> ds = Dataset.from_sql("SELECT sentence FROM test_data", "postgres:///db_name")
| >>> # Use a Selectable object to specify the query
| >>> from sqlalchemy import select, text
| >>> stmt = select([text("sentence")]).select_from(text("test_data"))
| >>> ds = Dataset.from_sql(stmt, "postgres:///db_name")
| ```
|
| > [!TIP]
| > The returned dataset can only be cached if `con` is specified as URI string.
|
| from_text(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, num_proc: Optional[int] = None, keep_linebreaks: bool = False, sample_by: Literal['line', 'paragraph', 'document'] = 'line', **kwargs) -> 'Dataset'
| Create Dataset from text file(s).
|
| Read the text files, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the text file(s).
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| keep_linebreaks: (`bool`, defaults to False):
| Whether to keep line breaks.
| sample_by (`Literal["line", "paragraph", "document"]`, defaults to "line"):
| Whether to load data per line, praragraph or document.
| By default one row in the dataset = one line.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`TextConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_text('path/to/dataset.txt')
| ```
|
| load_from_disk(dataset_path: Union[str, bytes, os.PathLike], keep_in_memory: Optional[bool] = None, storage_options: Optional[dict] = None) -> 'Dataset'
| Loads a dataset that was previously saved using [`save_to_disk`] from a dataset directory, or from a
| filesystem using any implementation of `fsspec.spec.AbstractFileSystem`.
|
| Args:
| dataset_path (`path-like`):
| Path (e.g. `"dataset/train"`) or remote URI (e.g. `"s3//my-bucket/dataset/train"`)
| of the dataset directory where the dataset will be loaded from.
| keep_in_memory (`bool`, defaults to `None`):
| Whether to copy the dataset in-memory. If `None`, the
| dataset will not be copied in-memory unless explicitly enabled by setting
| `datasets.config.IN_MEMORY_MAX_SIZE` to nonzero. See more details in the
| [improve performance](../cache#improve-performance) section.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.8.0"/>
|
| Returns:
| [`Dataset`] or [`DatasetDict`]:
| - If `dataset_path` is a path of a dataset directory, the dataset requested.
| - If `dataset_path` is a path of a dataset dict directory, a `datasets.DatasetDict` with each split.
|
| Example:
|
| ```py
| >>> ds = load_from_disk("path/to/dataset/directory")
| ```
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| cache_files
| The cache files containing the Apache Arrow table backing the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.cache_files
| [{'filename': '/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46/rotten_tomatoes_movie_review-validation.arrow'}]
| ```
|
| column_names
| Names of the columns in the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.column_names
| ['text', 'label']
| ```
|
| data
| The Apache Arrow table backing the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.data
| MemoryMappedTable
| text: string
| label: int64
| ----
| text: [["compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .","the soundtrack alone is worth the price of admission .","rodriguez does a splendid job of racial profiling hollywood style--casting excellent latin actors of all ages--a trend long overdue .","beneath the film's obvious determination to shock at any cost lies considerable skill and determination , backed by sheer nerve .","bielinsky is a filmmaker of impressive talent .","so beautifully acted and directed , it's clear that washington most certainly has a new career ahead of him if he so chooses .","a visual spectacle full of stunning images and effects .","a gentle and engrossing character study .","it's enough to watch huppert scheming , with her small , intelligent eyes as steady as any noir villain , and to enjoy the perfectly pitched web of tension that chabrol spins .","an engrossing portrait of uncompromising artists trying to create something original against the backdrop of a corporate music industry that only seems to care about the bottom line .",...,"ultimately , jane learns her place as a girl , softens up and loses some of the intensity that made her an interesting character to begin with .","ah-nuld's action hero days might be over .","it's clear why deuces wild , which was shot two years ago , has been gathering dust on mgm's shelf .","feels like nothing quite so much as a middle-aged moviemaker's attempt to surround himself with beautiful , half-naked women .","when the precise nature of matthew's predicament finally comes into sharp focus , the revelation fails to justify the build-up .","this picture is murder by numbers , and as easy to be bored by as your abc's , despite a few whopping shootouts .","hilarious musical comedy though stymied by accents thick as mud .","if you are into splatter movies , then you will probably have a reasonably good time with the salton sea .","a dull , simple-minded and stereotypical tale of drugs , death and mind-numbing indifference on the inner-city streets .","the feature-length stretch . . . strains the show's concept ."]]
| label: [[1,1,1,1,1,1,1,1,1,1,...,0,0,0,0,0,0,0,0,0,0]]
| ```
|
| features
|
| format
|
| num_columns
| Number of columns in the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.num_columns
| 2
| ```
|
| num_rows
| Number of rows in the dataset (same as [`Dataset.__len__`]).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.num_rows
| 1066
| ```
|
| shape
| Shape of the dataset (number of columns, number of rows).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.shape
| (1066, 2)
| ```
|
| ----------------------------------------------------------------------
| Readonly properties inherited from DatasetInfoMixin:
|
| builder_name
|
| citation
|
| config_name
|
| dataset_size
|
| description
|
| download_checksums
|
| download_size
|
| homepage
|
| info
| [`~datasets.DatasetInfo`] object containing all the metadata in the dataset.
|
| license
|
| size_in_bytes
|
| split
| [`~datasets.NamedSplit`] object corresponding to a named dataset split.
|
| supervised_keys
|
| version
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DatasetInfoMixin:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from datasets.search.IndexableMixin:
|
| drop_index(self, index_name: str)
| Drop the index with the specified column.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index.
|
| get_index(self, index_name: str) -> datasets.search.BaseIndex
| List the `index_name`/identifiers of all the attached indexes.
|
| Args:
| index_name (`str`): Index name.
|
| Returns:
| [`BaseIndex`]
|
| get_nearest_examples(self, index_name: str, query: Union[str, <built-in function array>], k: int = 10, **kwargs) -> datasets.search.NearestExamplesResults
| Find the nearest examples in the dataset to the query.
|
| Args:
| index_name (`str`):
| The index_name/identifier of the index.
| query (`Union[str, np.ndarray]`):
| The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve.
|
| Returns:
| `(scores, examples)`:
| A tuple of `(scores, examples)` where:
| - **scores** (`List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples
| - **examples** (`dict`): the retrieved examples
|
| get_nearest_examples_batch(self, index_name: str, queries: Union[list[str], <built-in function array>], k: int = 10, **kwargs) -> datasets.search.BatchedNearestExamplesResults
| Find the nearest examples in the dataset to the query.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index.
| queries (`Union[List[str], np.ndarray]`):
| The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve per query.
|
| Returns:
| `(total_scores, total_examples)`:
| A tuple of `(total_scores, total_examples)` where:
| - **total_scores** (`List[List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples per query
| - **total_examples** (`List[dict]`): the retrieved examples per query
|
| is_index_initialized(self, index_name: str) -> bool
|
| list_indexes(self) -> list[str]
| List the `colindex_nameumns`/identifiers of all the attached indexes.
|
| load_elasticsearch_index(self, index_name: str, es_index_name: str, host: Optional[str] = None, port: Optional[int] = None, es_client: Optional[ForwardRef('Elasticsearch')] = None, es_index_config: Optional[dict] = None)
| Load an existing text index using ElasticSearch for fast retrieval.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index. This is the index name that is used to call `get_nearest` or `search`.
| es_index_name (`str`):
| The name of elasticsearch index to load.
| host (`str`, *optional*, defaults to `localhost`):
| Host of where ElasticSearch is running.
| port (`str`, *optional*, defaults to `9200`):
| Port of where ElasticSearch is running.
| es_client (`elasticsearch.Elasticsearch`, *optional*):
| The elasticsearch client used to create the index if host and port are `None`.
| es_index_config (`dict`, *optional*):
| The configuration of the elasticsearch index.
| Default config is:
| ```
| {
| "settings": {
| "number_of_shards": 1,
| "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},
| },
| "mappings": {
| "properties": {
| "text": {
| "type": "text",
| "analyzer": "standard",
| "similarity": "BM25"
| },
| }
| },
| }
| ```
|
| load_faiss_index(self, index_name: str, file: Union[str, pathlib.PurePath], device: Union[int, list[int], NoneType] = None, storage_options: Optional[dict] = None)
| Load a FaissIndex from disk.
|
| If you want to do additional configurations, you can have access to the faiss index object by doing
| `.get_index(index_name).faiss_index` to make it fit your needs.
|
| Args:
| index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to
| call `.get_nearest` or `.search`.
| file (`str`): The path to the serialized faiss index on disk or remote URI (e.g. `"s3://my-bucket/index.faiss"`).
| device (Optional `Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs.
| If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.11.0"/>
|
| save_faiss_index(self, index_name: str, file: Union[str, pathlib.PurePath], storage_options: Optional[dict] = None)
| Save a FaissIndex on disk.
|
| Args:
| index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`.
| file (`str`): The path to the serialized faiss index on disk or remote URI (e.g. `"s3://my-bucket/index.faiss"`).
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.11.0"/>
|
| search(self, index_name: str, query: Union[str, <built-in function array>], k: int = 10, **kwargs) -> datasets.search.SearchResults
| Find the nearest examples indices in the dataset to the query.
|
| Args:
| index_name (`str`):
| The name/identifier of the index.
| query (`Union[str, np.ndarray]`):
| The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve.
|
| Returns:
| `(scores, indices)`:
| A tuple of `(scores, indices)` where:
| - **scores** (`List[List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples
| - **indices** (`List[List[int]]`): the indices of the retrieved examples
|
| search_batch(self, index_name: str, queries: Union[list[str], <built-in function array>], k: int = 10, **kwargs) -> datasets.search.BatchedSearchResults
| Find the nearest examples indices in the dataset to the query.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index.
| queries (`Union[List[str], np.ndarray]`):
| The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve per query.
|
| Returns:
| `(total_scores, total_indices)`:
| A tuple of `(total_scores, total_indices)` where:
| - **total_scores** (`List[List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples per query
| - **total_indices** (`List[List[int]]`): the indices of the retrieved examples per query
|
| ----------------------------------------------------------------------
| Methods inherited from TensorflowDatasetMixin:
|
| to_tf_dataset(self, batch_size: Optional[int] = None, columns: Union[str, list[str], NoneType] = None, shuffle: bool = False, collate_fn: Optional[Callable] = None, drop_remainder: bool = False, collate_fn_args: Optional[dict[str, Any]] = None, label_cols: Union[str, list[str], NoneType] = None, prefetch: bool = True, num_workers: int = 0, num_test_batches: int = 20)
| Create a `tf.data.Dataset` from the underlying Dataset. This `tf.data.Dataset` will load and collate batches from
| the Dataset, and is suitable for passing to methods like `model.fit()` or `model.predict()`. The dataset will yield
| `dicts` for both inputs and labels unless the `dict` would contain only a single key, in which case a raw
| `tf.Tensor` is yielded instead.
|
| Args:
| batch_size (`int`, *optional*):
| Size of batches to load from the dataset. Defaults to `None`, which implies that the dataset won't be
| batched, but the returned dataset can be batched later with `tf_dataset.batch(batch_size)`.
| columns (`List[str]` or `str`, *optional*):
| Dataset column(s) to load in the `tf.data.Dataset`.
| Column names that are created by the `collate_fn` and that do not exist in the original dataset can be used.
| shuffle(`bool`, defaults to `False`):
| Shuffle the dataset order when loading. Recommended `True` for training, `False` for
| validation/evaluation.
| drop_remainder(`bool`, defaults to `False`):
| Drop the last incomplete batch when loading. Ensures
| that all batches yielded by the dataset will have the same length on the batch dimension.
| collate_fn(`Callable`, *optional*):
| A function or callable object (such as a `DataCollator`) that will collate
| lists of samples into a batch.
| collate_fn_args (`Dict`, *optional*):
| An optional `dict` of keyword arguments to be passed to the
| `collate_fn`.
| label_cols (`List[str]` or `str`, defaults to `None`):
| Dataset column(s) to load as labels.
| Note that many models compute loss internally rather than letting Keras do it, in which case
| passing the labels here is optional, as long as they're in the input `columns`.
| prefetch (`bool`, defaults to `True`):
| Whether to run the dataloader in a separate thread and maintain
| a small buffer of batches for training. Improves performance by allowing data to be loaded in the
| background while the model is training.
| num_workers (`int`, defaults to `0`):
| Number of workers to use for loading the dataset.
| num_test_batches (`int`, defaults to `20`):
| Number of batches to use to infer the output signature of the dataset.
| The higher this number, the more accurate the signature will be, but the longer it will take to
| create the dataset.
|
| Returns:
| `tf.data.Dataset`
|
| Example:
|
| ```py
| >>> ds_train = ds["train"].to_tf_dataset(
| ... columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'],
| ... shuffle=True,
| ... batch_size=16,
| ... collate_fn=data_collator,
| ... )
| ```
datasets.DatasetBuilder
Python Library Documentation: class DatasetBuilder in module datasets.builder
class DatasetBuilder(builtins.object)
| DatasetBuilder(cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
|
| Abstract base class for all datasets.
|
| `DatasetBuilder` has 3 key methods:
|
| - [`DatasetBuilder.info`]: Documents the dataset, including feature
| names, types, shapes, version, splits, citation, etc.
| - [`DatasetBuilder.download_and_prepare`]: Downloads the source data
| and writes it to disk.
| - [`DatasetBuilder.as_dataset`]: Generates a [`Dataset`].
|
| Some `DatasetBuilder`s expose multiple variants of the
| dataset by defining a [`BuilderConfig`] subclass and accepting a
| config object (or name) on construction. Configurable datasets expose a
| pre-defined set of configurations in [`DatasetBuilder.builder_configs`].
|
| Args:
| cache_dir (`str`, *optional*):
| Directory to cache data. Defaults to `"~/.cache/huggingface/datasets"`.
| dataset_name (`str`, *optional*):
| Name of the dataset, if different from the builder name. Useful for packaged builders
| like csv, imagefolder, audiofolder, etc. to reflect the difference between datasets
| that use the same packaged builder.
| config_name (`str`, *optional*):
| Name of the dataset configuration.
| It affects the data generated on disk. Different configurations will have their own subdirectories and
| versions.
| If not provided, the default configuration is used (if it exists).
|
| <Added version="2.3.0">
|
| Parameter `name` was renamed to `config_name`.
|
| </Added>
| hash (`str`, *optional*):
| Hash specific to the dataset builder code. Used to update the caching directory when the
| dataset builder code is updated (to avoid reusing old data).
| The typical caching directory (defined in `self._relative_data_dir`) is `name/version/hash/`.
| base_path (`str`, *optional*):
| Base path for relative paths that are used to download files.
| This can be a remote URL.
| features ([`Features`], *optional*):
| Features types to use with this dataset.
| It can be used to change the [`Features`] types of a dataset, for example.
| token (`str` or `bool`, *optional*):
| String or boolean to use as Bearer token for remote files on the
| Datasets Hub. If `True`, will get token from `"~/.huggingface"`.
| repo_id (`str`, *optional*):
| ID of the dataset repository.
| Used to distinguish builders with the same name but not coming from the same namespace, for example "rajpurkar/squad"
| and "lhoestq/squad" repo IDs. In the latter, the builder name would be "lhoestq___squad".
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| For builders like "csv" or "json" that need the user to specify data files. They can be either
| local or remote files. For convenience, you can use a `DataFilesDict`.
| data_dir (`str`, *optional*):
| Path to directory containing source data file(s).
| Use only if `data_files` is not passed, in which case it is equivalent to passing
| `os.path.join(data_dir, "**")` as `data_files`.
| For builders that require manual download, it must be the path to the local directory containing the
| manually downloaded data.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the dataset file-system backend, if any.
| writer_batch_size (`int`, *optional*):
| Batch size used by the ArrowWriter.
| It defines the number of samples that are kept in memory before writing them
| and also the length of the arrow chunks.
| None means that the ArrowWriter will use its default value.
| **config_kwargs (additional keyword arguments): Keyword arguments to be passed to the corresponding builder
| configuration class, set on the class attribute [`DatasetBuilder.BUILDER_CONFIG_CLASS`]. The builder
| configuration class is [`BuilderConfig`] or a subclass of it.
|
| Methods defined here:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setstate__(self, d)
|
| as_dataset(self, split: Union[str, datasets.splits.Split, list[str], list[datasets.splits.Split], NoneType] = None, in_memory=False) -> Union[datasets.arrow_dataset.Dataset, datasets.dataset_dict.DatasetDict]
| Return a Dataset for the specified split.
|
| Args:
| split (`datasets.Split`):
| Which subset of the data to return.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the
| downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
|
| Returns:
| datasets.Dataset
|
| Example:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')
| >>> builder.download_and_prepare()
| >>> ds = builder.as_dataset(split='train')
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 8530
| })
| ```
|
| as_streaming_dataset(self, split: Optional[str] = None, base_path: Optional[str] = None) -> Union[dict[str, datasets.iterable_dataset.IterableDataset], datasets.iterable_dataset.IterableDataset]
|
| download_and_prepare(self, output_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, verification_mode: Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None, dl_manager: Optional[datasets.download.download_manager.DownloadManager] = None, base_path: Optional[str] = None, file_format: str = 'arrow', max_shard_size: Union[str, int, NoneType] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **download_and_prepare_kwargs)
| Downloads and prepares dataset for reading.
|
| Args:
| output_dir (`str`, *optional*):
| Output directory for the dataset.
| Default to this builder's `cache_dir`, which is inside `~/.cache/huggingface/datasets` by default.
|
| <Added version="2.5.0"/>
| download_config (`DownloadConfig`, *optional*):
| Specific download configuration parameters.
| download_mode ([`DownloadMode`] or `str`, *optional*):
| Select the download/generate mode, default to `REUSE_DATASET_IF_EXISTS`.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| dl_manager (`DownloadManager`, *optional*):
| Specific `DownloadManger` to use.
| base_path (`str`, *optional*):
| Base path for relative paths that are used to download files. This can be a remote url.
| If not specified, the value of the `base_path` attribute (`self.base_path`) will be used instead.
| file_format (`str`, *optional*):
| Format of the data files in which the dataset will be written.
| Supported formats: "arrow", "parquet". Default to "arrow" format.
| If the format is "parquet", then image and audio data are embedded into the Parquet files instead of pointing to local files.
|
| <Added version="2.5.0"/>
| max_shard_size (`Union[str, int]`, *optional*):
| Maximum number of bytes written per shard, default is "500MB".
| The size is based on uncompressed data size, so in practice your shard files may be smaller than
| `max_shard_size` thanks to Parquet compression for example.
|
| <Added version="2.5.0"/>
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.7.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the caching file-system backend, if any.
|
| <Added version="2.5.0"/>
| **download_and_prepare_kwargs (additional keyword arguments): Keyword arguments.
|
| Example:
|
| Download and prepare the dataset as Arrow files that can be loaded as a Dataset using `builder.as_dataset()`:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare()
| ```
|
| Download and prepare the dataset as sharded Parquet files locally:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("./output_dir", file_format="parquet")
| ```
|
| Download and prepare the dataset as sharded Parquet files in a cloud storage:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> storage_options = {"key": aws_access_key_id, "secret": aws_secret_access_key}
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("s3://my-bucket/my_rotten_tomatoes", storage_options=storage_options, file_format="parquet")
| ```
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| get_imported_module_dir()
| Return the path of the module of this class or subclass.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| builder_configs
| Dictionary of pre-defined configurations for this builder class.
|
| cache_dir
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| BUILDER_CONFIGS = []
|
| BUILDER_CONFIG_CLASS = <class 'datasets.builder.BuilderConfig'>
| Base class for `DatasetBuilder` data configuration.
|
| `DatasetBuilder` subclasses with data configuration options should subclass
| `BuilderConfig` and add their own properties.
|
| Attributes:
| name (`str`, defaults to `default`):
| The name of the configuration.
| version (`Version` or `str`, defaults to `0.0.0`):
| The version of the configuration.
| data_dir (`str`, *optional*):
| Path to the directory containing the source data.
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| description (`str`, *optional*):
| A human description of the configuration.
|
|
| DEFAULT_CONFIG_NAME = None
|
| DEFAULT_WRITER_BATCH_SIZE = None
|
| SLEEP_ON_THREADS_SHUTDOWNS = False
|
| VERSION = None
datasets.DatasetDict
Python Library Documentation: class DatasetDict in module datasets.dataset_dict
class DatasetDict(builtins.dict)
| A dictionary (dict of str: datasets.Dataset) with dataset transforms methods (map, filter, etc.)
|
| Method resolution order:
| DatasetDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| __getitem__(self, k) -> datasets.arrow_dataset.Dataset
| Return self[key].
|
| __repr__(self)
| Return repr(self).
|
| align_labels_with_mapping(self, label2id: dict, label_column: str) -> 'DatasetDict'
| Align the dataset's label ID and label name mapping to match an input `label2id` mapping.
| This is useful when you want to ensure that a model's predicted labels are aligned with the dataset.
| The alignment in done using the lowercase label names.
|
| Args:
| label2id (`dict`):
| The label name to ID mapping to align the dataset with.
| label_column (`str`):
| The column name of labels to align on.
|
| Example:
|
| ```python
| >>> # dataset with mapping {'entailment': 0, 'neutral': 1, 'contradiction': 2}
| >>> ds = load_dataset("nyu-mll/glue", "mnli", split="train")
| >>> # mapping to align with
| >>> label2id = {'CONTRADICTION': 0, 'NEUTRAL': 1, 'ENTAILMENT': 2}
| >>> ds_aligned = ds.align_labels_with_mapping(label2id, "label")
| ```
|
| cast(self, features: datasets.features.features.Features) -> 'DatasetDict'
| Cast the dataset to a new set of features.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Args:
| features ([`Features`]):
| New features to cast the dataset to.
| The name and order of the fields in the features must match the current column names.
| The type of the data must also be convertible from one type to the other.
| For non-trivial conversion, e.g. `string` <-> `ClassLabel` you should use [`~DatasetDict.map`] to update the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel, Value
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds["train"].features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> new_features = ds["train"].features.copy()
| >>> new_features['label'] = ClassLabel(names=['bad', 'good'])
| >>> new_features['text'] = Value('large_string')
| >>> ds = ds.cast(new_features)
| >>> ds["train"].features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('large_string')}
| ```
|
| cast_column(self, column: str, feature) -> 'DatasetDict'
| Cast column to feature for decoding.
|
| Args:
| column (`str`):
| Column name.
| feature ([`Feature`]):
| Target feature.
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds["train"].features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> ds = ds.cast_column('label', ClassLabel(names=['bad', 'good']))
| >>> ds["train"].features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('string')}
| ```
|
| class_encode_column(self, column: str, include_nulls: bool = False) -> 'DatasetDict'
| Casts the given column as [`~datasets.features.ClassLabel`] and updates the tables.
|
| Args:
| column (`str`):
| The name of the column to cast.
| include_nulls (`bool`, defaults to `False`):
| Whether to include null values in the class labels. If `True`, the null values will be encoded as the `"None"` class label.
|
| <Added version="1.14.2"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("google/boolq")
| >>> ds["train"].features
| {'answer': Value('bool'),
| 'passage': Value('string'),
| 'question': Value('string')}
| >>> ds = ds.class_encode_column("answer")
| >>> ds["train"].features
| {'answer': ClassLabel(num_classes=2, names=['False', 'True']),
| 'passage': Value('string'),
| 'question': Value('string')}
| ```
|
| cleanup_cache_files(self) -> dict[str, int]
| Clean up all cache files in the dataset cache directory, excepted the currently used cache file if there is one.
| Be careful when running this command that no other process is currently using other cache files.
|
| Return:
| `Dict` with the number of removed files for each split
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.cleanup_cache_files()
| {'test': 0, 'train': 0, 'validation': 0}
| ```
|
| filter(self, function: Optional[Callable] = None, with_indices: bool = False, with_rank: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, desc: Optional[str] = None) -> 'DatasetDict'
| Apply a filter function to all the elements in the table in batches
| and update the table so that the dataset only includes examples according to the filter function.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Args:
| function (`Callable`): Callable with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> bool` if `batched=False` and `with_indices=False` and `with_rank=False`
| - `function(example: Dict[str, Any], *extra_args) -> bool` if `batched=False` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
| - `function(batch: Dict[str, list]) -> list[bool]` if `batched=True` and `with_indices=False` and `with_rank=False`
| - `function(batch: Dict[str, list], *extra_args) -> list[bool]` if `batched=True` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
|
| If no function is provided, defaults to an always `True` function: `lambda x: True`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the
| signature of `function` should be `def function(example, idx[, rank]): ...`.
| with_rank (`bool`, defaults to `False`):
| Provide process rank to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], rank): ...`.
| input_columns (`[Union[str, list[str]]]`, *optional*, defaults to `None`):
| The columns to be passed into `function` as
| positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`
| `batch_size <= 0` or `batch_size == None` then provide the full dataset as a single batch to `function`.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_names (`[Dict[str, str]]`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`
| num_proc (`int`, *optional*, defaults to `None`):
| The number of processes to use for multiprocessing.
| - If `None` or `0`, no multiprocessing is used and the operation runs in the main process.
| - If greater than `1`, one or multiple worker processes are used to process data in parallel.
| Note: The function passed to `map()` must be picklable for multiprocessing to work correctly
| (i.e., prefer functions defined at the top level of a module, not inside another function or class).
| desc (`str`, *optional*, defaults to `None`):
| Meaningful description to be displayed alongside with the progress bar while filtering examples.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.filter(lambda x: x["label"] == 1)
| DatasetDict({
| train: Dataset({
| features: ['text', 'label'],
| num_rows: 4265
| })
| validation: Dataset({
| features: ['text', 'label'],
| num_rows: 533
| })
| test: Dataset({
| features: ['text', 'label'],
| num_rows: 533
| })
| })
| ```
|
| flatten(self, max_depth=16) -> 'DatasetDict'
| Flatten the Apache Arrow Table of each split (nested features are flatten).
| Each column with a struct type is flattened into one column per struct field.
| Other columns are left unchanged.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("rajpurkar/squad")
| >>> ds["train"].features
| {'id': Value('string'),
| 'title': Value('string'),
| 'context': Value('string'),
| 'question': Value('string'),
| 'answers.text': List(Value('string')),
| 'answers.answer_start': List(Value('int32'))}
| >>> ds.flatten()
| DatasetDict({
| train: Dataset({
| features: ['id', 'title', 'context', 'question', 'answers.text', 'answers.answer_start'],
| num_rows: 87599
| })
| validation: Dataset({
| features: ['id', 'title', 'context', 'question', 'answers.text', 'answers.answer_start'],
| num_rows: 10570
| })
| })
| ```
|
| flatten_indices(self, keep_in_memory: bool = False, cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000, features: Optional[datasets.features.features.Features] = None, disable_nullable: bool = False, num_proc: Optional[int] = None, new_fingerprint: Optional[str] = None) -> 'DatasetDict'
| Create and cache a new Dataset by flattening the indices mapping.
|
| Args:
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| cache_file_names (`Dict[str, str]`, *optional*, default `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| features (`Optional[datasets.Features]`, defaults to `None`):
| Use a specific [`Features`] to store the cache file
| instead of the automatically generated one.
| disable_nullable (`bool`, defaults to `False`):
| Allow null values in the table.
| num_proc (`int`, optional, default `None`):
| Max number of processes when generating cache. Already cached shards are loaded sequentially
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
|
| formatted_as(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| To be used in a `with` statement. Set `__getitem__` return format (type and columns).
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`list[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to False):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| map(self, function: Optional[Callable] = None, with_indices: bool = False, with_rank: bool = False, with_split: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, drop_last_batch: bool = False, remove_columns: Union[str, list[str], NoneType] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000, features: Optional[datasets.features.features.Features] = None, disable_nullable: bool = False, fn_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, desc: Optional[str] = None, try_original_type: Optional[bool] = True, on_mixed_types: Optional[Literal['use_json']] = None) -> 'DatasetDict'
| Apply a function to all the examples in the table (individually or in batches) and update the table.
| If your function returns a column that already exists, then it overwrites it.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| You can specify whether the function should be batched or not with the `batched` parameter:
|
| - If batched is `False`, then the function takes 1 example in and should return 1 example.
| An example is a dictionary, e.g. `{"text": "Hello there !"}`.
| - If batched is `True` and `batch_size` is 1, then the function takes a batch of 1 example as input and can return a batch with 1 or more examples.
| A batch is a dictionary, e.g. a batch of 1 example is `{"text": ["Hello there !"]}`.
| - If batched is `True` and `batch_size` is `n > 1`, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.
| Note that the last batch may have less than `n` examples.
| A batch is a dictionary, e.g. a batch of `n` examples is `{"text": ["Hello there !"] * n}`.
|
| If the function is asynchronous, then `map` will run your function in parallel, with up to one thousand simultaneous calls.
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`callable`): with one of the following signature:
| - `function(example: Dict[str, Any]) -> Dict[str, Any]` if `batched=False` and `with_indices=False`
| - `function(example: Dict[str, Any], indices: int) -> Dict[str, Any]` if `batched=False` and `with_indices=True`
| - `function(batch: Dict[str, list]) -> Dict[str, list]` if `batched=True` and `with_indices=False`
| - `function(batch: Dict[str, list], indices: list[int]) -> Dict[str, list]` if `batched=True` and `with_indices=True`
|
| For advanced usage, the function can also return a `pyarrow.Table`.
| If the function is asynchronous, then `map` will run your function in parallel.
| Moreover if your function returns nothing (`None`), then `map` will run your function and return the dataset unchanged.
| If no function is provided, default to identity function: `lambda x: x`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx): ...`.
| with_rank (`bool`, defaults to `False`):
| Provide process rank to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], rank): ...`.
| with_split (`bool`, defaults to `False`):
| Provide process split to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], split): ...`.
| input_columns (`[Union[str, list[str]]]`, *optional*, defaults to `None`):
| The columns to be passed into `function` as
| positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`,
| `batch_size <= 0` or `batch_size == None` then provide the full dataset as a single batch to `function`.
| drop_last_batch (`bool`, defaults to `False`):
| Whether a last batch smaller than the batch_size should be
| dropped instead of being processed by the function.
| remove_columns (`[Union[str, list[str]]]`, *optional*, defaults to `None`):
| Remove a selection of columns while doing the mapping.
| Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding
| columns with names in `remove_columns`, these columns will be kept.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_names (`[Dict[str, str]]`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, default `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| features (`[datasets.Features]`, *optional*, defaults to `None`):
| Use a specific [`Features`] to store the cache file
| instead of the automatically generated one.
| disable_nullable (`bool`, defaults to `False`):
| Disallow null values in the table.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`
| num_proc (`int`, *optional*, defaults to `None`):
| The number of processes to use for multiprocessing.
| - If `None` or `0`, no multiprocessing is used and the operation runs in the main process.
| - If greater than `1`, one or multiple worker processes are used to process data in parallel.
| Note: The function passed to `map()` must be picklable for multiprocessing to work correctly
| (i.e., prefer functions defined at the top level of a module, not inside another function or class).
| desc (`str`, *optional*, defaults to `None`):
| Meaningful description to be displayed alongside with the progress bar while mapping examples.
| try_original_type (`Optional[bool]`, defaults to `True`):
| Try to keep the types of the original columns (e.g. int32 -> int32).
| Set to False if you want to always infer new types.
| on_mixed_types (`Literal["use_json"]`, *optional*, defaults to `None`):
| If "use_json", use the Json() type for mixed-types fields,
| i.e. unstructured fields that contain data without a predefined schema.
| In this case, a field with mixed type is set to Json().
|
| This allow loading lists with a mix of strings/integers/floats
| for example, or dictionaries with arbitrary value types.
|
| <Added version="4.7.0"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> def add_prefix(example):
| ... example["text"] = "Review: " + example["text"]
| ... return example
| >>> ds = ds.map(add_prefix)
| >>> ds["train"][0:3]["text"]
| ['Review: the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',
| 'Review: the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .',
| 'Review: effective but too-tepid biopic']
|
| # process a batch of examples
| >>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
| # set number of processors
| >>> ds = ds.map(add_prefix, num_proc=4)
| ```
|
| push_to_hub(self, repo_id: str, config_name: str = 'default', set_default: Optional[bool] = None, data_dir: Optional[str] = None, commit_message: Optional[str] = None, commit_description: Optional[str] = None, private: Optional[bool] = None, token: Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = False, max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[dict[str, int]] = None, embed_external_files: bool = True, num_proc: Optional[int] = None) -> Optional[huggingface_hub.hf_api.CommitInfo]
| Pushes the [`DatasetDict`] to the hub as a Parquet dataset.
| The [`DatasetDict`] is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
|
| Each dataset split will be pushed independently. The pushed dataset will keep the original split names.
|
| The resulting Parquet files are self-contained by default: if your dataset contains [`Image`] or [`Audio`]
| data, the Parquet files will store the bytes of your images or audio files.
| You can disable this by setting `embed_external_files` to False.
|
| Args:
| repo_id (`str`):
| The ID of the repository to push to in the following format: `<user>/<dataset_name>` or
| `<org>/<dataset_name>`. Also accepts `<dataset_name>`, which will default to the namespace
| of the logged-in user.
|
| It could also be a location inside a bucket, e.g. `buckets/<user_or_org>/<bucket_name>/...`
| config_name (`str`):
| Configuration name of a dataset. Defaults to "default".
| set_default (`bool`, *optional*):
| Whether to set this configuration as the default one. Otherwise, the default configuration is the one
| named "default".
| data_dir (`str`, *optional*):
| Directory name that will contain the uploaded data files. Defaults to the `config_name` if different
| from "default", else "data".
|
| <Added version="2.17.0"/>
| commit_message (`str`, *optional*):
| Message to commit while pushing. Will default to `"Upload dataset"`.
| commit_description (`str`, *optional*):
| Description of the commit that will be created.
| Additionally, description of the PR if a PR is created (`create_pr` is True).
|
| <Added version="2.16.0"/>
| private (`bool`, *optional*):
| Whether to make the repo private. If `None` (default), the repo will be public unless the
| organization's default is private. This value is ignored if the repo already exists.
| token (`str`, *optional*):
| An optional authentication token for the Hugging Face Hub. If no token is passed, will default
| to the token saved locally when logging in with `huggingface-cli login`. Will raise an error
| if no token is passed and the user is not logged-in.
| revision (`str`, *optional*):
| Branch to push the uploaded files to. Defaults to the `"main"` branch.
|
| <Added version="2.15.0"/>
| create_pr (`bool`, *optional*, defaults to `False`):
| Whether to create a PR with the uploaded files or directly commit.
|
| <Added version="2.15.0"/>
| max_shard_size (`int` or `str`, *optional*, defaults to `"500MB"`):
| The maximum size of the dataset shards to be uploaded to the hub. If expressed as a string, needs to be digits followed by a unit
| (like `"500MB"` or `"1GB"`).
| num_shards (`Dict[str, int]`, *optional*):
| Number of shards to write. By default, the number of shards depends on `max_shard_size`.
| Use a dictionary to define a different num_shards for each split.
|
| <Added version="2.8.0"/>
| embed_external_files (`bool`, defaults to `True`):
| Whether to embed file bytes in the shards.
| In particular, this will do the following before the push for the fields of type:
|
| - [`Audio`] and [`Image`] removes local path information and embed file content in the Parquet files.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when preparing and uploading the dataset.
| This is helpful if the dataset is made of many samples or media files to embed.
| I uses "spawn" context to work with hf_xet, the rust client for fast uploads to HF.
| Multiprocessing is disabled by default.
|
| <Added version="4.0.0"/>
|
| Return:
| huggingface_hub.CommitInfo
|
| Example:
|
| ```python
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", private=True)
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", max_shard_size="1GB")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", num_shards={"train": 1024, "test": 8})
| ```
|
| If you want to add a new configuration (or subset) to a dataset (e.g. if the dataset has multiple tasks/versions/languages):
|
| ```python
| >>> english_dataset.push_to_hub("<organization>/<dataset_id>", "en")
| >>> french_dataset.push_to_hub("<organization>/<dataset_id>", "fr")
| >>> # later
| >>> english_dataset = load_dataset("<organization>/<dataset_id>", "en")
| >>> french_dataset = load_dataset("<organization>/<dataset_id>", "fr")
| ```
|
| remove_columns(self, column_names: Union[str, list[str]]) -> 'DatasetDict'
| Remove one or several column(s) from each split in the dataset
| and the features associated to the column(s).
|
| The transformation is applied to all the splits of the dataset dictionary.
|
| You can also remove a column using [`~DatasetDict.map`] with `remove_columns` but the present method
| doesn't copy the data of the remaining columns and is thus faster.
|
| Args:
| column_names (`Union[str, list[str]]`):
| Name of the column(s) to remove.
|
| Returns:
| [`DatasetDict`]: A copy of the dataset object without the columns to remove.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds = ds.remove_columns("label")
| DatasetDict({
| train: Dataset({
| features: ['text'],
| num_rows: 8530
| })
| validation: Dataset({
| features: ['text'],
| num_rows: 1066
| })
| test: Dataset({
| features: ['text'],
| num_rows: 1066
| })
| })
| ```
|
| rename_column(self, original_column_name: str, new_column_name: str) -> 'DatasetDict'
| Rename a column in the dataset and move the features associated to the original column under the new column name.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| You can also rename a column using [`~DatasetDict.map`] with `remove_columns` but the present method:
| - takes care of moving the original features under the new column name.
| - doesn't copy the data to a new dataset and is thus much faster.
|
| Args:
| original_column_name (`str`):
| Name of the column to rename.
| new_column_name (`str`):
| New name for the column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds = ds.rename_column("label", "label_new")
| DatasetDict({
| train: Dataset({
| features: ['text', 'label_new'],
| num_rows: 8530
| })
| validation: Dataset({
| features: ['text', 'label_new'],
| num_rows: 1066
| })
| test: Dataset({
| features: ['text', 'label_new'],
| num_rows: 1066
| })
| })
| ```
|
| rename_columns(self, column_mapping: dict[str, str]) -> 'DatasetDict'
| Rename several columns in the dataset, and move the features associated to the original columns under
| the new column names.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Args:
| column_mapping (`Dict[str, str]`):
| A mapping of columns to rename to their new names.
|
| Returns:
| [`DatasetDict`]: A copy of the dataset with renamed columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.rename_columns({'text': 'text_new', 'label': 'label_new'})
| DatasetDict({
| train: Dataset({
| features: ['text_new', 'label_new'],
| num_rows: 8530
| })
| validation: Dataset({
| features: ['text_new', 'label_new'],
| num_rows: 1066
| })
| test: Dataset({
| features: ['text_new', 'label_new'],
| num_rows: 1066
| })
| })
| ```
|
| reset_format(self)
| Reset `__getitem__` return format to python objects and all columns.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Same as `self.set_format()`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x["text"], truncation=True, padding=True), batched=True)
| >>> ds.set_format(type="numpy", columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
| >>> ds["train"].format
| {'columns': ['input_ids', 'token_type_ids', 'attention_mask', 'label'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'numpy'}
| >>> ds.reset_format()
| >>> ds["train"].format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': None}
| ```
|
| save_to_disk(self, dataset_dict_path: Union[str, bytes, os.PathLike], max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[dict[str, int]] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None)
| Saves a dataset dict to a filesystem using `fsspec.spec.AbstractFileSystem`.
|
| For [`Image`], [`Audio`] and [`Video`] data:
|
| All the Image(), Audio() and Video() data are stored in the arrow files.
| If you want to store paths or urls, please use the Value("string") type.
|
| Args:
| dataset_dict_path (`path-like`):
| Path (e.g. `dataset/train`) or remote URI (e.g. `s3://my-bucket/dataset/train`)
| of the dataset dict directory where the dataset dict will be saved to.
| max_shard_size (`int` or `str`, *optional*, defaults to `"500MB"`):
| The maximum size of the dataset shards to be saved to the filesystem. If expressed as a string, needs to be digits followed by a unit
| (like `"50MB"`).
| num_shards (`Dict[str, int]`, *optional*):
| Number of shards to write. By default the number of shards depends on `max_shard_size` and `num_proc`.
| You need to provide the number of shards for each dataset in the dataset dictionary.
| Use a dictionary to define a different num_shards for each split.
|
| <Added version="2.8.0"/>
| num_proc (`int`, *optional*, default `None`):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.8.0"/>
|
| Example:
|
| ```python
| >>> dataset_dict.save_to_disk("path/to/dataset/directory")
| >>> dataset_dict.save_to_disk("path/to/dataset/directory", max_shard_size="1GB")
| >>> dataset_dict.save_to_disk("path/to/dataset/directory", num_shards={"train": 1024, "test": 8})
| ```
|
| select_columns(self, column_names: Union[str, list[str]]) -> 'DatasetDict'
| Select one or several column(s) from each split in the dataset and
| the features associated to the column(s).
|
| The transformation is applied to all the splits of the dataset
| dictionary.
|
| Args:
| column_names (`Union[str, list[str]]`):
| Name of the column(s) to keep.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.select_columns("text")
| DatasetDict({
| train: Dataset({
| features: ['text'],
| num_rows: 8530
| })
| validation: Dataset({
| features: ['text'],
| num_rows: 1066
| })
| test: Dataset({
| features: ['text'],
| num_rows: 1066
| })
| })
| ```
|
| set_format(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| Set `__getitem__` return format (type and columns).
| The format is set for every dataset in the dataset dictionary.
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`list[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to False):
| Keep un-formatted columns as well in the output (as python objects),
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| It is possible to call `map` after calling `set_format`. Since `map` may add new columns, then the list of formatted columns
| gets updated. In this case, if you apply `map` on a dataset to add a new column, then this column will be formatted:
|
| `new formatted columns = (all columns - previously unformatted columns)`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x["text"], truncation=True, padding=True), batched=True)
| >>> ds.set_format(type="numpy", columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
| >>> ds["train"].format
| {'columns': ['input_ids', 'token_type_ids', 'attention_mask', 'label'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'numpy'}
| ```
|
| set_transform(self, transform: Optional[Callable], columns: Optional[list] = None, output_all_columns: bool = False)
| Set ``__getitem__`` return format using this transform. The transform is applied on-the-fly on batches when ``__getitem__`` is called.
| The transform is set for every dataset in the dataset dictionary
| As :func:`datasets.Dataset.set_format`, this can be reset using :func:`datasets.Dataset.reset_format`
|
| Args:
| transform (`Callable`, optional): user-defined formatting transform, replaces the format defined by :func:`datasets.Dataset.set_format`
| A formatting function is a callable that takes a batch (as a dict) as input and returns a batch.
| This function is applied right before returning the objects in ``__getitem__``.
| columns (`list[str]`, optional): columns to format in the output
| If specified, then the input batch of the transform only contains those columns.
| output_all_columns (`bool`, default to False): keep un-formatted columns as well in the output (as python objects)
| If set to True, then the other un-formatted columns are kept with the output of the transform.
|
| shuffle(self, seeds: Union[int, dict[str, Optional[int]], NoneType] = None, seed: Optional[int] = None, generators: Optional[dict[str, numpy.random._generator.Generator]] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, indices_cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000) -> 'DatasetDict'
| Create a new Dataset where the rows are shuffled.
|
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Currently shuffling uses numpy random generators.
| You can either supply a NumPy BitGenerator to use, or a seed to initiate NumPy's default random generator (PCG64).
|
| Args:
| seeds (`Dict[str, int]` or `int`, *optional*):
| A seed to initialize the default BitGenerator if `generator=None`.
| If `None`, then fresh, unpredictable entropy will be pulled from the OS.
| If an `int` or `array_like[ints]` is passed, then it will be passed to SeedSequence to derive the initial BitGenerator state.
| You can provide one `seed` per dataset in the dataset dictionary.
| seed (`int`, *optional*):
| A seed to initialize the default BitGenerator if `generator=None`. Alias for seeds (a `ValueError` is raised if both are provided).
| generators (`Dict[str, *optional*, np.random.Generator]`):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| You have to provide one `generator` per dataset in the dataset dictionary.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| indices_cache_file_names (`Dict[str, str]`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| indices mappings instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds["train"]["label"][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
|
| # set a seed
| >>> shuffled_ds = ds.shuffle(seed=42)
| >>> shuffled_ds["train"]["label"][:10]
| [0, 1, 0, 1, 0, 0, 0, 0, 0, 0]
| ```
|
| sort(self, column_names: Union[str, collections.abc.Sequence[str]], reverse: Union[bool, collections.abc.Sequence[bool]] = False, null_placement: str = 'at_end', keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, indices_cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000) -> 'DatasetDict'
| Create a new dataset sorted according to a single or multiple columns.
|
| Args:
| column_names (`Union[str, Sequence[str]]`):
| Column name(s) to sort by.
| reverse (`Union[bool, Sequence[bool]]`, defaults to `False`):
| If `True`, sort by descending order rather than ascending. If a single bool is provided,
| the value is applied to the sorting of all column names. Otherwise a list of bools with the
| same length and order as column_names must be provided.
| null_placement (`str`, defaults to `at_end`):
| Put `None` values at the beginning if `at_start` or `first` or at the end if `at_end` or `last`
| keep_in_memory (`bool`, defaults to `False`):
| Keep the sorted indices in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the sorted indices
| can be identified, use it instead of recomputing.
| indices_cache_file_names (`[Dict[str, str]]`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| indices mapping instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| Higher value gives smaller cache files, lower value consume less temporary memory.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset('cornell-movie-review-data/rotten_tomatoes')
| >>> ds['train']['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
| >>> sorted_ds = ds.sort('label')
| >>> sorted_ds['train']['label'][:10]
| [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
| >>> another_sorted_ds = ds.sort(['label', 'text'], reverse=[True, False])
| >>> another_sorted_ds['train']['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
| ```
|
| unique(self, column: str) -> dict[str, list]
| Return a list of the unique elements in a column for each split.
|
| This is implemented in the low-level backend and as such, very fast.
|
| Args:
| column (`str`):
| column name (list all the column names with [`~datasets.DatasetDict.column_names`])
|
| Returns:
| Dict[`str`, `list`]: Dictionary of unique elements in the given column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.unique("label")
| {'test': [1, 0], 'train': [1, 0], 'validation': [1, 0]}
| ```
|
| with_format(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs) -> 'DatasetDict'
| Set `__getitem__` return format (type and columns). The data formatting is applied on-the-fly.
| The format `type` (for example "numpy") is used to format batches when using `__getitem__`.
| The format is set for every dataset in the dataset dictionary.
|
| It's also possible to use custom transforms for formatting using [`~datasets.Dataset.with_transform`].
|
| Contrary to [`~datasets.DatasetDict.set_format`], `with_format` returns a new [`DatasetDict`] object with new [`Dataset`] objects.
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`list[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds["train"].format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': None}
| >>> ds = ds.with_format("torch")
| >>> ds["train"].format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'torch'}
| >>> ds["train"][0]
| {'text': 'compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'label': tensor(1),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
| 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| with_transform(self, transform: Optional[Callable], columns: Optional[list] = None, output_all_columns: bool = False) -> 'DatasetDict'
| Set `__getitem__` return format using this transform. The transform is applied on-the-fly on batches when `__getitem__` is called.
| The transform is set for every dataset in the dataset dictionary
|
| As [`~datasets.Dataset.set_format`], this can be reset using [`~datasets.Dataset.reset_format`].
|
| Contrary to [`~datasets.DatasetDict.set_transform`], `with_transform` returns a new [`DatasetDict`] object with new [`Dataset`] objects.
|
| Args:
| transform (`Callable`, *optional*):
| User-defined formatting transform, replaces the format defined by [`~datasets.Dataset.set_format`].
| A formatting function is a callable that takes a batch (as a dict) as input and returns a batch.
| This function is applied right before returning the objects in `__getitem__`.
| columns (`list[str]`, *optional*):
| Columns to format in the output.
| If specified, then the input batch of the transform only contains those columns.
| output_all_columns (`bool`, defaults to False):
| Keep un-formatted columns as well in the output (as python objects).
| If set to `True`, then the other un-formatted columns are kept with the output of the transform.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> def encode(example):
| ... return tokenizer(example['text'], truncation=True, padding=True, return_tensors="pt")
| >>> ds = ds.with_transform(encode)
| >>> ds["train"][0]
| {'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 1, 1, 1, 1]),
| 'input_ids': tensor([ 101, 1103, 2067, 1110, 17348, 1106, 1129, 1103, 6880, 1432,
| 112, 188, 1207, 107, 14255, 1389, 107, 1105, 1115, 1119,
| 112, 188, 1280, 1106, 1294, 170, 24194, 1256, 3407, 1190,
| 170, 11791, 5253, 188, 1732, 7200, 10947, 12606, 2895, 117,
| 179, 7766, 118, 172, 15554, 1181, 3498, 6961, 3263, 1137,
| 188, 1566, 7912, 14516, 6997, 119, 102]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| from_csv(path_or_paths: dict[str, typing.Union[str, bytes, os.PathLike]], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, **kwargs) -> 'DatasetDict'
| Create [`DatasetDict`] from CSV file(s).
|
| Args:
| path_or_paths (`dict` of path-like):
| Path(s) of the CSV file(s).
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (str, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`pandas.read_csv`].
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import DatasetDict
| >>> ds = DatasetDict.from_csv({'train': 'path/to/dataset.csv'})
| ```
|
| from_json(path_or_paths: dict[str, typing.Union[str, bytes, os.PathLike]], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, **kwargs) -> 'DatasetDict'
| Create [`DatasetDict`] from JSON Lines file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the JSON Lines file(s).
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (str, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`JsonConfig`].
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import DatasetDict
| >>> ds = DatasetDict.from_json({'train': 'path/to/dataset.json'})
| ```
|
| from_parquet(path_or_paths: dict[str, typing.Union[str, bytes, os.PathLike]], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, columns: Optional[list[str]] = None, **kwargs) -> 'DatasetDict'
| Create [`DatasetDict`] from Parquet file(s).
|
| Args:
| path_or_paths (`dict` of path-like):
| Path(s) of the CSV file(s).
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| columns (`list[str]`, *optional*):
| If not `None`, only these columns will be read from the file.
| A column name may be a prefix of a nested field, e.g. 'a' will select
| 'a.b', 'a.c', and 'a.d.e'.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`ParquetConfig`].
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import DatasetDict
| >>> ds = DatasetDict.from_parquet({'train': 'path/to/dataset/parquet'})
| ```
|
| from_text(path_or_paths: dict[str, typing.Union[str, bytes, os.PathLike]], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, **kwargs) -> 'DatasetDict'
| Create [`DatasetDict`] from text file(s).
|
| Args:
| path_or_paths (`dict` of path-like):
| Path(s) of the text file(s).
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`TextConfig`].
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import DatasetDict
| >>> ds = DatasetDict.from_text({'train': 'path/to/dataset.txt'})
| ```
|
| load_from_disk(dataset_dict_path: Union[str, bytes, os.PathLike], keep_in_memory: Optional[bool] = None, storage_options: Optional[dict] = None) -> 'DatasetDict'
| Load a dataset that was previously saved using [`save_to_disk`] from a filesystem using `fsspec.spec.AbstractFileSystem`.
|
| Args:
| dataset_dict_path (`path-like`):
| Path (e.g. `"dataset/train"`) or remote URI (e.g. `"s3//my-bucket/dataset/train"`)
| of the dataset dict directory where the dataset dict will be loaded from.
| keep_in_memory (`bool`, defaults to `None`):
| Whether to copy the dataset in-memory. If `None`, the
| dataset will not be copied in-memory unless explicitly enabled by setting
| `datasets.config.IN_MEMORY_MAX_SIZE` to nonzero. See more details in the
| [improve performance](../cache#improve-performance) section.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.8.0"/>
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> ds = load_from_disk('path/to/dataset/directory')
| ```
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| cache_files
| The cache files containing the Apache Arrow table backing each split.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.cache_files
| {'test': [{'filename': '/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46/rotten_tomatoes_movie_review-test.arrow'}],
| 'train': [{'filename': '/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46/rotten_tomatoes_movie_review-train.arrow'}],
| 'validation': [{'filename': '/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46/rotten_tomatoes_movie_review-validation.arrow'}]}
| ```
|
| column_names
| Names of the columns in each split of the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.column_names
| {'test': ['text', 'label'],
| 'train': ['text', 'label'],
| 'validation': ['text', 'label']}
| ```
|
| data
| The Apache Arrow tables backing each split.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.data
| ```
|
| num_columns
| Number of columns in each split of the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.num_columns
| {'test': 2, 'train': 2, 'validation': 2}
| ```
|
| num_rows
| Number of rows in each split of the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.num_rows
| {'test': 1066, 'train': 8530, 'validation': 1066}
| ```
|
| shape
| Shape of each split of the dataset (number of rows, number of columns).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.shape
| {'test': (1066, 2), 'train': (8530, 2), 'validation': (1066, 2)}
| ```
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (dict[typing.Union[str, datasets.splits.NamedSplit], ...
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __gt__(self, value, /)
| Return self>value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
datasets.DatasetInfo
Python Library Documentation: class DatasetInfo in module datasets.info
class DatasetInfo(builtins.object)
| DatasetInfo(description: str = <factory>, citation: str = <factory>, homepage: str = <factory>, license: str = <factory>, features: Optional[datasets.features.features.Features] = None, post_processed: Optional[datasets.info.PostProcessedInfo] = None, supervised_keys: Optional[datasets.info.SupervisedKeysData] = None, builder_name: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, version: Union[str, datasets.utils.version.Version, NoneType] = None, splits: Optional[datasets.splits.SplitDict] = None, download_checksums: Optional[dict] = None, download_size: Optional[int] = None, post_processing_size: Optional[int] = None, dataset_size: Optional[int] = None, size_in_bytes: Optional[int] = None) -> None
|
| Information about a dataset.
|
| `DatasetInfo` documents datasets, including its name, version, and features.
| See the constructor arguments and properties for a full list.
|
| Not all fields are known on construction and may be updated later.
|
| Attributes:
| description (`str`):
| A description of the dataset.
| citation (`str`):
| A BibTeX citation of the dataset.
| homepage (`str`):
| A URL to the official homepage for the dataset.
| license (`str`):
| The dataset's license. It can be the name of the license or a paragraph containing the terms of the license.
| features ([`Features`], *optional*):
| The features used to specify the dataset's column types.
| post_processed (`PostProcessedInfo`, *optional*):
| Deprecated. Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index.
| supervised_keys (`SupervisedKeysData`, *optional*):
| Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS).
| builder_name (`str`, *optional*):
| The name of the `GeneratorBasedBuilder` subclass used to create the dataset. It is also the snake_case version of the dataset builder class name.
| config_name (`str`, *optional*):
| The name of the configuration derived from [`BuilderConfig`].
| version (`str` or [`Version`], *optional*):
| The version of the dataset.
| splits (`dict`, *optional*):
| The mapping between split name and metadata.
| download_checksums (`dict`, *optional*):
| The mapping between the URL to download the dataset's checksums and corresponding metadata.
| download_size (`int`, *optional*):
| The size of the files to download to generate the dataset, in bytes.
| post_processing_size (`int`, *optional*):
| Deprecated. Size of the dataset in bytes after post-processing, if any.
| dataset_size (`int`, *optional*):
| The combined size in bytes of the Arrow tables for all splits.
| size_in_bytes (`int`, *optional*):
| The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files).
| **config_kwargs (additional keyword arguments):
| Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`].
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, description: str = <factory>, citation: str = <factory>, homepage: str = <factory>, license: str = <factory>, features: Optional[datasets.features.features.Features] = None, post_processed: Optional[datasets.info.PostProcessedInfo] = None, supervised_keys: Optional[datasets.info.SupervisedKeysData] = None, builder_name: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, version: Union[str, datasets.utils.version.Version, NoneType] = None, splits: Optional[datasets.splits.SplitDict] = None, download_checksums: Optional[dict] = None, download_size: Optional[int] = None, post_processing_size: Optional[int] = None, dataset_size: Optional[int] = None, size_in_bytes: Optional[int] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| copy(self) -> 'DatasetInfo'
|
| update(self, other_dataset_info: 'DatasetInfo', ignore_none=True)
|
| write_to_directory(self, dataset_info_dir, pretty_print=False, storage_options: Optional[dict] = None)
| Write `DatasetInfo` and license (if present) as JSON files to `dataset_info_dir`.
|
| Args:
| dataset_info_dir (`str`):
| Destination directory.
| pretty_print (`bool`, defaults to `False`):
| If `True`, the JSON will be pretty-printed with the indent level of 4.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.9.0"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.info.write_to_directory("/path/to/directory/")
| ```
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_dict(dataset_info_dict: dict) -> 'DatasetInfo'
|
| from_directory(dataset_info_dir: str, storage_options: Optional[dict] = None) -> 'DatasetInfo'
| Create [`DatasetInfo`] from the JSON file in `dataset_info_dir`.
|
| This function updates all the dynamically generated fields (num_examples,
| hash, time of creation,...) of the [`DatasetInfo`].
|
| This will overwrite all previous metadata.
|
| Args:
| dataset_info_dir (`str`):
| The directory containing the metadata file. This
| should be the root directory of a specific dataset version.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.9.0"/>
|
| Example:
|
| ```py
| >>> from datasets import DatasetInfo
| >>> ds_info = DatasetInfo.from_directory("/path/to/directory/")
| ```
|
| from_merge(dataset_infos: list['DatasetInfo'])
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_INCLUDED_INFO_IN_YAML': typing.ClassVar[list[str]...
|
| __dataclass_fields__ = {'_INCLUDED_INFO_IN_YAML': Field(name='_INCLUDE...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('description', 'citation', 'homepage', 'license', 'f...
|
| builder_name = None
|
| config_name = None
|
| dataset_name = None
|
| dataset_size = None
|
| download_checksums = None
|
| download_size = None
|
| features = None
|
| post_processed = None
|
| post_processing_size = None
|
| size_in_bytes = None
|
| splits = None
|
| supervised_keys = None
|
| version = None
datasets.DownloadConfig
Python Library Documentation: class DownloadConfig in module datasets.download.download_config
class DownloadConfig(builtins.object)
| DownloadConfig(cache_dir: Union[str, pathlib.Path, NoneType] = None, force_download: bool = False, resume_download: bool = False, local_files_only: bool = False, proxies: Optional[dict] = None, user_agent: Optional[str] = None, extract_compressed_file: bool = False, force_extract: bool = False, delete_extracted: bool = False, extract_on_the_fly: bool = False, use_etag: bool = True, num_proc: Optional[int] = None, max_retries: int = 1, token: Union[str, bool, NoneType] = None, storage_options: dict[str, typing.Any] = <factory>, download_desc: Optional[str] = None, disable_tqdm: bool = False) -> None
|
| Configuration for our cached path manager.
|
| Attributes:
| cache_dir (`str` or `Path`, *optional*):
| Specify a cache directory to save the file to (overwrite the
| default cache dir).
| force_download (`bool`, defaults to `False`):
| If `True`, re-download the file even if it's already cached in
| the cache dir.
| resume_download (`bool`, defaults to `False`):
| If `True`, resume the download if an incompletely received file is
| found.
| proxies (`dict`, *optional*):
| user_agent (`str`, *optional*):
| Optional string or dict that will be appended to the user-agent on remote
| requests.
| extract_compressed_file (`bool`, defaults to `False`):
| If `True` and the path point to a zip or tar file,
| extract the compressed file in a folder along the archive.
| force_extract (`bool`, defaults to `False`):
| If `True` when `extract_compressed_file` is `True` and the archive
| was already extracted, re-extract the archive and override the folder where it was extracted.
| delete_extracted (`bool`, defaults to `False`):
| Whether to delete (or keep) the extracted files.
| extract_on_the_fly (`bool`, defaults to `False`):
| If `True`, extract compressed files while they are being read.
| use_etag (`bool`, defaults to `True`):
| Whether to use the ETag HTTP response header to validate the cached files.
| num_proc (`int`, *optional*):
| The number of processes to launch to download the files in parallel.
| max_retries (`int`, default to `1`):
| The number of times to retry an HTTP request if it fails.
| token (`str` or `bool`, *optional*):
| Optional string or boolean to use as Bearer token
| for remote files on the Datasets Hub. If `True`, or not specified, will get token from `~/.huggingface`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the dataset file-system backend, if any.
| download_desc (`str`, *optional*):
| A description to be displayed alongside with the progress bar while downloading the files.
| disable_tqdm (`bool`, defaults to `False`):
| Whether to disable the individual files download progress bar
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, cache_dir: Union[str, pathlib.Path, NoneType] = None, force_download: bool = False, resume_download: bool = False, local_files_only: bool = False, proxies: Optional[dict] = None, user_agent: Optional[str] = None, extract_compressed_file: bool = False, force_extract: bool = False, delete_extracted: bool = False, extract_on_the_fly: bool = False, use_etag: bool = True, num_proc: Optional[int] = None, max_retries: int = 1, token: Union[str, bool, NoneType] = None, storage_options: dict[str, typing.Any] = <factory>, download_desc: Optional[str] = None, disable_tqdm: bool = False) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| __setattr__(self, name, value)
| Implement setattr(self, name, value).
|
| copy(self) -> 'DownloadConfig'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'cache_dir': typing.Union[str, pathlib.Path, NoneTy...
|
| __dataclass_fields__ = {'cache_dir': Field(name='cache_dir',type=typin...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('cache_dir', 'force_download', 'resume_download', 'l...
|
| cache_dir = None
|
| delete_extracted = False
|
| disable_tqdm = False
|
| download_desc = None
|
| extract_compressed_file = False
|
| extract_on_the_fly = False
|
| force_download = False
|
| force_extract = False
|
| local_files_only = False
|
| max_retries = 1
|
| num_proc = None
|
| proxies = None
|
| resume_download = False
|
| token = None
|
| use_etag = True
|
| user_agent = None
datasets.DownloadManager
Python Library Documentation: class DownloadManager in module datasets.download.download_manager
class DownloadManager(builtins.object)
| DownloadManager(dataset_name: Optional[str] = None, data_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, base_path: Optional[str] = None, record_checksums=False)
|
| Methods defined here:
|
| __init__(self, dataset_name: Optional[str] = None, data_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, base_path: Optional[str] = None, record_checksums=False)
| Download manager constructor.
|
| Args:
| data_dir:
| can be used to specify a manual directory to get the files from.
| dataset_name (`str`):
| name of dataset this instance will be used for. If
| provided, downloads will contain which datasets they were used for.
| download_config (`DownloadConfig`):
| to specify the cache directory and other
| download options
| base_path (`str`):
| base path that is used when relative paths are used to
| download files. This can be a remote url.
| record_checksums (`bool`, defaults to `False`):
| Whether to record the checksums of the downloaded files. If None, the value is inferred from the builder.
|
| delete_extracted_files(self)
|
| download(self, url_or_urls)
| Download given URL(s).
|
| By default, only one process is used for download. Pass customized `download_config.num_proc` to change this behavior.
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL or `list` or `dict` of URLs to download. Each URL is a `str`.
|
| Returns:
| `str` or `list` or `dict`:
| The downloaded paths matching the given input `url_or_urls`.
|
| Example:
|
| ```py
| >>> downloaded_files = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| ```
|
| download_and_extract(self, url_or_urls)
| Download and extract given `url_or_urls`.
|
| Is roughly equivalent to:
|
| ```
| extracted_paths = dl_manager.extract(dl_manager.download(url_or_urls))
| ```
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL or `list` or `dict` of URLs to download and extract. Each URL is a `str`.
|
| Returns:
| extracted_path(s): `str`, extracted paths of given URL(s).
|
| extract(self, path_or_paths)
| Extract given path(s).
|
| Args:
| path_or_paths (path or `list` or `dict`):
| Path of file to extract. Each path is a `str`.
|
| Returns:
| extracted_path(s): `str`, The extracted paths matching the given input
| path_or_paths.
|
| Example:
|
| ```py
| >>> downloaded_files = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| >>> extracted_files = dl_manager.extract(downloaded_files)
| ```
|
| get_recorded_sizes_checksums(self)
|
| iter_archive(self, path_or_buf: Union[str, _io.BufferedReader])
| Iterate over files within an archive.
|
| Args:
| path_or_buf (`str` or `io.BufferedReader`):
| Archive path or archive binary file object.
|
| Yields:
| `tuple[str, io.BufferedReader]`:
| 2-tuple (path_within_archive, file_object).
| File object is opened in binary mode.
|
| Example:
|
| ```py
| >>> archive = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| >>> files = dl_manager.iter_archive(archive)
| ```
|
| iter_files(self, paths: Union[str, list[str]])
| Iterate over file paths.
|
| Args:
| paths (`str` or `list` of `str`):
| Root paths.
|
| Yields:
| `str`: File path.
|
| Example:
|
| ```py
| >>> files = dl_manager.download_and_extract('https://huggingface.co/datasets/AI-Lab-Makerere/beans/resolve/main/data/train.zip')
| >>> files = dl_manager.iter_files(files)
| ```
|
| manage_extracted_files(self)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| downloaded_size
| Returns the total size of downloaded files.
|
| manual_dir
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| is_streaming = False
datasets.DownloadMode
Python Library Documentation: class DownloadMode in module datasets.download.download_manager class DownloadMode(enum.Enum) | DownloadMode(*values) | | `Enum` for how to treat pre-existing downloads and data. | | The default mode is `REUSE_DATASET_IF_EXISTS`, which will reuse both | raw downloads and the prepared dataset if they exist. | | The generations modes: | | | | Downloads | Dataset | | |-------------------------------------|-----------|---------| | | `REUSE_DATASET_IF_EXISTS` (default) | Reuse | Reuse | | | `REUSE_CACHE_IF_EXISTS` | Reuse | Fresh | | | `FORCE_REDOWNLOAD` | Fresh | Fresh | | | Method resolution order: | DownloadMode | enum.Enum | builtins.object | | Data and other attributes defined here: | | FORCE_REDOWNLOAD = <DownloadMode.FORCE_REDOWNLOAD: 'force_redownload'> | | REUSE_CACHE_IF_EXISTS = <DownloadMode.REUSE_CACHE_IF_EXISTS: 'reuse_ca... | | REUSE_DATASET_IF_EXISTS = <DownloadMode.REUSE_DATASET_IF_EXISTS: 'reus... | | ---------------------------------------------------------------------- | Data descriptors inherited from enum.Enum: | | name | The name of the Enum member. | | value | The value of the Enum member. | | ---------------------------------------------------------------------- | Static methods inherited from enum.EnumType: | | __contains__(value) | Return True if `value` is in `cls`. | | `value` is in `cls` if: | 1) `value` is a member of `cls`, or | 2) `value` is the value of one of the `cls`'s members. | | __getitem__(name) | Return the member matching `name`. | | __iter__() | Return members in definition order. | | __len__() | Return the number of members (no aliases) | | ---------------------------------------------------------------------- | Readonly properties inherited from enum.EnumType: | | __members__ | Returns a mapping of member name->value. | | This mapping lists all enum members, including aliases. Note that this | is a read-only view of the internal mapping.
datasets.Features
Python Library Documentation: class Features in module datasets.features.features
class Features(builtins.dict)
| Features(*args, **kwargs)
|
| A special dictionary that defines the internal structure of a dataset.
|
| Instantiated with a dictionary of type `dict[str, FieldType]`, where keys are the desired column names,
| and values are the type of that column.
|
| `FieldType` can be one of the following:
| - [`Value`] feature specifies a single data type value, e.g. `int64` or `string`.
| - [`ClassLabel`] feature specifies a predefined set of classes which can have labels associated to them and
| will be stored as integers in the dataset.
| - Python `dict` specifies a composite feature containing a mapping of sub-fields to sub-features.
| It's possible to have nested fields of nested fields in an arbitrary manner.
| - [`List`] or [`LargeList`] specifies a composite feature containing a sequence of
| sub-features, all of the same feature type.
| - [`Array2D`], [`Array3D`], [`Array4D`] or [`Array5D`] feature for multidimensional arrays.
| - [`Audio`] feature to store the absolute path to an audio file or a dictionary with the relative path
| to an audio file ("path" key) and its bytes content ("bytes" key).
| This feature loads the audio lazily with a decoder.
| - [`Image`] feature to store the absolute path to an image file, an `np.ndarray` object, a `PIL.Image.Image` object
| or a dictionary with the relative path to an image file ("path" key) and its bytes content ("bytes" key).
| This feature extracts the image data.
| - [`Video`] feature to store the absolute path to a video file, a `torchcodec.decoders.VideoDecoder` object
| or a dictionary with the relative path to a video file ("path" key) and its bytes content ("bytes" key).
| This feature loads the video lazily with a decoder.
| - [`Pdf`] feature to store the absolute path to a PDF file, a `pdfplumber.pdf.PDF` object
| or a dictionary with the relative path to a PDF file ("path" key) and its bytes content ("bytes" key).
| This feature loads the PDF lazily with a PDF reader.
| - [`Nifti`] feature to store the absolute path to a NIfTI neuroimaging file, a `nibabel.Nifti1Image` object
| or a dictionary with the relative path to a NIfTI file ("path" key) and its bytes content ("bytes" key).
| This feature loads the NIfTI file lazily with nibabel.
| - [`Translation`] or [`TranslationVariableLanguages`] feature specific to Machine Translation.
| - [`Json`] feature to store unstructred data, e.g. containing mixed/abritrary types. Under the hood
|
| Method resolution order:
| Features
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __delitem__(self, key, /) from datasets.features.features.dict
| Delete self[key].
|
| __init__(*args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __reduce__(self)
| Helper for pickle.
|
| __setitem__(self, key, value, /) from datasets.features.features.dict
| Set self[key] to value.
|
| clear(...) from datasets.features.features.dict
| D.clear() -> None. Remove all items from D.
|
| copy(self) -> 'Features'
| Make a deep copy of [`Features`].
|
| Returns:
| [`Features`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> copy_of_features = ds.features.copy()
| >>> copy_of_features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| ```
|
| decode_batch(self, batch: dict, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
| Decode batch with custom feature decoding.
|
| Args:
| batch (`dict[str, list[Any]]`):
| Dataset batch data.
| token_per_repo_id (`dict`, *optional*):
| To access and decode audio or image files from private repositories on the Hub, you can pass
| a dictionary repo_id (str) -> token (bool or str)
|
| Returns:
| `dict[str, list[Any]]`
|
| decode_column(self, column: list, column_name: str, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
| Decode column with custom feature decoding.
|
| Args:
| column (`list[Any]`):
| Dataset column data.
| column_name (`str`):
| Dataset column name.
|
| Returns:
| `list[Any]`
|
| decode_example(self, example: dict, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
| Decode example with custom feature decoding.
|
| Args:
| example (`dict[str, Any]`):
| Dataset row data.
| token_per_repo_id (`dict`, *optional*):
| To access and decode audio or image files from private repositories on the Hub, you can pass
| a dictionary `repo_id (str) -> token (bool or str)`.
|
| Returns:
| `dict[str, Any]`
|
| encode_batch(self, batch)
| Encode batch into a format for Arrow.
|
| Args:
| batch (`dict[str, list[Any]]`):
| Data in a Dataset batch.
|
| Returns:
| `dict[str, list[Any]]`
|
| encode_column(self, column, column_name: str)
| Encode column into a format for Arrow.
|
| Args:
| column (`list[Any]`):
| Data in a Dataset column.
| column_name (`str`):
| Dataset column name.
|
| Returns:
| `list[Any]`
|
| encode_example(self, example)
| Encode example into a format for Arrow.
|
| Args:
| example (`dict[str, Any]`):
| Data in a Dataset row.
|
| Returns:
| `dict[str, Any]`
|
| flatten(self, max_depth=16) -> 'Features'
| Flatten the features. Every dictionary column is removed and is replaced by
| all the subfields it contains. The new fields are named by concatenating the
| name of the original column and the subfield name like this: `<original>.<subfield>`.
|
| If a column contains nested dictionaries, then all the lower-level subfields names are
| also concatenated to form new columns: `<original>.<subfield>.<subsubfield>`, etc.
|
| Returns:
| [`Features`]:
| The flattened features.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("rajpurkar/squad", split="train")
| >>> ds.features.flatten()
| {'answers.answer_start': List(Value('int32'), id=None),
| 'answers.text': List(Value('string'), id=None),
| 'context': Value('string'),
| 'id': Value('string'),
| 'question': Value('string'),
| 'title': Value('string')}
| ```
|
| pop(...) from datasets.features.features.dict
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /) from datasets.features.features.dict
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| reorder_fields_as(self, other: 'Features') -> 'Features'
| Reorder Features fields to match the field order of other [`Features`].
|
| The order of the fields is important since it matters for the underlying arrow data.
| Re-ordering the fields allows to make the underlying arrow data type match.
|
| Args:
| other ([`Features`]):
| The other [`Features`] to align with.
|
| Returns:
| [`Features`]
|
| Example::
|
| >>> from datasets import Features, List, Value
| >>> # let's say we have two features with a different order of nested fields (for a and b for example)
| >>> f1 = Features({"root": {"a": Value("string"), "b": Value("string")}})
| >>> f2 = Features({"root": {"b": Value("string"), "a": Value("string")}})
| >>> assert f1.type != f2.type
| >>> # re-ordering keeps the base structure (here List is defined at the root level), but makes the fields order match
| >>> f1.reorder_fields_as(f2)
| {'root': List({'b': Value('string'), 'a': Value('string')})}
| >>> assert f1.reorder_fields_as(f2).type == f2.type
|
| setdefault(self, key, default=None, /) from datasets.features.features.dict
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| to_dict(self)
|
| update(...) from datasets.features.features.dict
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_arrow_schema(pa_schema: pyarrow.lib.Schema) -> 'Features'
| Construct [`Features`] from Arrow Schema.
| It also checks the schema metadata for Hugging Face Datasets features.
| Non-nullable fields are not supported and set to nullable.
|
| Also, pa.dictionary is not supported and it uses its underlying type instead.
| Therefore datasets convert DictionaryArray objects to their actual values.
|
| Args:
| pa_schema (`pyarrow.Schema`):
| Arrow Schema.
|
| Returns:
| [`Features`]
|
| from_dict(dic) -> 'Features'
| Construct [`Features`] from dict.
|
| Regenerate the nested feature object from a deserialized dict.
| We use the `_type` key to infer the dataclass name of the feature `FieldType`.
|
| It allows for a convenient constructor syntax
| to define features from deserialized JSON dictionaries. This function is used in particular when deserializing
| a [`DatasetInfo`] that was dumped to a JSON object. This acts as an analogue to
| [`Features.from_arrow_schema`] and handles the recursive field-by-field instantiation, but doesn't require
| any mapping to/from pyarrow, except for the fact that it takes advantage of the mapping of pyarrow primitive
| dtypes that [`Value`] automatically performs.
|
| Args:
| dic (`dict[str, Any]`):
| Python dictionary.
|
| Returns:
| `Features`
|
| Example::
| >>> Features.from_dict({'_type': {'dtype': 'string', 'id': None, '_type': 'Value'}})
| {'_type': Value('string')}
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| arrow_schema
| Features schema.
|
| Returns:
| :obj:`pyarrow.Schema`
|
| type
| Features field types.
|
| Returns:
| :obj:`pyarrow.DataType`
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
datasets.GeneratorBasedBuilder
Python Library Documentation: class GeneratorBasedBuilder in module datasets.builder
class GeneratorBasedBuilder(DatasetBuilder)
| GeneratorBasedBuilder(cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
|
| Base class for datasets with data generation based on dict generators.
|
| `GeneratorBasedBuilder` is a convenience class that abstracts away much
| of the data writing and reading of `DatasetBuilder`. It expects subclasses to
| implement generators of feature dictionaries across the dataset splits
| (`_split_generators`). See the method docstrings for details.
|
| Method resolution order:
| GeneratorBasedBuilder
| DatasetBuilder
| builtins.object
|
| Methods inherited from DatasetBuilder:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setstate__(self, d)
|
| as_dataset(self, split: Union[str, datasets.splits.Split, list[str], list[datasets.splits.Split], NoneType] = None, in_memory=False) -> Union[datasets.arrow_dataset.Dataset, datasets.dataset_dict.DatasetDict]
| Return a Dataset for the specified split.
|
| Args:
| split (`datasets.Split`):
| Which subset of the data to return.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the
| downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
|
| Returns:
| datasets.Dataset
|
| Example:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')
| >>> builder.download_and_prepare()
| >>> ds = builder.as_dataset(split='train')
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 8530
| })
| ```
|
| as_streaming_dataset(self, split: Optional[str] = None, base_path: Optional[str] = None) -> Union[dict[str, datasets.iterable_dataset.IterableDataset], datasets.iterable_dataset.IterableDataset]
|
| download_and_prepare(self, output_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, verification_mode: Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None, dl_manager: Optional[datasets.download.download_manager.DownloadManager] = None, base_path: Optional[str] = None, file_format: str = 'arrow', max_shard_size: Union[str, int, NoneType] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **download_and_prepare_kwargs)
| Downloads and prepares dataset for reading.
|
| Args:
| output_dir (`str`, *optional*):
| Output directory for the dataset.
| Default to this builder's `cache_dir`, which is inside `~/.cache/huggingface/datasets` by default.
|
| <Added version="2.5.0"/>
| download_config (`DownloadConfig`, *optional*):
| Specific download configuration parameters.
| download_mode ([`DownloadMode`] or `str`, *optional*):
| Select the download/generate mode, default to `REUSE_DATASET_IF_EXISTS`.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| dl_manager (`DownloadManager`, *optional*):
| Specific `DownloadManger` to use.
| base_path (`str`, *optional*):
| Base path for relative paths that are used to download files. This can be a remote url.
| If not specified, the value of the `base_path` attribute (`self.base_path`) will be used instead.
| file_format (`str`, *optional*):
| Format of the data files in which the dataset will be written.
| Supported formats: "arrow", "parquet". Default to "arrow" format.
| If the format is "parquet", then image and audio data are embedded into the Parquet files instead of pointing to local files.
|
| <Added version="2.5.0"/>
| max_shard_size (`Union[str, int]`, *optional*):
| Maximum number of bytes written per shard, default is "500MB".
| The size is based on uncompressed data size, so in practice your shard files may be smaller than
| `max_shard_size` thanks to Parquet compression for example.
|
| <Added version="2.5.0"/>
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.7.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the caching file-system backend, if any.
|
| <Added version="2.5.0"/>
| **download_and_prepare_kwargs (additional keyword arguments): Keyword arguments.
|
| Example:
|
| Download and prepare the dataset as Arrow files that can be loaded as a Dataset using `builder.as_dataset()`:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare()
| ```
|
| Download and prepare the dataset as sharded Parquet files locally:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("./output_dir", file_format="parquet")
| ```
|
| Download and prepare the dataset as sharded Parquet files in a cloud storage:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> storage_options = {"key": aws_access_key_id, "secret": aws_secret_access_key}
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("s3://my-bucket/my_rotten_tomatoes", storage_options=storage_options, file_format="parquet")
| ```
|
| ----------------------------------------------------------------------
| Class methods inherited from DatasetBuilder:
|
| get_imported_module_dir()
| Return the path of the module of this class or subclass.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from DatasetBuilder:
|
| builder_configs
| Dictionary of pre-defined configurations for this builder class.
|
| cache_dir
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DatasetBuilder:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from DatasetBuilder:
|
| BUILDER_CONFIGS = []
|
| BUILDER_CONFIG_CLASS = <class 'datasets.builder.BuilderConfig'>
| Base class for `DatasetBuilder` data configuration.
|
| `DatasetBuilder` subclasses with data configuration options should subclass
| `BuilderConfig` and add their own properties.
|
| Attributes:
| name (`str`, defaults to `default`):
| The name of the configuration.
| version (`Version` or `str`, defaults to `0.0.0`):
| The version of the configuration.
| data_dir (`str`, *optional*):
| Path to the directory containing the source data.
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| description (`str`, *optional*):
| A human description of the configuration.
|
|
| DEFAULT_CONFIG_NAME = None
|
| DEFAULT_WRITER_BATCH_SIZE = None
|
| SLEEP_ON_THREADS_SHUTDOWNS = False
|
| VERSION = None
datasets.Image
Python Library Documentation: class Image in module datasets.features.image
class Image(builtins.object)
| Image(mode: Optional[str] = None, decode: bool = True, id: Optional[str] = None) -> None
|
| Image [`Feature`] to read image data from an image file.
|
| Input: The Image feature accepts as input:
| - A `str`: Absolute path to the image file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the image file (i.e. random access is allowed).
| - A `dict` with the keys:
|
| - `path`: String with relative path of the image file to the archive file.
| - `bytes`: Bytes of the image file.
|
| This is useful for parquet or webdataset files which embed image files.
|
| - An `np.ndarray`: NumPy array representing an image.
| - A `PIL.Image.Image`: PIL image object.
|
| Output: The Image features output data as `PIL.Image.Image` objects.
|
| Args:
| mode (`str`, *optional*):
| The mode to convert the image to. If `None`, the native mode of the image is used.
| decode (`bool`, defaults to `True`):
| Whether to decode the image data. If `False`,
| returns the underlying dictionary in the format `{"path": image_path, "bytes": image_bytes}`.
|
| Examples:
|
| ```py
| >>> from datasets import load_dataset, Image
| >>> ds = load_dataset("AI-Lab-Makerere/beans", split="train")
| >>> ds.features["image"]
| Image(decode=True, id=None)
| >>> ds[0]["image"]
| <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x500 at 0x15E52E7F0>
| >>> ds = ds.cast_column('image', Image(decode=False))
| {'bytes': None,
| 'path': '/root/.cache/huggingface/datasets/downloads/extracted/b0a21163f78769a2cf11f58dfc767fb458fc7cea5c05dccc0144a2c0f0bc1292/train/healthy/healthy_train.85.jpg'}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, mode: Optional[str] = None, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray, pyarrow.lib.ListArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Image arrow storage type.
| The Arrow types that can be converted to the Image pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.large_string()` - it must contain the "path" data (will be cast to string if possible)
| - `pa.binary()` - it must contain the image bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
| - `pa.list(*)` - it must contain the image array data
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Image arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: dict, token_per_repo_id=None) -> 'PIL.Image.Image'
| Decode example image file into image data.
|
| Args:
| value (`str` or `dict`):
| A string with the absolute image file path, a dictionary with
| keys:
|
| - `path`: String with absolute or relative image file path.
| - `bytes`: The bytes of the image file.
| token_per_repo_id (`dict`, *optional*):
| To access and decode
| image files from private repositories on the Hub, you can pass
| a dictionary repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `PIL.Image.Image`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed image files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the Image arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, numpy.ndarray, ForwardRef('PIL.Image.Image')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `np.ndarray`, `PIL.Image.Image` or `dict`):
| Data passed as input to Image feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('mode', 'decode', 'id')
|
| decode = True
|
| dtype = 'PIL.Image.Image'
|
| id = None
|
| mode = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
datasets.IterableColumn
Python Library Documentation: class IterableColumn in module datasets.iterable_dataset
class IterableColumn(builtins.object)
| IterableColumn(source: Union[ForwardRef('IterableDataset'), ForwardRef('IterableColumn')], column_name: str)
|
| An iterable for a specific column of an [`IterableDataset`].
|
| Example:
|
| Iterate on the texts of the "text" column of a dataset:
|
| ```python
| for text in dataset["text"]:
| ...
| ```
|
| It also works with nested columns:
|
| ```python
| for source in dataset["metadata"]["source"]:
| ...
| ```
|
| Methods defined here:
|
| __getitem__(self, column_name: str) -> 'IterableColumn'
|
| __init__(self, source: Union[ForwardRef('IterableDataset'), ForwardRef('IterableColumn')], column_name: str)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> collections.abc.Iterator[typing.Any]
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
datasets.IterableDataset
Python Library Documentation: class IterableDataset in module datasets.iterable_dataset
class IterableDataset(datasets.arrow_dataset.DatasetInfoMixin)
| IterableDataset(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, formatting: Optional[datasets.iterable_dataset.FormattingConfig] = None, distributed: Optional[datasets.iterable_dataset.DistributedConfig] = None, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
|
| A Dataset backed by an iterable.
|
| Method resolution order:
| IterableDataset
| datasets.arrow_dataset.DatasetInfoMixin
| builtins.object
|
| Methods defined here:
|
| __getitem__(self, column_name: str) -> datasets.iterable_dataset.IterableColumn
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, formatting: Optional[datasets.iterable_dataset.FormattingConfig] = None, distributed: Optional[datasets.iterable_dataset.DistributedConfig] = None, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
|
| __repr__(self)
| Return repr(self).
|
| __setstate__(self, d)
|
| add_column(self, name: str, column: Union[list, <built-in function array>]) -> 'IterableDataset'
| Add column to Dataset.
|
| Args:
| name (str): Column name.
| column (list or np.array): Column data to be added.
|
| Returns:
| `IterableDataset`
|
| batch(self, batch_size: Optional[int] = None, by_column: Union[str, list[str], NoneType] = None, drop_last_batch: bool = False) -> 'IterableDataset'
| Group samples from the dataset into batches.
|
| Args:
| batch_size (`int`, optional):
| The number of samples in each batch.
| by_column (`Union[str, list[str]`, optional):
| The column used to batch examples together.
| Successive examples with the same value for that column are in grouped the same batch.
| This can also be a list of columns if you want to batch by multiple columns.
| If batching by column, the batch_size is only used to control the size of the batches
| to group together or slice during acculumation.
|
| <Added version="4.9.0"/>
| drop_last_batch (`bool`, defaults to `False`):
| Whether to drop the last incomplete batch.
|
| Example:
| ```py
| >>> ds = load_dataset("some_dataset", streaming=True)
| >>> batched_ds = ds.batch(batch_size=32)
| ```
|
| cast(self, features: datasets.features.features.Features) -> 'IterableDataset'
| Cast the dataset to a new set of features.
|
| Args:
| features ([`Features`]):
| New features to cast the dataset to.
| The name of the fields in the features must match the current column names.
| The type of the data must also be convertible from one type to the other.
| For non-trivial conversion, e.g. `string` <-> `ClassLabel` you should use [`~Dataset.map`] to update the Dataset.
|
| Returns:
| `IterableDataset`: A copy of the dataset with casted features.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel, Value
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> ds.features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> new_features = ds.features.copy()
| >>> new_features["label"] = ClassLabel(names=["bad", "good"])
| >>> new_features["text"] = Value("large_string")
| >>> ds = ds.cast(new_features)
| >>> ds.features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('large_string')}
| ```
|
| cast_column(self, column: str, feature: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti]) -> 'IterableDataset'
| Cast column to feature for decoding.
|
| Args:
| column (`str`):
| Column name.
| feature (`Feature`):
| Target feature.
|
| Returns:
| `IterableDataset`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, Audio
| >>> ds = load_dataset("PolyAI/minds14", name="en-US", split="train", streaming=True)
| >>> ds.features
| {'audio': Audio(sampling_rate=8000, mono=True, decode=True, id=None),
| 'english_transcription': Value('string'),
| 'intent_class': ClassLabel(num_classes=14, names=['abroad', 'address', 'app_error', 'atm_limit', 'balance', 'business_loan', 'card_issues', 'cash_deposit', 'direct_debit', 'freeze', 'high_value_payment', 'joint_account', 'latest_transactions', 'pay_bill']),
| 'lang_id': ClassLabel(num_classes=14, names=['cs-CZ', 'de-DE', 'en-AU', 'en-GB', 'en-US', 'es-ES', 'fr-FR', 'it-IT', 'ko-KR', 'nl-NL', 'pl-PL', 'pt-PT', 'ru-RU', 'zh-CN']),
| 'path': Value('string'),
| 'transcription': Value('string')}
| >>> ds = ds.cast_column("audio", Audio(sampling_rate=16000))
| >>> ds.features
| {'audio': Audio(sampling_rate=16000, mono=True, decode=True, id=None),
| 'english_transcription': Value('string'),
| 'intent_class': ClassLabel(num_classes=14, names=['abroad', 'address', 'app_error', 'atm_limit', 'balance', 'business_loan', 'card_issues', 'cash_deposit', 'direct_debit', 'freeze', 'high_value_payment', 'joint_account', 'latest_transactions', 'pay_bill']),
| 'lang_id': ClassLabel(num_classes=14, names=['cs-CZ', 'de-DE', 'en-AU', 'en-GB', 'en-US', 'es-ES', 'fr-FR', 'it-IT', 'ko-KR', 'nl-NL', 'pl-PL', 'pt-PT', 'ru-RU', 'zh-CN']),
| 'path': Value('string'),
| 'transcription': Value('string')}
| ```
|
| decode(self, enable: bool = True, num_threads: int = 0) -> 'IterableDataset'
| Enable or disable the dataset features decoding for audio, image, video.
|
| When enabled (default), media types are decoded:
|
| * audio -> dict of "array" and "sampling_rate" and "path"
| * image -> PIL.Image
| * video -> torchcodec.decoders.VideoDecoder
|
| You can enable multithreading using `num_threads`. This is especially useful to speed up remote
| data streaming. However it can be slower than `num_threads=0` for local data on fast disks.
|
| Disabling decoding is useful if you want to iterate on the paths or bytes of the media files
| without actually decoding their content. To disable decoding you can use `.decode(False)`, which
| is equivalent to calling `.cast()` or `.cast_column()` with all the Audio, Image and Video types
| set to `decode=False`.
|
| Args:
| enable (`bool`, defaults to `True`):
| Enable or disable features decoding.
| num_threads (`int`, defaults to `0`):
| Enable multithreading for features decoding.
|
| Returns:
| `IterableDataset`: A copy of the dataset with casted features.
|
| Examples:
|
| Disable decoding:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True)
| >>> next(iter(ds))
| {'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=2048x1024>,
| 'text': 'A distant celestial object with an icy crust, displaying a light blue shade, covered with round pits and rugged terrains.'}
| >>> ds = ds.decode(False)
| >>> ds.features
| {'image': Image(mode=None, decode=False, id=None),
| 'text': Value('string')}
| >>> next(iter(ds))
| {
| 'image': {
| 'path': 'hf://datasets/sshh12/planet-textures@69dc4cef7a5c4b2cfe387727ec8ea73d4bff7302/train/textures/0000.png',
| 'bytes': None
| },
| 'text': 'A distant celestial object with an icy crust, displaying a light blue shade, covered with round pits and rugged terrains.'
| }
| ```
|
| Speed up streaming with multithreading:
|
| ```py
| >>> import os
| >>> from datasets import load_dataset
| >>> from tqdm import tqdm
| >>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True)
| >>> num_threads = min(32, (os.cpu_count() or 1) + 4)
| >>> ds = ds.decode(num_threads=num_threads)
| >>> for _ in tqdm(ds): # 20 times faster !
| ... ...
| ```
|
| filter(self, function: Optional[Callable] = None, with_indices=False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None) -> 'IterableDataset'
| Apply a filter function to all the elements so that the dataset only includes examples according to the filter function.
| The filtering is done on-the-fly when iterating over the dataset.
|
| If the function is asynchronous, then `filter` will run your function in parallel, with up to one thousand simulatenous calls (configurable).
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`):
| Callable with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> bool` if `with_indices=False, batched=False`
| - `function(example: Dict[str, Any], indices: int) -> bool` if `with_indices=True, batched=False`
| - `function(example: Dict[str, List]) -> List[bool]` if `with_indices=False, batched=True`
| - `function(example: Dict[str, List], indices: List[int]) -> List[bool]` if `with_indices=True, batched=True`
|
| If the function is asynchronous, then `filter` will run your function in parallel.
| If no function is provided, defaults to an always True function: `lambda x: True`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx): ...`.
| input_columns (`str` or `List[str]`, *optional*):
| The columns to be passed into `function` as
| positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, default `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| fn_kwargs (`Dict`, *optional*, default `None`):
| Keyword arguments to be passed to `function`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> ds = ds.filter(lambda x: x["label"] == 0)
| >>> list(ds.take(3))
| [{'label': 0, 'movie_review': 'simplistic , silly and tedious .'},
| {'label': 0,
| 'movie_review': "it's so laddish and juvenile , only teenage boys could possibly find it funny ."},
| {'label': 0,
| 'movie_review': 'exploitative and largely devoid of the depth or sophistication that would make watching such a graphic treatment of the crimes bearable .'}]
| ```
|
| iter(self, batch_size: int, drop_last_batch: bool = False)
| Iterate through the batches of size `batch_size`.
|
| Args:
| batch_size (:obj:`int`): size of each batch to yield.
| drop_last_batch (:obj:`bool`, default `False`): Whether a last batch smaller than the batch_size should be
| dropped
|
| load_state_dict(self, state_dict: dict) -> None
| Load the state_dict of the dataset.
| The iteration will restart at the next example from when the state was saved.
|
| Resuming returns exactly where the checkpoint was saved except in two cases:
|
| 1. examples from shuffle buffers are lost when resuming and the buffers are refilled with new data
| 2. combinations of `.with_format(arrow)` and batched `.map()` may skip one batch.
|
| Example:
|
| ```py
| >>> from datasets import Dataset, concatenate_datasets
| >>> ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3)
| >>> for idx, example in enumerate(ds):
| ... print(example)
| ... if idx == 2:
| ... state_dict = ds.state_dict()
| ... print("checkpoint")
| ... break
| >>> ds.load_state_dict(state_dict)
| >>> print(f"restart from checkpoint")
| >>> for example in ds:
| ... print(example)
| ```
|
| which returns:
| ```
| {'a': 0}
| {'a': 1}
| {'a': 2}
| checkpoint
| restart from checkpoint
| {'a': 3}
| {'a': 4}
| {'a': 5}
| ```
|
| ```py
| >>> from torchdata.stateful_dataloader import StatefulDataLoader
| >>> ds = load_dataset("deepmind/code_contests", streaming=True, split="train")
| >>> dataloader = StatefulDataLoader(ds, batch_size=32, num_workers=4)
| >>> # checkpoint
| >>> state_dict = dataloader.state_dict() # uses ds.state_dict() under the hood
| >>> # resume from checkpoint
| >>> dataloader.load_state_dict(state_dict) # uses ds.load_state_dict() under the hood
| ```
|
| map(self, function: Optional[Callable] = None, with_indices: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, drop_last_batch: bool = False, remove_columns: Union[str, list[str], NoneType] = None, features: Optional[datasets.features.features.Features] = None, fn_kwargs: Optional[dict] = None) -> 'IterableDataset'
| Apply a function to all the examples in the iterable dataset (individually or in batches) and update them.
| If your function returns a column that already exists, then it overwrites it.
| The function is applied on-the-fly on the examples when iterating over the dataset.
|
| You can specify whether the function should be batched or not with the `batched` parameter:
|
| - If batched is `False`, then the function takes 1 example in and should return 1 example.
| An example is a dictionary, e.g. `{"text": "Hello there !"}`.
| - If batched is `True` and `batch_size` is 1, then the function takes a batch of 1 example as input and can return a batch with 1 or more examples.
| A batch is a dictionary, e.g. a batch of 1 example is {"text": ["Hello there !"]}.
| - If batched is `True` and `batch_size` is `n` > 1, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.
| Note that the last batch may have less than `n` examples.
| A batch is a dictionary, e.g. a batch of `n` examples is `{"text": ["Hello there !"] * n}`.
|
| If the function is asynchronous, then `map` will run your function in parallel, with up to one thousand simulatenous calls.
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`, *optional*, defaults to `None`):
| Function applied on-the-fly on the examples when you iterate on the dataset.
| It must have one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> Dict[str, Any]` if `batched=False` and `with_indices=False`
| - `function(example: Dict[str, Any], idx: int) -> Dict[str, Any]` if `batched=False` and `with_indices=True`
| - `function(batch: Dict[str, List]) -> Dict[str, List]` if `batched=True` and `with_indices=False`
| - `function(batch: Dict[str, List], indices: List[int]) -> Dict[str, List]` if `batched=True` and `with_indices=True`
|
| For advanced usage, the function can also return a `pyarrow.Table`.
| If the function is asynchronous, then `map` will run your function in parallel.
| Moreover if your function returns nothing (`None`), then `map` will run your function and return the dataset unchanged.
| If no function is provided, default to identity function: `lambda x: x`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx[, rank]): ...`.
| input_columns (`Optional[Union[str, List[str]]]`, defaults to `None`):
| The columns to be passed into `function`
| as positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| `batch_size <= 0` or `batch_size == None` then provide the full dataset as a single batch to `function`.
| drop_last_batch (`bool`, defaults to `False`):
| Whether a last batch smaller than the batch_size should be
| dropped instead of being processed by the function.
| remove_columns (`[List[str]]`, *optional*, defaults to `None`):
| Remove a selection of columns while doing the mapping.
| Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding
| columns with names in `remove_columns`, these columns will be kept.
| features (`[Features]`, *optional*, defaults to `None`):
| Feature types of the resulting dataset.
| fn_kwargs (`Dict`, *optional*, default `None`):
| Keyword arguments to be passed to `function`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> def add_prefix(example):
| ... example["text"] = "Review: " + example["text"]
| ... return example
| >>> ds = ds.map(add_prefix)
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'Review: the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'Review: the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'Review: effective but too-tepid biopic'}]
| ```
|
| push_to_hub(self, repo_id: str, config_name: str = 'default', set_default: Optional[bool] = None, split: Optional[str] = None, data_dir: Optional[str] = None, commit_message: Optional[str] = None, commit_description: Optional[str] = None, private: Optional[bool] = None, token: Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = False, max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[int] = None, embed_external_files: bool = True, num_proc: Optional[int] = None) -> huggingface_hub.hf_api.CommitInfo
| Pushes the dataset to the hub as a Parquet dataset.
| The dataset is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
|
| The resulting Parquet files are self-contained by default. If your dataset contains [`Image`], [`Audio`] or [`Video`]
| data, the Parquet files will store the bytes of your images or audio files.
| You can disable this by setting `embed_external_files` to `False`.
|
| Args:
| repo_id (`str`):
| The ID of the repository to push to in the following format: `<user>/<dataset_name>` or
| `<org>/<dataset_name>`. Also accepts `<dataset_name>`, which will default to the namespace
| of the logged-in user.
|
| It could also be a location inside a bucket, e.g. `buckets/<user_or_org>/<bucket_name>/...`
| config_name (`str`, defaults to "default"):
| The configuration name (or subset) of a dataset. Defaults to "default".
| set_default (`bool`, *optional*):
| Whether to set this configuration as the default one. Otherwise, the default configuration is the one
| named "default".
| split (`str`, *optional*):
| The name of the split that will be given to that dataset. Defaults to `self.split`.
| data_dir (`str`, *optional*):
| Directory name that will contain the uploaded data files. Defaults to the `config_name` if different
| from "default", else "data".
| commit_message (`str`, *optional*):
| Message to commit while pushing. Will default to `"Upload dataset"`.
| commit_description (`str`, *optional*):
| Description of the commit that will be created.
| Additionally, description of the PR if a PR is created (`create_pr` is True).
| private (`bool`, *optional*):
| Whether to make the repo private. If `None` (default), the repo will be public unless the
| organization's default is private. This value is ignored if the repo already exists.
| token (`str`, *optional*):
| An optional authentication token for the Hugging Face Hub. If no token is passed, will default
| to the token saved locally when logging in with `huggingface-cli login`. Will raise an error
| if no token is passed and the user is not logged-in.
| revision (`str`, *optional*):
| Branch to push the uploaded files to. Defaults to the `"main"` branch.
| create_pr (`bool`, *optional*, defaults to `False`):
| Whether to create a PR with the uploaded files or directly commit.
| max_shard_size (`int` or `str`, *optional*):
| Optional maximum size of the dataset shards to be uploaded to the hub. If expressed as a string, needs to be digits followed
| by a unit (like `"5MB"`). If not provided, shard count defaults to this dataset's `.num_shards`.
| num_shards (`int`, *optional*):
| Number of shards to write. If `max_shard_size` is provided and `num_shards` is not, then the number of shards is estimated
| from `max_shard_size`.
| embed_external_files (`bool`, defaults to `True`):
| Whether to embed file bytes in the shards.
| In particular, this will do the following before the push for the fields of type:
|
| - [`Audio`] and [`Image`]: remove local path information and embed file content in the Parquet files.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when preparing and uploading the dataset.
| This is helpful if the dataset is made of many samples and transformations.
| I uses "spawn" context to work with hf_xet, the rust client for fast uploads to HF.
| Multiprocessing is disabled by default.
|
| Return:
| huggingface_hub.CommitInfo
|
| Example:
|
| ```python
| >>> dataset.push_to_hub("<organization>/<dataset_id>")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", private=True)
| >>> dataset.push_to_hub("<organization>/<dataset_id>", max_shard_size="1GB")
| >>> dataset.push_to_hub("<organization>/<dataset_id>", num_shards=1024)
| ```
|
| If your dataset has multiple splits (e.g. train/validation/test):
|
| ```python
| >>> train_dataset.push_to_hub("<organization>/<dataset_id>", split="train")
| >>> val_dataset.push_to_hub("<organization>/<dataset_id>", split="validation")
| >>> # later
| >>> dataset = load_dataset("<organization>/<dataset_id>")
| >>> train_dataset = dataset["train"]
| >>> val_dataset = dataset["validation"]
| ```
|
| If you want to add a new configuration (or subset) to a dataset (e.g. if the dataset has multiple tasks/versions/languages):
|
| ```python
| >>> english_dataset.push_to_hub("<organization>/<dataset_id>", "en")
| >>> french_dataset.push_to_hub("<organization>/<dataset_id>", "fr")
| >>> # later
| >>> english_dataset = load_dataset("<organization>/<dataset_id>", "en")
| >>> french_dataset = load_dataset("<organization>/<dataset_id>", "fr")
| ```
|
| remove_columns(self, column_names: Union[str, list[str]]) -> 'IterableDataset'
| Remove one or several column(s) in the dataset and the features associated to them.
| The removal is done on-the-fly on the examples when iterating over the dataset.
|
|
| Args:
| column_names (`Union[str, List[str]]`):
| Name of the column(s) to remove.
|
| Returns:
| `IterableDataset`: A copy of the dataset object without the columns to remove.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> next(iter(ds))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .', 'label': 1}
| >>> ds = ds.remove_columns("label")
| >>> next(iter(ds))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| rename_column(self, original_column_name: str, new_column_name: str) -> 'IterableDataset'
| Rename a column in the dataset, and move the features associated to the original column under the new column
| name.
|
| Args:
| original_column_name (`str`):
| Name of the column to rename.
| new_column_name (`str`):
| New name for the column.
|
| Returns:
| `IterableDataset`: A copy of the dataset with a renamed column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> next(iter(ds))
| {'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| >>> ds = ds.rename_column("text", "movie_review")
| >>> next(iter(ds))
| {'label': 1,
| 'movie_review': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| rename_columns(self, column_mapping: dict[str, str]) -> 'IterableDataset'
| Rename several columns in the dataset, and move the features associated to the original columns under
| the new column names.
|
| Args:
| column_mapping (`Dict[str, str]`): A mapping of columns to rename to their new names
|
| Returns:
| `IterableDataset`: A copy of the dataset with renamed columns
|
| repeat(self, num_times: Optional[int]) -> 'IterableDataset'
| Create a new [`IterableDataset`] that repeats the underlying dataset `num_times` times.
|
| N.B. The effect of calling shuffle after repeat depends significantly on buffer size.
| With buffer_size 1, duplicate data is never seen in the same iteration, even after shuffling:
| ds.repeat(n).shuffle(seed=42, buffer_size=1) is equivalent to ds.shuffle(seed=42, buffer_size=1).repeat(n),
| and only shuffles shard orders within each iteration.
| With buffer size >= (num samples in the dataset * num_times), we get full shuffling of the repeated data, i.e. we can observe duplicates in
| the same iteration.
|
| Args:
| num_times (`int`) or (`None`):
| Number of times to repeat the dataset. If `None`, the dataset will be repeated indefinitely.
|
| Example:
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> ds = ds.take(2).repeat(2)
| >>> list(ds)
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'},
| {'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| ```
|
| reshard(self) -> 'IterableDataset'
| Reshard the dataset if possible, i.e. split the current shards further into more shards.
| This increases the number of shards and the resulting dataset has num_shards >= previous_num_shards.
| Equality may happen if no shard can be split further.
|
| The resharding mechanism depends on the dataset file format:
|
| * Parquet: shard per row group instead of per file
| * Other: not implemented yet (contributions are welcome !)
|
| Be sure to reshard/shard before using any randomizing operator (such as `shuffle`).
| It is best if the shard operator is used early in the dataset pipeline.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("fancyzhx/amazon_polarity", split="train", streaming=True)
| >>> ds
| IterableDataset({
| features: ['label', 'title', 'content'],
| num_shards: 4
| })
| >>> ds.reshard()
| IterableDataset({
| features: ['label', 'title', 'content'],
| num_shards: 3600
| })
| ```
|
| select_columns(self, column_names: Union[str, list[str]]) -> 'IterableDataset'
| Select one or several column(s) in the dataset and the features
| associated to them. The selection is done on-the-fly on the examples
| when iterating over the dataset.
|
|
| Args:
| column_names (`Union[str, List[str]]`):
| Name of the column(s) to select.
|
| Returns:
| `IterableDataset`: A copy of the dataset object with selected columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> next(iter(ds))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .', 'label': 1}
| >>> ds = ds.select_columns("text")
| >>> next(iter(ds))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| set_epoch(self, epoch: int)
|
| shard(self, num_shards: int, index: int, contiguous: bool = True) -> 'IterableDataset'
| Return the `index`-nth shard from dataset split into `num_shards` pieces.
|
| This shards deterministically. `dataset.shard(n, i)` splits the dataset into contiguous chunks,
| so it can be easily concatenated back together after processing. If `dataset.num_shards % n == l`, then the
| first `l` datasets each have `(dataset.num_shards // n) + 1` shards, and the remaining datasets have `(dataset.num_shards // n)` shards.
| `datasets.concatenate_datasets([dset.shard(n, i) for i in range(n)])` returns a dataset with the same order as the original.
| In particular, `dataset.shard(dataset.num_shards, i)` returns a dataset with 1 shard.
|
| Note: n should be less or equal to the number of shards in the dataset `dataset.num_shards`.
|
| On the other hand, `dataset.shard(n, i, contiguous=False)` contains all the shards of the dataset whose index mod `n = i`.
|
| Be sure to shard before using any randomizing operator (such as `shuffle`).
| It is best if the shard operator is used early in the dataset pipeline.
|
| Args:
| num_shards (`int`):
| How many shards to split the dataset into.
| index (`int`):
| Which shard to select and return.
| contiguous: (`bool`, defaults to `True`):
| Whether to select contiguous blocks of indices for shards.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("fancyzhx/amazon_polarity", split="train", streaming=True)
| >>> ds
| IterableDataset({
| features: ['label', 'title', 'content'],
| num_shards: 4
| })
| >>> ds.shard(num_shards=2, index=0)
| IterableDataset({
| features: ['label', 'title', 'content'],
| num_shards: 2
| })
| ```
|
| shuffle(self, seed=None, generator: Optional[numpy.random._generator.Generator] = None, buffer_size: int = 1000, max_buffer_input_shards: int = 10) -> 'IterableDataset'
| Randomly shuffles the elements of this dataset.
|
| This dataset fills a buffer with `buffer_size` elements, then randomly samples elements from this buffer,
| replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or
| equal to the full size of the dataset is required.
|
| For instance, if your dataset contains 10,000 elements but `buffer_size` is set to 1000, then `shuffle` will
| initially select a random element from only the first 1000 elements in the buffer. Once an element is
| selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element,
| maintaining the 1000 element buffer.
|
| If the dataset is made of several shards, it fills the buffer using up to `max_buffer_input_shards` shards
| at a time and also does shuffle the order of the shards. This greatly improves the quality of the shuffling.
|
| However if the order has been fixed by using [`~datasets.IterableDataset.skip`]
| or [`~datasets.IterableDataset.take`] then the order of the shards is kept unchanged and only one shard at
| a time is used to fill the buffer.
|
| Args:
| seed (`int`, *optional*, defaults to `None`):
| Random seed that will be used to shuffle the dataset.
| It is used to sample from the shuffle buffer and also to shuffle the data shards.
| generator (`numpy.random.Generator`, *optional*):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| buffer_size (`int`, defaults to `1000`):
| Size of the buffer.
| max_buffer_input_shards (`int`, defaults to `101000`):
| Maximum number of shards to use to feed the buffer at a time.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> list(ds.take(3))
| [{'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',
| 'label': 1},
| {'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .',
| 'label': 1},
| {'text': 'effective but too-tepid biopic', 'label': 1}]
| >>> shuffled_ds = ds.shuffle(seed=42)
| >>> list(shuffled_ds.take(3))
| [{'text': "a sports movie with action that's exciting on the field and a story you care about off it .",
| 'label': 1},
| {'text': 'at its best , the good girl is a refreshingly adult take on adultery . . .',
| 'label': 1},
| {'text': "sam jones became a very lucky filmmaker the day wilco got dropped from their record label , proving that one man's ruin may be another's fortune .",
| 'label': 1}]
| >>> resharded_ds = ds.reshard() # useful to shard Parquet datasets by row group instead of by file, improving shuffling quality on dataset with one or few files.
| >>> shuffled_resharded_ds = resharded_ds.shuffle(seed=42)
| >>> list(shuffled_resharded_ds.take(3))
| [{'text': 'this mistaken-identity picture is so film-culture referential that the final product is a ghost .',
| 'label': 0},
| {'text': 'woody allen used to ridicule movies like hollywood ending . now he makes them .',
| 'label': 0},
| {'text': "not only is undercover brother as funny , if not more so , than both austin powers films , but it's also one of the smarter , savvier spoofs to come along in some time .",
| 'label': 1}]
| ```
|
| skip(self, n: int) -> 'IterableDataset'
| Create a new [`IterableDataset`] that skips the first `n` elements.
|
| Args:
| n (`int`):
| Number of elements to skip.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| >>> ds = ds.skip(1)
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'},
| {'label': 1,
| 'text': 'if you sometimes like to go to the movies to have fun , wasabi is a good place to start .'}]
| ```
|
| state_dict(self) -> dict
| Get the current state_dict of the dataset.
| It corresponds to the state at the latest example it yielded.
|
| Resuming returns exactly where the checkpoint was saved except in two cases:
|
| 1. examples from shuffle buffers are lost when resuming and the buffers are refilled with new data
| 2. combinations of `.with_format(arrow)` and batched `.map()` may skip one batch.
|
| Returns:
| `dict`
|
| Example:
|
| ```py
| >>> from datasets import Dataset, concatenate_datasets
| >>> ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3)
| >>> for idx, example in enumerate(ds):
| ... print(example)
| ... if idx == 2:
| ... state_dict = ds.state_dict()
| ... print("checkpoint")
| ... break
| >>> ds.load_state_dict(state_dict)
| >>> print(f"restart from checkpoint")
| >>> for example in ds:
| ... print(example)
| ```
|
| which returns:
| ```
| {'a': 0}
| {'a': 1}
| {'a': 2}
| checkpoint
| restart from checkpoint
| {'a': 3}
| {'a': 4}
| {'a': 5}
| ```
|
| ```py
| >>> from torchdata.stateful_dataloader import StatefulDataLoader
| >>> ds = load_dataset("deepmind/code_contests", streaming=True, split="train")
| >>> dataloader = StatefulDataLoader(ds, batch_size=32, num_workers=4)
| >>> # checkpoint
| >>> state_dict = dataloader.state_dict() # uses ds.state_dict() under the hood
| >>> # resume from checkpoint
| >>> dataloader.load_state_dict(state_dict) # uses ds.load_state_dict() under the hood
| ```
|
| take(self, n: int) -> 'IterableDataset'
| Create a new [`IterableDataset`] with only the first `n` elements.
|
| Args:
| n (`int`):
| Number of elements to take.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> small_ds = ds.take(2)
| >>> list(small_ds)
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}]
| ```
|
| to_csv(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, storage_options: Optional[dict] = None, **to_csv_kwargs) -> int
| Exports the dataset to csv.
|
| This iterates on the dataset and loads it completely in memory before writing it.
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.csv`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.csv`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
| **to_csv_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_csv`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html).
| The parameter `index` defaults to `False` if not specified.
| If you would like to write the index, pass `index=True` and also set a name for the index column by
| passing `index_label`.
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_csv("path/to/dataset/directory")
| ```
|
| to_dict(self, batch_size: Optional[int] = None, batched: bool = False) -> Union[dict, collections.abc.Iterator[dict]]
| Returns the dataset as a Python dict. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*): The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
|
| Returns:
| `dict` or `Iterator[dict]`
|
| Example:
|
| ```py
| >>> ds.to_dict()
| ```
|
| to_json(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, storage_options: Optional[dict] = None, **to_json_kwargs) -> int
| Export the dataset to JSON Lines or JSON.
|
| This iterates on the dataset and loads it completely in memory before writing it.
|
| The default output format is [JSON Lines](https://jsonlines.org/).
| To export to [JSON](https://www.json.org), pass `lines=False` argument and the desired `orient`.
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.json`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.json`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
| **to_json_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_json`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html).
| Default arguments are `lines=True` and `orient="records".
| The parameter `index` defaults to `False` if `orient` is `"split"` or `"table"`.
| If you would like to write the index, pass `index=True`.
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_json("path/to/dataset/directory/filename.jsonl")
| ```
|
| ```py
| >>> num_shards = dataset.num_shards
| >>> for index in range(num_shards):
| ... shard = dataset.shard(index, num_shards)
| ... shard.to_json(f"path/of/my/dataset/data-{index:05d}.jsonl")
| ```
|
| to_list(self) -> list
| Returns the dataset as a Python list.
|
| Returns:
| `list`
|
| Example:
|
| ```py
| >>> ds.to_list()
| ```
|
| to_pandas(self, batch_size: Optional[int] = None, batched: bool = False) -> Union[pandas.core.frame.DataFrame, collections.abc.Iterator[pandas.core.frame.DataFrame]]
| Returns the dataset as a `pandas.DataFrame`. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*):
| The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
|
| Returns:
| `pandas.DataFrame` or `Iterator[pandas.DataFrame]`
|
| Example:
|
| ```py
| >>> ds.to_pandas()
| ```
|
| to_parquet(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, storage_options: Optional[dict] = None, **parquet_writer_kwargs) -> int
| Exports the dataset to parquet
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.parquet`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.parquet`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.19.0"/>
| **parquet_writer_kwargs (additional keyword arguments):
| Parameters to pass to PyArrow's `pyarrow.parquet.ParquetWriter`.
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_parquet("path/to/dataset/directory")
| ```
|
| ```py
| >>> num_shards = dataset.num_shards
| >>> for index in range(num_shards):
| ... shard = dataset.shard(index, num_shards)
| ... shard.to_parquet(f"path/of/my/dataset/data-{index:05d}.parquet")
| ```
|
| to_polars(self, batch_size: Optional[int] = None, batched: bool = False, schema_overrides: Optional[dict] = None, rechunk: bool = True) -> Union[ForwardRef('pl.DataFrame'), collections.abc.Iterator['pl.DataFrame']]
| Returns the dataset as a `polars.DataFrame`. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*):
| The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
| schema_overrides (`dict`, *optional*):
| Support type specification or override of one or more columns; note that
| any dtypes inferred from the schema param will be overridden.
| rechunk (`bool`):
| Make sure that all data is in contiguous memory. Defaults to `True`.
| Returns:
| `polars.DataFrame` or `Iterator[polars.DataFrame]`
|
| Example:
|
| ```py
| >>> ds.to_polars()
| ```
|
| to_sql(self, name: str, con: Union[str, ForwardRef('sqlalchemy.engine.Connection'), ForwardRef('sqlalchemy.engine.Engine'), ForwardRef('sqlite3.Connection')], batch_size: Optional[int] = None, **sql_writer_kwargs) -> int
| Exports the dataset to a SQL database.
|
| Args:
| name (`str`):
| Name of SQL table.
| con (`str` or `sqlite3.Connection` or `sqlalchemy.engine.Connection` or `sqlalchemy.engine.Connection`):
| A [URI string](https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls) or a SQLite3/SQLAlchemy connection object used to write to a database.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| **sql_writer_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_sql`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_sql.html).
| The parameter `index` defaults to `False` if not specified.
| If you would like to write the index, pass `index=True` and also set a name for the index column by
| passing `index_label`.
|
|
| Returns:
| `int`: The number of records written.
|
| Example:
|
| ```py
| >>> # con provided as a connection URI string
| >>> ds.to_sql("data", "sqlite:///my_own_db.sql")
| >>> # con provided as a sqlite3 connection object
| >>> import sqlite3
| >>> con = sqlite3.connect("my_own_db.sql")
| >>> with con:
| ... ds.to_sql("data", con)
| ```
|
| with_format(self, type: Optional[str] = None) -> 'IterableDataset'
| Return a dataset with the specified format.
|
| Args:
|
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means it returns python objects (default).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation", streaming=True)
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds = ds.with_format("torch")
| >>> next(iter(ds))
| {'text': 'compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'label': tensor(1),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
| 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_dict(mapping: dict, features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, num_shards: Optional[int] = 1) -> 'IterableDataset'
| Convert `dict` to a `pyarrow.Table` to create an [`IterableDataset`].
|
| Important: a dataset created with from_dict() lives in memory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. to_parquet / from_parquet.
|
| Args:
| mapping (`Mapping`):
| Mapping of strings to Arrays or Python lists.
| features ([`Features`], *optional*):
| Dataset features.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| num_shards (`int`, default to `1`):
| Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly,
| and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example.
|
| Returns:
| [`IterableDataset`]
|
| from_list(mapping: list[dict], features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, num_shards: Optional[int] = 1) -> 'IterableDataset'
| Convert a list of dicts to a `pyarrow.Table` to create an [`IterableDataset`]`.
|
| Note that the keys of the first entry will be used to determine the dataset columns,
| regardless of what is passed to features.
|
| Important: a dataset created with from_list() lives in memory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. from_parquet / to_parquet.
|
| Args:
| mapping (`List[dict]`): A list of mappings of strings to row values.
| features (`Features`, optional): Dataset features.
| info (`DatasetInfo`, optional): Dataset information, like description, citation, etc.
| split (`NamedSplit`, optional): Name of the dataset split.
| num_shards (`int`, default to `1`):
| Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly,
| and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example.
|
| Returns:
| [`IterableDataset`]
|
| from_pandas(df: pandas.core.frame.DataFrame, features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, preserve_index: Optional[bool] = None, num_shards: Optional[int] = 1) -> 'IterableDataset'
| Convert `pandas.DataFrame` to a `pyarrow.Table` to create an [`IterableDataset`].
|
| The column types in the resulting Arrow Table are inferred from the dtypes of the `pandas.Series` in the
| DataFrame. In the case of non-object Series, the NumPy dtype is translated to its Arrow equivalent. In the
| case of `object`, we need to guess the datatype by looking at the Python objects in this Series.
|
| Be aware that Series of the `object` dtype don't carry enough information to always lead to a meaningful Arrow
| type. In the case that we cannot infer a type, e.g. because the DataFrame is of length 0 or the Series only
| contains `None/nan` objects, the type is set to `null`. This behavior can be avoided by constructing explicit
| features and passing it to this function.
|
| Important: a dataset created with from_pandas() lives in memory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it on disk
| and reload using e.g. to_parquet / from_parquet.
|
| Args:
| df (`pandas.DataFrame`):
| Dataframe that contains the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| preserve_index (`bool`, *optional*):
| Whether to store the index as an additional column in the resulting Dataset.
| The default of `None` will store the index as a column, except for `RangeIndex` which is stored as metadata only.
| Use `preserve_index=True` to force it to be stored as a column.
| num_shards (`int`, default to `1`):
| Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly,
| and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example.
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_pandas(df)
| ```
|
| from_polars(df: Union[ForwardRef('pl.DataFrame'), ForwardRef('pl.LazyFrame')], features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None) -> 'IterableDataset'
| Create an IterableDataset from a polars DataFrame or LazyFrame.
|
| Iterating over the dataset is mostly zero copy.
| Under the hood, the dataset iterates over the polars DataFrame batches/slices.
|
| Data types that do copy:
| * CategoricalType
|
| Args:
| df (`polars.DataFrame`): DataFrame to convert to Arrow Table
| features (`Features`, optional): Dataset features.
| info (`DatasetInfo`, optional): Dataset information, like description, citation, etc.
| split (`NamedSplit`, optional): Name of the dataset split.
|
| Returns:
| [`IterableDataset`]
|
| Examples:
| ```py
| >>> ds = IterableDataset.from_polars(df)
| ```
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| from_csv(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, **kwargs) -> 'IterableDataset'
| Create an IterableDataset from CSV file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the CSV file(s).
| split ([`NamedSplit`], *optional*):
| Split name to be assigned to the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`pandas.read_csv`].
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_csv('path/to/dataset.csv')
| ```
|
| from_file(filename: str) -> 'IterableDataset'
| Instantiate a IterableDataset from Arrow table at filename.
|
| Args:
| filename (`str`):
| File name of the dataset.
|
| Returns:
| [`IterableDataset`]
|
| from_generator(generator: Callable, features: Optional[datasets.features.features.Features] = None, gen_kwargs: Optional[dict] = None, split: datasets.splits.NamedSplit = NamedSplit('train')) -> 'IterableDataset'
| Create an Iterable Dataset from a generator.
|
| Args:
| generator (`Callable`):
| A generator function that `yields` examples.
| features (`Features`, *optional*):
| Dataset features.
| gen_kwargs(`dict`, *optional*):
| Keyword arguments to be passed to the `generator` callable.
| You can define a sharded iterable dataset by passing the list of shards in `gen_kwargs`.
| This can be used to improve shuffling and when iterating over the dataset with multiple workers.
| split ([`NamedSplit`], defaults to `Split.TRAIN`):
| Split name to be assigned to the dataset.
|
| <Added version="2.21.0"/>
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> def gen():
| ... yield {"text": "Good", "label": 0}
| ... yield {"text": "Bad", "label": 1}
| ...
| >>> ds = IterableDataset.from_generator(gen)
| ```
|
| ```py
| >>> def gen(shards):
| ... for shard in shards:
| ... with open(shard) as f:
| ... for line in f:
| ... yield {"line": line}
| ...
| >>> shards = [f"data{i}.txt" for i in range(32)]
| >>> ds = IterableDataset.from_generator(gen, gen_kwargs={"shards": shards})
| >>> ds = ds.shuffle(seed=42, buffer_size=10_000) # shuffles the shards order + uses a shuffle buffer
| >>> from torch.utils.data import DataLoader
| >>> dataloader = DataLoader(ds.with_format("torch"), num_workers=4) # give each worker a subset of 32/4=8 shards
| ```
|
| from_json(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, field: Optional[str] = None, **kwargs) -> 'IterableDataset'
| Create an IterableDataset from JSON or JSON Lines file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the JSON or JSON Lines file(s).
| split ([`NamedSplit`], *optional*):
| Split name to be assigned to the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| field (`str`, *optional*):
| Field name of the JSON file where the dataset is contained in.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`JsonConfig`].
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_json('path/to/dataset.json')
| ```
|
| from_parquet(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, columns: Optional[list[str]] = None, filters: Union[pyarrow._compute.Expression, list[tuple], list[list[tuple]], NoneType] = None, fragment_scan_options: Optional[pyarrow._dataset_parquet.ParquetFragmentScanOptions] = None, on_bad_files: Literal['error', 'warn', 'skip'] = 'error', **kwargs) -> 'IterableDataset'
| Create an IterableDataset from Parquet file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the Parquet file(s).
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| columns (`List[str]`, *optional*):
| If not `None`, only these columns will be read from the file.
| A column name may be a prefix of a nested field, e.g. 'a' will select
| 'a.b', 'a.c', and 'a.d.e'.
| filters (`Union[pyarrow.dataset.Expression, list[tuple], list[list[tuple]]]`, *optional*):
| Return only the rows matching the filter.
| If possible the predicate will be pushed down to exploit the partition information
| or internal metadata found in the data source, e.g. Parquet statistics.
| Otherwise filters the loaded RecordBatches before yielding them.
| fragment_scan_options (`pyarrow.dataset.ParquetFragmentScanOptions`, *optional*)
| Scan-specific options for Parquet fragments.
| This is especially useful to configure buffering and caching.
|
| <Added version="4.2.0"/>
| on_bad_files (`Literal["error", "warn", "skip"]`, *optional*, defaults to "error")
| Specify what to do upon encountering a bad file (a file that can't be read). Allowed values are :
| * 'error', raise an Exception when a bad file is encountered.
| * 'warn', raise a warning when a bad file is encountered and skip that file.
| * 'skip', skip bad files without raising or warning when they are encountered.
|
| <Added version="4.2.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`ParquetConfig`].
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_parquet('path/to/dataset.parquet')
| ```
|
| Load a subset of columns:
|
| ```python
| >>> ds = IterableDataset.from_parquet('path/to/dataset.parquet', columns=["col_0", "col_1"])
| ```
|
| Efficiently filter data, possibly skipping entire files or row groups:
|
| ```python
| >>> filters = [("col_0", "==", 0)]
| >>> ds = IterableDataset.from_parquet(parquet_files_list, filters=filters)
| ```
|
| from_spark(df: 'pyspark.sql.DataFrame', split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, **kwargs) -> 'IterableDataset'
| Create an IterableDataset from Spark DataFrame. The dataset is streamed to the driver in batches.
|
| Args:
| df (`pyspark.sql.DataFrame`):
| The DataFrame containing the desired data.
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> df = spark.createDataFrame(
| >>> data=[[1, "Elia"], [2, "Teo"], [3, "Fang"]],
| >>> columns=["id", "name"],
| >>> )
| >>> ds = IterableDataset.from_spark(df)
| ```
|
| from_text(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, keep_linebreaks: bool = False, sample_by: Literal['line', 'paragraph', 'document'] = 'line', **kwargs) -> 'IterableDataset'
| Create an IterableDataset from text file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the text file(s).
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| keep_linebreaks: (`bool`, defaults to False):
| Whether to keep line breaks.
| sample_by (`Literal["line", "paragraph", "document"]`, defaults to "line"):
| Whether to load data per line, praragraph or document.
| By default one row in the dataset = one line.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`TextConfig`].
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_text('path/to/dataset.txt')
| ```
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| column_names
| Names of the columns in the dataset.
| This can be None if the dataset has unknown features (e.g. after a map() operation).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation", streaming=True)
| >>> ds.column_names
| ['text', 'label']
| ```
|
| epoch
|
| n_shards
|
| num_columns
| Number of columns in the dataset.
| This can be None if the dataset has unknown features (e.g. after a map() operation).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.num_columns
| 2
| ```
|
| num_shards
|
| ----------------------------------------------------------------------
| Readonly properties inherited from datasets.arrow_dataset.DatasetInfoMixin:
|
| builder_name
|
| citation
|
| config_name
|
| dataset_size
|
| description
|
| download_checksums
|
| download_size
|
| features
|
| homepage
|
| info
| [`~datasets.DatasetInfo`] object containing all the metadata in the dataset.
|
| license
|
| size_in_bytes
|
| split
| [`~datasets.NamedSplit`] object corresponding to a named dataset split.
|
| supervised_keys
|
| version
|
| ----------------------------------------------------------------------
| Data descriptors inherited from datasets.arrow_dataset.DatasetInfoMixin:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
datasets.IterableDatasetDict
Python Library Documentation: class IterableDatasetDict in module datasets.dataset_dict
class IterableDatasetDict(builtins.dict)
| Method resolution order:
| IterableDatasetDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __repr__(self)
| Return repr(self).
|
| cast(self, features: datasets.features.features.Features) -> 'IterableDatasetDict'
| Cast the dataset to a new set of features.
| The type casting is applied to all the datasets of the dataset dictionary.
|
| Args:
| features (`Features`):
| New features to cast the dataset to.
| The name of the fields in the features must match the current column names.
| The type of the data must also be convertible from one type to the other.
| For non-trivial conversion, e.g. `string` <-> `ClassLabel` you should use [`map`] to update the Dataset.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset with casted features.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds["train"].features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> new_features = ds["train"].features.copy()
| >>> new_features['label'] = ClassLabel(names=['bad', 'good'])
| >>> new_features['text'] = Value('large_string')
| >>> ds = ds.cast(new_features)
| >>> ds["train"].features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('large_string')}
| ```
|
| cast_column(self, column: str, feature: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti]) -> 'IterableDatasetDict'
| Cast column to feature for decoding.
| The type casting is applied to all the datasets of the dataset dictionary.
|
| Args:
| column (`str`):
| Column name.
| feature ([`Feature`]):
| Target feature.
|
| Returns:
| [`IterableDatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds["train"].features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> ds = ds.cast_column('label', ClassLabel(names=['bad', 'good']))
| >>> ds["train"].features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('string')}
| ```
|
| filter(self, function: Optional[Callable] = None, with_indices=False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None) -> 'IterableDatasetDict'
| Apply a filter function to all the elements so that the dataset only includes examples according to the filter function.
| The filtering is done on-the-fly when iterating over the dataset.
| The filtering is applied to all the datasets of the dataset dictionary.
|
| Args:
| function (`Callable`):
| Callable with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> bool` if `with_indices=False, batched=False`
| - `function(example: Dict[str, Any], indices: int) -> bool` if `with_indices=True, batched=False`
| - `function(example: Dict[str, list]) -> list[bool]` if `with_indices=False, batched=True`
| - `function(example: Dict[str, list], indices: list[int]) -> list[bool]` if `with_indices=True, batched=True`
|
| If no function is provided, defaults to an always True function: `lambda x: True`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx): ...`.
| input_columns (`str` or `list[str]`, *optional*):
| The columns to be passed into `function` as
| positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.filter(lambda x: x["label"] == 0)
| >>> list(ds["train"].take(3))
| [{'label': 0, 'text': 'Review: simplistic , silly and tedious .'},
| {'label': 0,
| 'text': "Review: it's so laddish and juvenile , only teenage boys could possibly find it funny ."},
| {'label': 0,
| 'text': 'Review: exploitative and largely devoid of the depth or sophistication that would make watching such a graphic treatment of the crimes bearable .'}]
| ```
|
| map(self, function: Optional[Callable] = None, with_indices: bool = False, with_split: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: int = 1000, drop_last_batch: bool = False, remove_columns: Union[str, list[str], NoneType] = None, fn_kwargs: Optional[dict] = None) -> 'IterableDatasetDict'
| Apply a function to all the examples in the iterable dataset (individually or in batches) and update them.
| If your function returns a column that already exists, then it overwrites it.
| The function is applied on-the-fly on the examples when iterating over the dataset.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| You can specify whether the function should be batched or not with the `batched` parameter:
|
| - If batched is `False`, then the function takes 1 example in and should return 1 example.
| An example is a dictionary, e.g. `{"text": "Hello there !"}`.
| - If batched is `True` and `batch_size` is 1, then the function takes a batch of 1 example as input and can return a batch with 1 or more examples.
| A batch is a dictionary, e.g. a batch of 1 example is `{"text": ["Hello there !"]}`.
| - If batched is `True` and `batch_size` is `n` > 1, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.
| Note that the last batch may have less than `n` examples.
| A batch is a dictionary, e.g. a batch of `n` examples is `{"text": ["Hello there !"] * n}`.
|
| If the function is asynchronous, then `map` will run your function in parallel, with up to one thousand simultaneous calls.
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`, *optional*, defaults to `None`):
| Function applied on-the-fly on the examples when you iterate on the dataset.
| It must have one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> Dict[str, Any]` if `batched=False` and `with_indices=False`
| - `function(example: Dict[str, Any], idx: int) -> Dict[str, Any]` if `batched=False` and `with_indices=True`
| - `function(batch: Dict[str, list]) -> Dict[str, list]` if `batched=True` and `with_indices=False`
| - `function(batch: Dict[str, list], indices: list[int]) -> Dict[str, list]` if `batched=True` and `with_indices=True`
|
| For advanced usage, the function can also return a `pyarrow.Table`.
| If the function is asynchronous, then `map` will run your function in parallel.
| Moreover if your function returns nothing (`None`), then `map` will run your function and return the dataset unchanged.
| If no function is provided, default to identity function: `lambda x: x`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx[, rank]): ...`.
| input_columns (`[Union[str, list[str]]]`, *optional*, defaults to `None`):
| The columns to be passed into `function`
| as positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| drop_last_batch (`bool`, defaults to `False`):
| Whether a last batch smaller than the `batch_size` should be
| dropped instead of being processed by the function.
| remove_columns (`[list[str]]`, *optional*, defaults to `None`):
| Remove a selection of columns while doing the mapping.
| Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding
| columns with names in `remove_columns`, these columns will be kept.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> def add_prefix(example):
| ... example["text"] = "Review: " + example["text"]
| ... return example
| >>> ds = ds.map(add_prefix)
| >>> next(iter(ds["train"]))
| {'label': 1,
| 'text': 'Review: the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| push_to_hub(self, repo_id: str, config_name: str = 'default', set_default: Optional[bool] = None, data_dir: Optional[str] = None, commit_message: Optional[str] = None, commit_description: Optional[str] = None, private: Optional[bool] = None, token: Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = False, max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[dict[str, int]] = None, embed_external_files: bool = True, num_proc: Optional[int] = None) -> huggingface_hub.hf_api.CommitInfo
| Pushes the [`IterableDatasetDict`] to the hub as a Parquet dataset.
| The [`IterableDatasetDict`] is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
|
| Each dataset split will be pushed independently. The pushed dataset will keep the original split names.
|
| The resulting Parquet files are self-contained by default: if your dataset contains [`Image`] or [`Audio`]
| data, the Parquet files will store the bytes of your images or audio files.
| You can disable this by setting `embed_external_files` to False.
|
| Args:
| repo_id (`str`):
| The ID of the repository to push to in the following format: `<user>/<dataset_name>` or
| `<org>/<dataset_name>`. Also accepts `<dataset_name>`, which will default to the namespace
| of the logged-in user.
|
| It could also be a location inside a bucket, e.g. `buckets/<user_or_org>/<bucket_name>/...`
| config_name (`str`):
| Configuration name of a dataset. Defaults to "default".
| set_default (`bool`, *optional*):
| Whether to set this configuration as the default one. Otherwise, the default configuration is the one
| named "default".
| data_dir (`str`, *optional*):
| Directory name that will contain the uploaded data files. Defaults to the `config_name` if different
| from "default", else "data".
|
| <Added version="2.17.0"/>
| commit_message (`str`, *optional*):
| Message to commit while pushing. Will default to `"Upload dataset"`.
| commit_description (`str`, *optional*):
| Description of the commit that will be created.
| Additionally, description of the PR if a PR is created (`create_pr` is True).
|
| <Added version="2.16.0"/>
| private (`bool`, *optional*):
| Whether to make the repo private. If `None` (default), the repo will be public unless the
| organization's default is private. This value is ignored if the repo already exists.
| token (`str`, *optional*):
| An optional authentication token for the Hugging Face Hub. If no token is passed, will default
| to the token saved locally when logging in with `huggingface-cli login`. Will raise an error
| if no token is passed and the user is not logged-in.
| revision (`str`, *optional*):
| Branch to push the uploaded files to. Defaults to the `"main"` branch.
| create_pr (`bool`, *optional*, defaults to `False`):
| Whether to create a PR with the uploaded files or directly commit.
| max_shard_size (`int` or `str`, *optional*):
| Optional maximum size of the dataset shards to be uploaded to the hub. If expressed as a string, needs to be digits followed
| by a unit (like `"500MB"` or `"1GB"`). If not provided, each split keeps its dataset-native shard count.
| num_shards (`Dict[str, int]`, *optional*):
| Number of shards to write. Use a dictionary to define a different `num_shards` for each split.
| If `max_shard_size` is provided and a split's `num_shards` is not, then the number of shards for that split is estimated
| from `max_shard_size`.
| embed_external_files (`bool`, defaults to `True`):
| Whether to embed file bytes in the shards.
| In particular, this will do the following before the push for the fields of type:
|
| - [`Audio`] and [`Image`] removes local path information and embed file content in the Parquet files.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when preparing and uploading the dataset.
| This is helpful if the dataset is made of many samples or media files to embed.
| I uses "spawn" context to work with hf_xet, the rust client for fast uploads to HF.
| Multiprocessing is disabled by default.
|
| <Added version="4.0.0"/>
|
| Return:
| huggingface_hub.CommitInfo
|
| Example:
|
| ```python
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", private=True)
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", max_shard_size="1GB")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", num_shards={"train": 1024, "test": 8})
| ```
|
| If you want to add a new configuration (or subset) to a dataset (e.g. if the dataset has multiple tasks/versions/languages):
|
| ```python
| >>> english_dataset.push_to_hub("<organization>/<dataset_id>", "en")
| >>> french_dataset.push_to_hub("<organization>/<dataset_id>", "fr")
| >>> # later
| >>> english_dataset = load_dataset("<organization>/<dataset_id>", "en")
| >>> french_dataset = load_dataset("<organization>/<dataset_id>", "fr")
| ```
|
| remove_columns(self, column_names: Union[str, list[str]]) -> 'IterableDatasetDict'
| Remove one or several column(s) in the dataset and the features associated to them.
| The removal is done on-the-fly on the examples when iterating over the dataset.
| The removal is applied to all the datasets of the dataset dictionary.
|
|
| Args:
| column_names (`Union[str, list[str]]`):
| Name of the column(s) to remove.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset object without the columns to remove.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.remove_columns("label")
| >>> next(iter(ds["train"]))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| rename_column(self, original_column_name: str, new_column_name: str) -> 'IterableDatasetDict'
| Rename a column in the dataset, and move the features associated to the original column under the new column
| name.
| The renaming is applied to all the datasets of the dataset dictionary.
|
| Args:
| original_column_name (`str`):
| Name of the column to rename.
| new_column_name (`str`):
| New name for the column.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset with a renamed column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.rename_column("text", "movie_review")
| >>> next(iter(ds["train"]))
| {'label': 1,
| 'movie_review': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| rename_columns(self, column_mapping: dict[str, str]) -> 'IterableDatasetDict'
| Rename several columns in the dataset, and move the features associated to the original columns under
| the new column names.
| The renaming is applied to all the datasets of the dataset dictionary.
|
| Args:
| column_mapping (`Dict[str, str]`):
| A mapping of columns to rename to their new names.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset with renamed columns
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.rename_columns({"text": "movie_review", "label": "rating"})
| >>> next(iter(ds["train"]))
| {'movie_review': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',
| 'rating': 1}
| ```
|
| select_columns(self, column_names: Union[str, list[str]]) -> 'IterableDatasetDict'
| Select one or several column(s) in the dataset and the features
| associated to them. The selection is done on-the-fly on the examples
| when iterating over the dataset. The selection is applied to all the
| datasets of the dataset dictionary.
|
|
| Args:
| column_names (`Union[str, list[str]]`):
| Name of the column(s) to keep.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset object with only selected columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.select("text")
| >>> next(iter(ds["train"]))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| shuffle(self, seed=None, generator: Optional[numpy.random._generator.Generator] = None, buffer_size: int = 1000) -> 'IterableDatasetDict'
| Randomly shuffles the elements of this dataset.
| The shuffling is applied to all the datasets of the dataset dictionary.
|
| This dataset fills a buffer with buffer_size elements, then randomly samples elements from this buffer,
| replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or
| equal to the full size of the dataset is required.
|
| For instance, if your dataset contains 10,000 elements but `buffer_size` is set to 1000, then `shuffle` will
| initially select a random element from only the first 1000 elements in the buffer. Once an element is
| selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element,
| maintaining the 1000 element buffer.
|
| If the dataset is made of several shards, it also does `shuffle` the order of the shards.
| However if the order has been fixed by using [`~datasets.IterableDataset.skip`] or [`~datasets.IterableDataset.take`]
| then the order of the shards is kept unchanged.
|
| Args:
| seed (`int`, *optional*, defaults to `None`):
| Random seed that will be used to shuffle the dataset.
| It is used to sample from the shuffle buffer and also to shuffle the data shards.
| generator (`numpy.random.Generator`, *optional*):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| buffer_size (`int`, defaults to `1000`):
| Size of the buffer.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> list(ds["train"].take(3))
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| >>> ds = ds.shuffle(seed=42)
| >>> list(ds["train"].take(3))
| [{'label': 1,
| 'text': "a sports movie with action that's exciting on the field and a story you care about off it ."},
| {'label': 1,
| 'text': 'at its best , the good girl is a refreshingly adult take on adultery . . .'},
| {'label': 1,
| 'text': "sam jones became a very lucky filmmaker the day wilco got dropped from their record label , proving that one man's ruin may be another's fortune ."}]
| ```
|
| with_format(self, type: Optional[str] = None) -> 'IterableDatasetDict'
| Return a dataset with the specified format.
|
| Args:
|
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means it returns python objects (default).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation", streaming=True)
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds = ds.with_format("torch")
| >>> next(iter(ds))
| {'text': 'compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'label': tensor(1),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
| 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| column_names
| Names of the columns in each split of the dataset.
| This can contain None valies if some splits have unknown features (e.g. after a map() operation).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.column_names
| {'test': ['text', 'label'],
| 'train': ['text', 'label'],
| 'validation': ['text', 'label']}
| ```
|
| num_columns
| Number of columns in each split of the dataset.
| This can contain None valies if some splits have unknown features (e.g. after a map() operation).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.num_columns
| {'test': 2, 'train': 2, 'validation': 2}
| ```
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (dict[typing.Union[str, datasets.splits.NamedSplit], ...
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
datasets.Json
Python Library Documentation: class Json in module datasets.features.features
class Json(builtins.object)
| Json(decode: bool = True, id: Optional[str] = None) -> None
|
| Feature type for JSON objects.
|
| Under the hood the objects are stored as JSON-encoded strings.
|
| Example:
|
| ```py
| >>> from datasets import Features, Json
| >>> features = Features({'json': Json()})
| >>> features
| {'json': Json()}
| ```
|
| ```py
| >>> from datasets import Dataset, Features, Json, List
| >>> features = Features({"a": List(Json())})
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, features=features)
| >>> # OR
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, on_mixed_types="use_json")
| >>> ds.features
| {'a': List(Json())}
| >>> ds[0]
| {'a': [{'b': 0}, {'c': 0}]}
| >>> def f(x):
| ... for y in x["a"]:
| ... y["d"] = "foo"
| ... return x
| >>> ds = ds.map(f)
| >>> ds.features
| >>> ds[0]
| {'a': [{'b': 0, 'd': 'foo'}, {'c': 0, 'd': 'foo'}]}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: pyarrow.lib.Array) -> pyarrow.lib.JsonArray
| Cast an Arrow array to the `Json` arrow storage type.
|
| Args:
| storage (`Union[pa.StringArray, pa.IntegerArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.JsonArray`: Array in the `Json` arrow storage type.
|
| decode_example(self, example_data, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
|
| encode_example(self, example_data)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'id')
|
| decode = True
|
| id = None
|
| pa_type = JsonType(extension<arrow.json>)
datasets.LargeList
Python Library Documentation: class LargeList in module datasets.features.features
class LargeList(builtins.object)
| LargeList(feature: Any, id: Optional[str] = None) -> None
|
| Feature type for large list data composed of child feature data type.
|
| It is backed by `pyarrow.LargeListType`, which is like `pyarrow.ListType` but with 64-bit rather than 32-bit offsets.
|
| Args:
| feature ([`FeatureType`]):
| Child feature data type of each item within the large list.
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, feature: Any, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'feature': typing.Any, 'id'...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=False,eq=True,o...
|
| __hash__ = None
|
| __match_args__ = ('feature', 'id')
|
| id = None
|
| pa_type = None
datasets.List
Python Library Documentation: class List in module datasets.features.features
class List(Sequence)
| List(feature: Any, length: int = -1, id: Optional[str] = None) -> None
|
| Feature type for large list data composed of child feature data type.
|
| It is backed by `pyarrow.ListType`, which uses 32-bit offsets or a fixed length.
|
| Args:
| feature ([`FeatureType`]):
| Child feature data type of each item within the large list.
| length (optional `int`, default to -1):
| Length of the list if it is fixed.
| Defaults to -1 which means an arbitrary length.
|
| Method resolution order:
| List
| Sequence
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, feature: Any, length: int = -1, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'feature': typing.Any, 'id'...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=False,eq=True,o...
|
| __hash__ = None
|
| __match_args__ = ('feature', 'length', 'id')
|
| id = None
|
| length = -1
|
| pa_type = None
|
| ----------------------------------------------------------------------
| Static methods inherited from Sequence:
|
| __new__(cls, feature=None, length=-1, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from Sequence:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
datasets.Mesh
Python Library Documentation: class Mesh in module datasets.features.mesh
class Mesh(builtins.object)
| Mesh(decode: bool = True, id: Optional[str] = None) -> None
|
| Mesh [`Feature`] to read 3D mesh data from a file.
|
| Input: The Mesh feature accepts as input:
| - A `str`: Absolute path to the mesh file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the mesh file (i.e. random access is allowed).
| - A `dict` with the keys:
|
| - `path`: String with relative path of the mesh file to the archive file.
| - `bytes`: Bytes of the mesh file.
|
| This is useful for parquet or webdataset files which embed mesh files.
|
| - A `trimesh.Trimesh` or `trimesh.Scene`: 3D mesh or scene object.
|
| Output: The Mesh feature outputs data as `trimesh.Trimesh` or `trimesh.Scene` objects.
|
| Args:
| decode (`bool`, defaults to `True`):
| Whether to decode the mesh data. If `False`,
| returns the underlying dictionary in the format `{"path": mesh_path, "bytes": mesh_bytes}`.
| Mesh decoding uses `trimesh` and supports `.glb`, `.ply`, and `.stl` files.
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Mesh arrow storage type.
| The Arrow types that can be converted to the Mesh pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.large_string()` - it must contain the "path" data (will be cast to string if possible)
| - `pa.binary()` - it must contain the mesh bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Mesh arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: dict, token_per_repo_id=None) -> Union[ForwardRef('trimesh.Trimesh'), ForwardRef('trimesh.Scene')]
| Decode example mesh file.
|
| Args:
| value (`dict`):
| A dictionary with keys:
|
| - `path`: String with absolute or relative mesh file path.
| - `bytes`: The bytes of the mesh file.
| token_per_repo_id (`dict`, *optional*):
| To access and decode
| mesh files from private repositories on the Hub, you can pass
| a dictionary repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `trimesh.Trimesh` or `trimesh.Scene`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed mesh files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`):
| Whether to embed local files data in the array.
| remote_files (`bool`, defaults to `True`):
| Whether to embed remote files data in the array.
|
| Returns:
| `pa.StructArray`: Array in the Mesh arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, ForwardRef('trimesh.Trimesh'), ForwardRef('trimesh.Scene')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `bytes`, `dict`, `trimesh.Trimesh`, or `trimesh.Scene`):
| Data passed as input to Mesh feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'id')
|
| decode = True
|
| dtype = 'trimesh.Trimesh'
|
| id = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
datasets.NamedSplit
Python Library Documentation: class NamedSplit in module datasets.splits class NamedSplit(SplitBase) | NamedSplit(name: str) | | Descriptor corresponding to a named split (train, test, ...). | | Example: | Each descriptor can be composed with other using addition or slice: | | ```py | split = datasets.Split.TRAIN.subsplit(datasets.percent[0:25]) + datasets.Split.TEST | ``` | | The resulting split will correspond to 25% of the train split merged with | 100% of the test split. | | A split cannot be added twice, so the following will fail: | | ```py | split = ( | datasets.Split.TRAIN.subsplit(datasets.percent[:25]) + | datasets.Split.TRAIN.subsplit(datasets.percent[75:]) | ) # Error | split = datasets.Split.TEST + datasets.Split.ALL # Error | ``` | | The slices can be applied only one time. So the following are valid: | | ```py | split = ( | datasets.Split.TRAIN.subsplit(datasets.percent[:25]) + | datasets.Split.TEST.subsplit(datasets.percent[:50]) | ) | split = (datasets.Split.TRAIN + datasets.Split.TEST).subsplit(datasets.percent[:50]) | ``` | | But this is not valid: | | ```py | train = datasets.Split.TRAIN | test = datasets.Split.TEST | split = train.subsplit(datasets.percent[:25]).subsplit(datasets.percent[:25]) | split = (train.subsplit(datasets.percent[:25]) + test).subsplit(datasets.percent[:50]) | ``` | | Method resolution order: | NamedSplit | SplitBase | builtins.object | | Methods defined here: | | __eq__(self, other) | Equality: datasets.Split.TRAIN == 'train'. | | __hash__(self) | Return hash(self). | | __init__(self, name: str) | Initialize self. See help(type(self)) for accurate signature. | | __lt__(self, other) | Return self<value. | | __repr__(self) | Return repr(self). | | __str__(self) | Return str(self). | | get_read_instruction(self, split_dict) | Parse the descriptor tree and compile all read instructions together. | | Args: | split_dict: `dict`, The `dict[split_name, SplitInfo]` of the dataset | | Returns: | split_read_instruction: `SplitReadInstruction` | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __abstractmethods__ = frozenset() | | ---------------------------------------------------------------------- | Methods inherited from SplitBase: | | __add__(self, other) | Merging: datasets.Split.TRAIN + datasets.Split.TEST. | | __ne__(self, other) | InEquality: datasets.Split.TRAIN != 'test'. | | subsplit(self, arg=None, k=None, percent=None, weighted=None) | Divides this split into subsplits. | | There are 3 ways to define subsplits, which correspond to the 3 | arguments `k` (get `k` even subsplits), `percent` (get a slice of the | dataset with `datasets.percent`), and `weighted` (get subsplits with proportions | specified by `weighted`). | | Example:: | | ``` | # 50% train, 50% test | train, test = split.subsplit(k=2) | # 50% train, 25% test, 25% validation | train, test, validation = split.subsplit(weighted=[2, 1, 1]) | # Extract last 20% | subsplit = split.subsplit(datasets.percent[-20:]) | ``` | | Warning: k and weighted will be converted into percent which mean that | values below the percent will be rounded up or down. The final split may be | bigger to deal with remainders. For instance: | | ``` | train, test, valid = split.subsplit(k=3) # 33%, 33%, 34% | s1, s2, s3, s4 = split.subsplit(weighted=[2, 2, 1, 1]) # 33%, 33%, 16%, 18% | ``` | | Args: | arg: If no kwargs are given, `arg` will be interpreted as one of | `k`, `percent`, or `weighted` depending on the type. | For example: | ``` | split.subsplit(10) # Equivalent to split.subsplit(k=10) | split.subsplit(datasets.percent[:-20]) # percent=datasets.percent[:-20] | split.subsplit([1, 1, 2]) # weighted=[1, 1, 2] | ``` | k: `int` If set, subdivide the split into `k` equal parts. | percent: `datasets.percent slice`, return a single subsplit corresponding to | a slice of the original split. For example: | `split.subsplit(datasets.percent[-20:]) # Last 20% of the dataset`. | weighted: `list[int]`, return a list of subsplits whose proportions match | the normalized sum of the list. For example: | `split.subsplit(weighted=[1, 1, 2]) # 25%, 25%, 50%`. | | Returns: | A subsplit or list of subsplits extracted from this split object. | | ---------------------------------------------------------------------- | Data descriptors inherited from SplitBase: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
datasets.NamedSplitAll
Python Library Documentation: class NamedSplitAll in module datasets.splits class NamedSplitAll(NamedSplit) | Split corresponding to the union of all defined dataset splits. | | Method resolution order: | NamedSplitAll | NamedSplit | SplitBase | builtins.object | | Methods defined here: | | __init__(self) | Initialize self. See help(type(self)) for accurate signature. | | __repr__(self) | Return repr(self). | | get_read_instruction(self, split_dict) | Parse the descriptor tree and compile all read instructions together. | | Args: | split_dict: `dict`, The `dict[split_name, SplitInfo]` of the dataset | | Returns: | split_read_instruction: `SplitReadInstruction` | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __abstractmethods__ = frozenset() | | ---------------------------------------------------------------------- | Methods inherited from NamedSplit: | | __eq__(self, other) | Equality: datasets.Split.TRAIN == 'train'. | | __hash__(self) | Return hash(self). | | __lt__(self, other) | Return self<value. | | __str__(self) | Return str(self). | | ---------------------------------------------------------------------- | Methods inherited from SplitBase: | | __add__(self, other) | Merging: datasets.Split.TRAIN + datasets.Split.TEST. | | __ne__(self, other) | InEquality: datasets.Split.TRAIN != 'test'. | | subsplit(self, arg=None, k=None, percent=None, weighted=None) | Divides this split into subsplits. | | There are 3 ways to define subsplits, which correspond to the 3 | arguments `k` (get `k` even subsplits), `percent` (get a slice of the | dataset with `datasets.percent`), and `weighted` (get subsplits with proportions | specified by `weighted`). | | Example:: | | ``` | # 50% train, 50% test | train, test = split.subsplit(k=2) | # 50% train, 25% test, 25% validation | train, test, validation = split.subsplit(weighted=[2, 1, 1]) | # Extract last 20% | subsplit = split.subsplit(datasets.percent[-20:]) | ``` | | Warning: k and weighted will be converted into percent which mean that | values below the percent will be rounded up or down. The final split may be | bigger to deal with remainders. For instance: | | ``` | train, test, valid = split.subsplit(k=3) # 33%, 33%, 34% | s1, s2, s3, s4 = split.subsplit(weighted=[2, 2, 1, 1]) # 33%, 33%, 16%, 18% | ``` | | Args: | arg: If no kwargs are given, `arg` will be interpreted as one of | `k`, `percent`, or `weighted` depending on the type. | For example: | ``` | split.subsplit(10) # Equivalent to split.subsplit(k=10) | split.subsplit(datasets.percent[:-20]) # percent=datasets.percent[:-20] | split.subsplit([1, 1, 2]) # weighted=[1, 1, 2] | ``` | k: `int` If set, subdivide the split into `k` equal parts. | percent: `datasets.percent slice`, return a single subsplit corresponding to | a slice of the original split. For example: | `split.subsplit(datasets.percent[-20:]) # Last 20% of the dataset`. | weighted: `list[int]`, return a list of subsplits whose proportions match | the normalized sum of the list. For example: | `split.subsplit(weighted=[1, 1, 2]) # 25%, 25%, 50%`. | | Returns: | A subsplit or list of subsplits extracted from this split object. | | ---------------------------------------------------------------------- | Data descriptors inherited from SplitBase: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
datasets.Nifti
Python Library Documentation: class Nifti in module datasets.features.nifti
class Nifti(builtins.object)
| Nifti(decode: bool = True, id: Optional[str] = None) -> None
|
| **Experimental.**
| Nifti [`Feature`] to read NIfTI neuroimaging files.
|
| Input: The Nifti feature accepts as input:
| - A `str`: Absolute path to the NIfTI file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the NIfTI file (i.e. random access is allowed).
| - A `dict` with the keys:
| - `path`: String with relative path of the NIfTI file in a dataset repository.
| - `bytes`: Bytes of the NIfTI file.
| This is useful for archived files with sequential access.
|
| - A `nibabel` image object (e.g., `nibabel.nifti1.Nifti1Image`).
|
| Args:
| decode (`bool`, defaults to `True`):
| Whether to decode the NIfTI data. If `False` a string with the bytes is returned. `decode=False` is not supported when decoding examples.
|
| Examples:
|
| ```py
| >>> from datasets import Dataset, Nifti
| >>> ds = Dataset.from_dict({"nifti": ["path/to/file.nii.gz"]}).cast_column("nifti", Nifti())
| >>> ds.features["nifti"]
| Nifti(decode=True, id=None)
| >>> ds[0]["nifti"]
| <nibabel.nifti1.Nifti1Image object at 0x7f8a1c2d8f40>
| >>> ds = ds.cast_column("nifti", Nifti(decode=False))
| >>> ds[0]["nifti"]
| {'bytes': None,
| 'path': 'path/to/file.nii.gz'}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray, pyarrow.lib.BinaryArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Nifti arrow storage type.
| The Arrow types that can be converted to the Nifti pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.binary()` - it must contain the NIfTI bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray, pa.BinaryArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Nifti arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: dict, token_per_repo_id=None) -> 'Nifti1ImageWrapper'
| Decode example NIfTI file into nibabel image object.
|
| Args:
| value (`str` or `dict`):
| A string with the absolute NIfTI file path, a dictionary with
| keys:
|
| - `path`: String with absolute or relative NIfTI file path.
| - `bytes`: The bytes of the NIfTI file.
|
| token_per_repo_id (`dict`, *optional*):
| To access and decode NIfTI files from private repositories on
| the Hub, you can pass a dictionary
| repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `nibabel.Nifti1Image` objects
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed NifTI files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the NifTI arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, ForwardRef('nib.Nifti1Image')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `bytes`, `nibabel.Nifti1Image` or `dict`):
| Data passed as input to Nifti feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), Dict[str, ForwardRef('FeatureType')]]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'id')
|
| decode = True
|
| dtype = 'nibabel.nifti1.Nifti1Image'
|
| id = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
datasets.Pdf
Python Library Documentation: class Pdf in module datasets.features.pdf
class Pdf(builtins.object)
| Pdf(decode: bool = True, id: Optional[str] = None) -> None
|
| **Experimental.**
| Pdf [`Feature`] to read pdf documents from a pdf file.
|
| Input: The Pdf feature accepts as input:
| - A `str`: Absolute path to the pdf file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the pdf file (i.e. random access is allowed).
| - A `dict` with the keys:
| - `path`: String with relative path of the pdf file in a dataset repository.
| - `bytes`: Bytes of the pdf file.
| This is useful for archived files with sequential access.
|
| - A `pdfplumber.pdf.PDF`: pdfplumber pdf object.
|
| Args:
| decode (`bool`, defaults to `True`):
| Whether to decode the pdf data. If `False`,
| returns the underlying dictionary in the format `{"path": pdf_path, "bytes": pdf_bytes}`.
|
| Examples:
|
| ```py
| >>> from datasets import Dataset, Pdf
| >>> ds = Dataset.from_dict({"pdf": ["path/to/pdf/file.pdf"]}).cast_column("pdf", Pdf())
| >>> ds.features["pdf"]
| Pdf(decode=True, id=None)
| >>> ds[0]["pdf"]
| <pdfplumber.pdf.PDF object at 0x7f8a1c2d8f40>
| >>> ds = ds.cast_column("pdf", Pdf(decode=False))
| >>> ds[0]["pdf"]
| {'bytes': None,
| 'path': 'path/to/pdf/file.pdf'}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray, pyarrow.lib.ListArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Pdf arrow storage type.
| The Arrow types that can be converted to the Pdf pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.binary()` - it must contain the image bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
| - `pa.list(*)` - it must contain the pdf array data
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Pdf arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: dict, token_per_repo_id=None) -> 'pdfplumber.pdf.PDF'
| Decode example pdf file into pdf data.
|
| Args:
| value (`str` or `dict`):
| A string with the absolute pdf file path, a dictionary with
| keys:
|
| - `path`: String with absolute or relative pdf file path.
| - `bytes`: The bytes of the pdf file.
|
| token_per_repo_id (`dict`, *optional*):
| To access and decode pdf files from private repositories on
| the Hub, you can pass a dictionary
| repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `pdfplumber.pdf.PDF`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed PDF files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the PDF arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, ForwardRef('pdfplumber.pdf.PDF')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `bytes`, `pdfplumber.pdf.PDF` or `dict`):
| Data passed as input to Pdf feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), Dict[str, ForwardRef('FeatureType')]]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'id')
|
| decode = True
|
| dtype = 'pdfplumber.pdf.PDF'
|
| id = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
datasets.ReadInstruction
Python Library Documentation: class ReadInstruction in module datasets.arrow_reader
class ReadInstruction(builtins.object)
| ReadInstruction(split_name, rounding=None, from_=None, to=None, unit=None)
|
| Reading instruction for a dataset.
|
| Examples::
|
| # The following lines are equivalent:
| ds = datasets.load_dataset('ylecun/mnist', split='test[:33%]')
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction.from_spec('test[:33%]'))
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction('test', to=33, unit='%'))
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction(
| 'test', from_=0, to=33, unit='%'))
|
| # The following lines are equivalent:
| ds = datasets.load_dataset('ylecun/mnist', split='test[:33%]+train[1:-1]')
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction.from_spec(
| 'test[:33%]+train[1:-1]'))
| ds = datasets.load_dataset('ylecun/mnist', split=(
| datasets.ReadInstruction('test', to=33, unit='%') +
| datasets.ReadInstruction('train', from_=1, to=-1, unit='abs')))
|
| # The following lines are equivalent:
| ds = datasets.load_dataset('ylecun/mnist', split='test[:33%](pct1_dropremainder)')
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction.from_spec(
| 'test[:33%](pct1_dropremainder)'))
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction(
| 'test', from_=0, to=33, unit='%', rounding="pct1_dropremainder"))
|
| # 10-fold validation:
| tests = datasets.load_dataset(
| 'ylecun/mnist',
| [datasets.ReadInstruction('train', from_=k, to=k+10, unit='%')
| for k in range(0, 100, 10)])
| trains = datasets.load_dataset(
| 'ylecun/mnist',
| [datasets.ReadInstruction('train', to=k, unit='%') + datasets.ReadInstruction('train', from_=k+10, unit='%')
| for k in range(0, 100, 10)])
|
| Methods defined here:
|
| __add__(self, other)
| Returns a new ReadInstruction obj, result of appending other to self.
|
| __init__(self, split_name, rounding=None, from_=None, to=None, unit=None)
| Initialize ReadInstruction.
|
| Args:
| split_name (str): name of the split to read. Eg: 'train'.
| rounding (str, optional): The rounding behaviour to use when percent slicing is
| used. Ignored when slicing with absolute indices.
| Possible values:
| - 'closest' (default): The specified percentages are rounded to the
| closest value. Use this if you want specified percents to be as
| much exact as possible.
| - 'pct1_dropremainder': the specified percentages are treated as
| multiple of 1%. Use this option if you want consistency. Eg:
| len(5%) == 5 * len(1%).
| Using this option, one might not be able to use the full set of
| examples, if the number of those is not a multiple of 100.
| from_ (int):
| to (int): alternative way of specifying slicing boundaries. If any of
| {from_, to, unit} argument is used, slicing cannot be specified as
| string.
| unit (str): optional, one of:
| '%': to set the slicing unit as percents of the split size.
| 'abs': to set the slicing unit as absolute numbers.
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| to_absolute(self, name2len)
| Translate instruction into a list of absolute instructions.
|
| Those absolute instructions are then to be added together.
|
| Args:
| name2len (`dict`):
| Associating split names to number of examples.
|
| Returns:
| list of _AbsoluteInstruction instances (corresponds to the + in spec).
|
| to_spec(self)
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_spec(spec)
| Creates a `ReadInstruction` instance out of a string spec.
|
| Args:
| spec (`str`):
| Split(s) + optional slice(s) to read + optional rounding
| if percents are used as the slicing unit. A slice can be specified,
| using absolute numbers (`int`) or percentages (`int`).
|
| Examples:
|
| ```
| test: test split.
| test + validation: test split + validation split.
| test[10:]: test split, minus its first 10 records.
| test[:10%]: first 10% records of test split.
| test[:20%](pct1_dropremainder): first 10% records, rounded with the pct1_dropremainder rounding.
| test[:-5%]+train[40%:60%]: first 95% of test + middle 20% of train.
| ```
|
| Returns:
| ReadInstruction instance.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
datasets.Sequence
Python Library Documentation: class Sequence in module datasets.features.features
class Sequence(builtins.object)
| Sequence(feature=None, length=-1, **kwargs)
|
| A `Sequence` is a utility that automatically converts internal dictionary feature into a dictionary of
| lists. This behavior is implemented to have a compatibility layer with the TensorFlow Datasets library but may be
| un-wanted in some cases. If you don't want this behavior, you can use a [`List`] or a [`LargeList`]
| instead of the [`Sequence`].
|
| Args:
| feature ([`FeatureType`]):
| Child feature data type of each item within the large list.
| length (optional `int`, default to -1):
| Length of the list if it is fixed.
| Defaults to -1 which means an arbitrary length.
|
| Returns:
| [`List`] of the specified feature, except `dict` of sub-features
| which are converted to `dict` of lists of sub-features for compatibility with TFDS.
|
| Static methods defined here:
|
| __new__(cls, feature=None, length=-1, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
datasets.Split
Python Library Documentation: class Split in module datasets.splits
class Split(builtins.object)
| Split(name)
|
| `Enum` for dataset splits.
|
| Datasets are typically split into different subsets to be used at various
| stages of training and evaluation.
|
| - `TRAIN`: the training data.
| - `VALIDATION`: the validation data. If present, this is typically used as
| evaluation data while iterating on a model (e.g. changing hyperparameters,
| model architecture, etc.).
| - `TEST`: the testing data. This is the data to report metrics on. Typically
| you do not want to use this during model iteration as you may overfit to it.
| - `ALL`: the union of all defined dataset splits.
|
| All splits, including compositions inherit from `datasets.SplitBase`.
|
| See the [guide](../load_hub#splits) on splits for more information.
|
| Example:
|
| ```py
| >>> datasets.SplitGenerator(
| ... name=datasets.Split.TRAIN,
| ... gen_kwargs={"split_key": "train", "files": dl_manager.download_and extract(url)},
| ... ),
| ... datasets.SplitGenerator(
| ... name=datasets.Split.VALIDATION,
| ... gen_kwargs={"split_key": "validation", "files": dl_manager.download_and extract(url)},
| ... ),
| ... datasets.SplitGenerator(
| ... name=datasets.Split.TEST,
| ... gen_kwargs={"split_key": "test", "files": dl_manager.download_and extract(url)},
| ... )
| ```
|
| Static methods defined here:
|
| __new__(cls, name)
| Create a custom split with datasets.Split('custom_name').
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| ALL = NamedSplitAll()
|
| TEST = NamedSplit('test')
|
| TRAIN = NamedSplit('train')
|
| VALIDATION = NamedSplit('validation')
datasets.SplitBase
Python Library Documentation: class SplitBase in module datasets.splits
class SplitBase(builtins.object)
| Abstract base class for Split compositionality.
|
| See the
| [guide on splits](../loading#slice-splits)
| for more information.
|
| There are three parts to the composition:
| 1) The splits are composed (defined, merged, split,...) together before
| calling the `.as_dataset()` function. This is done with the `__add__`,
| `__getitem__`, which return a tree of `SplitBase` (whose leaf
| are the `NamedSplit` objects)
|
| ```
| split = datasets.Split.TRAIN + datasets.Split.TEST.subsplit(datasets.percent[:50])
| ```
|
| 2) The `SplitBase` is forwarded to the `.as_dataset()` function
| to be resolved into actual read instruction. This is done by the
| `.get_read_instruction()` method which takes the real dataset splits
| (name, number of shards,...) and parse the tree to return a
| `SplitReadInstruction()` object
|
| ```
| read_instruction = split.get_read_instruction(self.info.splits)
| ```
|
| 3) The `SplitReadInstruction` is then used in the `tf.data.Dataset` pipeline
| to define which files to read and how to skip examples within file.
|
| Methods defined here:
|
| __add__(self, other)
| Merging: datasets.Split.TRAIN + datasets.Split.TEST.
|
| __eq__(self, other)
| Equality: datasets.Split.TRAIN == 'train'.
|
| __ne__(self, other)
| InEquality: datasets.Split.TRAIN != 'test'.
|
| get_read_instruction(self, split_dict)
| Parse the descriptor tree and compile all read instructions together.
|
| Args:
| split_dict: `dict`, The `dict[split_name, SplitInfo]` of the dataset
|
| Returns:
| split_read_instruction: `SplitReadInstruction`
|
| subsplit(self, arg=None, k=None, percent=None, weighted=None)
| Divides this split into subsplits.
|
| There are 3 ways to define subsplits, which correspond to the 3
| arguments `k` (get `k` even subsplits), `percent` (get a slice of the
| dataset with `datasets.percent`), and `weighted` (get subsplits with proportions
| specified by `weighted`).
|
| Example::
|
| ```
| # 50% train, 50% test
| train, test = split.subsplit(k=2)
| # 50% train, 25% test, 25% validation
| train, test, validation = split.subsplit(weighted=[2, 1, 1])
| # Extract last 20%
| subsplit = split.subsplit(datasets.percent[-20:])
| ```
|
| Warning: k and weighted will be converted into percent which mean that
| values below the percent will be rounded up or down. The final split may be
| bigger to deal with remainders. For instance:
|
| ```
| train, test, valid = split.subsplit(k=3) # 33%, 33%, 34%
| s1, s2, s3, s4 = split.subsplit(weighted=[2, 2, 1, 1]) # 33%, 33%, 16%, 18%
| ```
|
| Args:
| arg: If no kwargs are given, `arg` will be interpreted as one of
| `k`, `percent`, or `weighted` depending on the type.
| For example:
| ```
| split.subsplit(10) # Equivalent to split.subsplit(k=10)
| split.subsplit(datasets.percent[:-20]) # percent=datasets.percent[:-20]
| split.subsplit([1, 1, 2]) # weighted=[1, 1, 2]
| ```
| k: `int` If set, subdivide the split into `k` equal parts.
| percent: `datasets.percent slice`, return a single subsplit corresponding to
| a slice of the original split. For example:
| `split.subsplit(datasets.percent[-20:]) # Last 20% of the dataset`.
| weighted: `list[int]`, return a list of subsplits whose proportions match
| the normalized sum of the list. For example:
| `split.subsplit(weighted=[1, 1, 2]) # 25%, 25%, 50%`.
|
| Returns:
| A subsplit or list of subsplits extracted from this split object.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'get_read_instruction'})
|
| __hash__ = None
datasets.SplitDict
Python Library Documentation: class SplitDict in module datasets.splits class SplitDict(builtins.dict) | SplitDict(*args, dataset_name=None, **kwargs) | | Split info object. | | Method resolution order: | SplitDict | builtins.dict | builtins.object | | Methods defined here: | | __getitem__(self, key: Union[datasets.splits.SplitBase, str]) | Return self[key]. | | __init__(self, *args, dataset_name=None, **kwargs) | Initialize self. See help(type(self)) for accurate signature. | | __setitem__(self, key: Union[datasets.splits.SplitBase, str], value: datasets.splits.SplitInfo) | Set self[key] to value. | | add(self, split_info: datasets.splits.SplitInfo) | Add the split info. | | copy(self) | D.copy() -> a shallow copy of D | | to_split_dict(self) | Returns a list of SplitInfo protos that we have. | | ---------------------------------------------------------------------- | Class methods defined here: | | from_split_dict(split_infos: Union[list, dict], dataset_name: Optional[str] = None) | Returns a new SplitDict initialized from a Dict or List of `split_infos`. | | ---------------------------------------------------------------------- | Readonly properties defined here: | | total_num_examples | Return the total number of examples. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __orig_bases__ = (dict[str, datasets.splits.SplitInfo],) | | ---------------------------------------------------------------------- | Methods inherited from builtins.dict: | | __contains__(self, key, /) | True if the dictionary has the specified key, else False. | | __delitem__(self, key, /) | Delete self[key]. | | __eq__(self, value, /) | Return self==value. | | __ge__(self, value, /) | Return self>=value. | | __getattribute__(self, name, /) | Return getattr(self, name). | | __gt__(self, value, /) | Return self>value. | | __ior__(self, value, /) | Return self|=value. | | __iter__(self, /) | Implement iter(self). | | __le__(self, value, /) | Return self<=value. | | __len__(self, /) | Return len(self). | | __lt__(self, value, /) | Return self<value. | | __ne__(self, value, /) | Return self!=value. | | __or__(self, value, /) | Return self|value. | | __repr__(self, /) | Return repr(self). | | __reversed__(self, /) | Return a reverse iterator over the dict keys. | | __ror__(self, value, /) | Return value|self. | | __sizeof__(...) | D.__sizeof__() -> size of D in memory, in bytes | | clear(...) | D.clear() -> None. Remove all items from D. | | get(self, key, default=None, /) | Return the value for key if key is in the dictionary, else default. | | items(...) | D.items() -> a set-like object providing a view on D's items | | keys(...) | D.keys() -> a set-like object providing a view on D's keys | | pop(...) | D.pop(k[,d]) -> v, remove specified key and return the corresponding value. | | If the key is not found, return the default if given; otherwise, | raise a KeyError. | | popitem(self, /) | Remove and return a (key, value) pair as a 2-tuple. | | Pairs are returned in LIFO (last-in, first-out) order. | Raises KeyError if the dict is empty. | | setdefault(self, key, default=None, /) | Insert key with a value of default if key is not in the dictionary. | | Return the value for key if key is in the dictionary, else default. | | update(...) | D.update([E, ]**F) -> None. Update D from dict/iterable E and F. | If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] | If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v | In either case, this is followed by: for k in F: D[k] = F[k] | | values(...) | D.values() -> an object providing a view on D's values | | ---------------------------------------------------------------------- | Class methods inherited from builtins.dict: | | __class_getitem__(...) | See PEP 585 | | fromkeys(iterable, value=None, /) | Create a new dictionary with keys from iterable and values set to value. | | ---------------------------------------------------------------------- | Static methods inherited from builtins.dict: | | __new__(*args, **kwargs) class method of builtins.dict | Create and return a new object. See help(type) for accurate signature. | | ---------------------------------------------------------------------- | Data and other attributes inherited from builtins.dict: | | __hash__ = None
datasets.SplitGenerator
Python Library Documentation: class SplitGenerator in module datasets.splits
class SplitGenerator(builtins.object)
| SplitGenerator(name: str, gen_kwargs: dict = <factory>) -> None
|
| Defines the split information for the generator.
|
| This should be used as returned value of
| `GeneratorBasedBuilder._split_generators`.
| See `GeneratorBasedBuilder._split_generators` for more info and example
| of usage.
|
| Args:
| name (`str`):
| Name of the `Split` for which the generator will
| create the examples.
| **gen_kwargs (additional keyword arguments):
| Keyword arguments to forward to the `DatasetBuilder._generate_examples` method
| of the builder.
|
| Example:
|
| ```py
| >>> datasets.SplitGenerator(
| ... name=datasets.Split.TRAIN,
| ... gen_kwargs={"split_key": "train", "files": dl_manager.download_and_extract(url)},
| ... )
| ```
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, name: str, gen_kwargs: dict = <factory>) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'gen_kwargs': <class 'dict'>, 'name': <class 'str'>...
|
| __dataclass_fields__ = {'gen_kwargs': Field(name='gen_kwargs',type=<cl...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('name', 'gen_kwargs')
datasets.SplitInfo
Python Library Documentation: class SplitInfo in module datasets.splits
class SplitInfo(builtins.object)
| SplitInfo(name: str = '', num_bytes: int = 0, num_examples: int = 0, shard_lengths: Optional[list[int]] = None, original_shard_lengths: Optional[list[int]] = None, dataset_name: Optional[str] = None) -> None
|
| SplitInfo(name: str = '', num_bytes: int = 0, num_examples: int = 0, shard_lengths: Optional[list[int]] = None, original_shard_lengths: Optional[list[int]] = None, dataset_name: Optional[str] = None)
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, name: str = '', num_bytes: int = 0, num_examples: int = 0, shard_lengths: Optional[list[int]] = None, original_shard_lengths: Optional[list[int]] = None, dataset_name: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| file_instructions
| Returns the list of dict(filename, take, skip).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'dataset_name': typing.Optional[str], 'name': <clas...
|
| __dataclass_fields__ = {'dataset_name': Field(name='dataset_name',type...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('name', 'num_bytes', 'num_examples', 'shard_lengths'...
|
| dataset_name = None
|
| name = ''
|
| num_bytes = 0
|
| num_examples = 0
|
| original_shard_lengths = None
|
| shard_lengths = None
datasets.StreamingDownloadManager
Python Library Documentation: class StreamingDownloadManager in module datasets.download.streaming_download_manager
class StreamingDownloadManager(builtins.object)
| StreamingDownloadManager(dataset_name: Optional[str] = None, data_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, base_path: Optional[str] = None)
|
| Download manager that uses the "::" separator to navigate through (possibly remote) compressed archives.
| Contrary to the regular `DownloadManager`, the `download` and `extract` methods don't actually download nor extract
| data, but they rather return the path or url that could be opened using the `xopen` function which extends the
| built-in `open` function to stream data from remote files.
|
| Methods defined here:
|
| __init__(self, dataset_name: Optional[str] = None, data_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, base_path: Optional[str] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| download(self, url_or_urls)
| Normalize URL(s) of files to stream data from.
| This is the lazy version of `DownloadManager.download` for streaming.
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL(s) of files to stream data from. Each url is a `str`.
|
| Returns:
| url(s): (`str` or `list` or `dict`), URL(s) to stream data from matching the given input url_or_urls.
|
| Example:
|
| ```py
| >>> downloaded_files = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| ```
|
| download_and_extract(self, url_or_urls)
| Prepare given `url_or_urls` for streaming (add extraction protocol).
|
| This is the lazy version of `DownloadManager.download_and_extract` for streaming.
|
| Is equivalent to:
|
| ```
| urls = dl_manager.extract(dl_manager.download(url_or_urls))
| ```
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL(s) to stream from data from. Each url is a `str`.
|
| Returns:
| url(s): (`str` or `list` or `dict`), URL(s) to stream data from matching the given input `url_or_urls`.
|
| extract(self, url_or_urls)
| Add extraction protocol for given url(s) for streaming.
|
| This is the lazy version of `DownloadManager.extract` for streaming.
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL(s) of files to stream data from. Each url is a `str`.
|
| Returns:
| url(s): (`str` or `list` or `dict`), URL(s) to stream data from matching the given input `url_or_urls`.
|
| Example:
|
| ```py
| >>> downloaded_files = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| >>> extracted_files = dl_manager.extract(downloaded_files)
| ```
|
| get_recorded_sizes_checksums(self)
|
| iter_archive(self, urlpath_or_buf: Union[str, _io.BufferedReader]) -> collections.abc.Iterable[tuple]
| Iterate over files within an archive.
|
| Args:
| urlpath_or_buf (`str` or `io.BufferedReader`):
| Archive path or archive binary file object.
|
| Yields:
| `tuple[str, io.BufferedReader]`:
| 2-tuple (path_within_archive, file_object).
| File object is opened in binary mode.
|
| Example:
|
| ```py
| >>> archive = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| >>> files = dl_manager.iter_archive(archive)
| ```
|
| iter_files(self, urlpaths: Union[str, list[str]]) -> collections.abc.Iterable[str]
| Iterate over files.
|
| Args:
| urlpaths (`str` or `list` of `str`):
| Root paths.
|
| Yields:
| str: File URL path.
|
| Example:
|
| ```py
| >>> files = dl_manager.download_and_extract('https://huggingface.co/datasets/AI-Lab-Makerere/beans/resolve/main/data/train.zip')
| >>> files = dl_manager.iter_files(files)
| ```
|
| manage_extracted_files(self)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| manual_dir
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| is_streaming = True
datasets.SubSplitInfo
Python Library Documentation: class SubSplitInfo in module datasets.splits
class SubSplitInfo(builtins.object)
| SubSplitInfo(instructions: datasets.arrow_reader.FileInstructions) -> None
|
| Wrapper around a sub split info.
| This class exposes info on the subsplit:
| ```
| ds, info = datasets.load_dataset(..., split='train[75%:]', with_info=True)
| info.splits['train[75%:]'].num_examples
| ```
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, instructions: datasets.arrow_reader.FileInstructions) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| file_instructions
| Returns the list of dict(filename, take, skip).
|
| num_examples
| Returns the number of examples in the subsplit.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'instructions': <class 'datasets.arrow_reader.FileI...
|
| __dataclass_fields__ = {'instructions': Field(name='instructions',type...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('instructions',)
datasets.Translation
Python Library Documentation: class Translation in module datasets.features.translation
class Translation(builtins.object)
| Translation(languages: list[str], id: Optional[str] = None) -> None
|
| `Feature` for translations with fixed languages per example.
| Here for compatibility with tfds.
|
| Args:
| languages (`dict`):
| A dictionary for each example mapping string language codes to string translations.
|
| Example:
|
| ```python
| >>> # At construction time:
| >>> datasets.features.Translation(languages=['en', 'fr', 'de'])
| >>> # During data generation:
| >>> yield {
| ... 'en': 'the cat',
| ... 'fr': 'le chat',
| ... 'de': 'die katze'
| ... }
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, languages: list[str], id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| Flatten the Translation feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': typing.ClassVar[st...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('languages', 'id')
|
| dtype = 'dict'
|
| id = None
|
| pa_type = None
datasets.TranslationVariableLanguages
Python Library Documentation: class TranslationVariableLanguages in module datasets.features.translation
class TranslationVariableLanguages(builtins.object)
| TranslationVariableLanguages(languages: Optional[list] = None, num_languages: Optional[int] = None, id: Optional[str] = None) -> None
|
| `Feature` for translations with variable languages per example.
| Here for compatibility with tfds.
|
| Args:
| languages (`dict`):
| A dictionary for each example mapping string language codes to one or more string translations.
| The languages present may vary from example to example.
|
| Returns:
| - `language` or `translation` (variable-length 1D `tf.Tensor` of `tf.string`):
| Language codes sorted in ascending order or plain text translations, sorted to align with language codes.
|
| Example:
|
| ```python
| >>> # At construction time:
| >>> datasets.features.TranslationVariableLanguages(languages=['en', 'fr', 'de'])
| >>> # During data generation:
| >>> yield {
| ... 'en': 'the cat',
| ... 'fr': ['le chat', 'la chatte,']
| ... 'de': 'die katze'
| ... }
| >>> # Tensor returned :
| >>> {
| ... 'language': ['en', 'de', 'fr', 'fr'],
| ... 'translation': ['the cat', 'die katze', 'la chatte', 'le chat'],
| ... }
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, languages: Optional[list] = None, num_languages: Optional[int] = None, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| encode_example(self, translation_dict)
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| Flatten the TranslationVariableLanguages feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': typing.ClassVar[st...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('languages', 'num_languages', 'id')
|
| dtype = 'dict'
|
| id = None
|
| languages = None
|
| num_languages = None
|
| pa_type = None
datasets.Value
Python Library Documentation: class Value in module datasets.features.features
class Value(builtins.object)
| Value(dtype: str, id: Optional[str] = None) -> None
|
| Scalar feature value of a particular data type.
|
| The possible dtypes of `Value` are as follows:
| - `null`
| - `bool`
| - `int8`
| - `int16`
| - `int32`
| - `int64`
| - `uint8`
| - `uint16`
| - `uint32`
| - `uint64`
| - `float16`
| - `float32` (alias float)
| - `float64` (alias double)
| - `time32[(s|ms)]`
| - `time64[(us|ns)]`
| - `timestamp[(s|ms|us|ns)]`
| - `timestamp[(s|ms|us|ns), tz=(tzstring)]`
| - `date32`
| - `date64`
| - `duration[(s|ms|us|ns)]`
| - `decimal128(precision, scale)`
| - `decimal256(precision, scale)`
| - `binary`
| - `large_binary`
| - `binary_view`
| - `string`
| - `large_string`
| - `string_view`
|
| Args:
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'stars': Value('int32')})
| >>> features
| {'stars': Value('int32')}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=False,eq=True,o...
|
| __hash__ = None
|
| __match_args__ = ('dtype', 'id')
|
| id = None
|
| pa_type = None
datasets.VerificationMode
Python Library Documentation: class VerificationMode in module datasets.utils.info_utils class VerificationMode(enum.Enum) | VerificationMode(*values) | | `Enum` that specifies which verification checks to run. | | The default mode is `BASIC_CHECKS`, which will perform only rudimentary checks to avoid slowdowns | when generating/downloading a dataset for the first time. | | The verification modes: | | | | Verification checks | | |---------------------------|------------------------------------------------------------------------------ | | | `ALL_CHECKS` | Split checks and validity (number of files, checksums) of downloaded files | | | `BASIC_CHECKS` (default) | Same as `ALL_CHECKS` but without checking downloaded files | | | `NO_CHECKS` | None | | | Method resolution order: | VerificationMode | enum.Enum | builtins.object | | Data and other attributes defined here: | | ALL_CHECKS = <VerificationMode.ALL_CHECKS: 'all_checks'> | | BASIC_CHECKS = <VerificationMode.BASIC_CHECKS: 'basic_checks'> | | NO_CHECKS = <VerificationMode.NO_CHECKS: 'no_checks'> | | ---------------------------------------------------------------------- | Data descriptors inherited from enum.Enum: | | name | The name of the Enum member. | | value | The value of the Enum member. | | ---------------------------------------------------------------------- | Static methods inherited from enum.EnumType: | | __contains__(value) | Return True if `value` is in `cls`. | | `value` is in `cls` if: | 1) `value` is a member of `cls`, or | 2) `value` is the value of one of the `cls`'s members. | | __getitem__(name) | Return the member matching `name`. | | __iter__() | Return members in definition order. | | __len__() | Return the number of members (no aliases) | | ---------------------------------------------------------------------- | Readonly properties inherited from enum.EnumType: | | __members__ | Returns a mapping of member name->value. | | This mapping lists all enum members, including aliases. Note that this | is a read-only view of the internal mapping.
datasets.Version
Python Library Documentation: class Version in module datasets.utils.version
class Version(builtins.object)
| Version(version_str: str, description: Optional[str] = None, major: Union[str, int, NoneType] = None, minor: Union[str, int, NoneType] = None, patch: Union[str, int, NoneType] = None) -> None
|
| Dataset version `MAJOR.MINOR.PATCH`.
|
| Args:
| version_str (`str`):
| The dataset version.
| description (`str`):
| A description of what is new in this version.
| major (`str`):
| minor (`str`):
| patch (`str`):
|
| Example:
|
| ```py
| >>> VERSION = datasets.Version("1.0.0")
| ```
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __ge__(self, other) from functools
| Return a >= b. Computed by @total_ordering from (not a < b).
|
| __gt__(self, other) from functools
| Return a > b. Computed by @total_ordering from (not a < b) and (a != b).
|
| __hash__(self)
| Return hash(self).
|
| __init__(self, version_str: str, description: Optional[str] = None, major: Union[str, int, NoneType] = None, minor: Union[str, int, NoneType] = None, patch: Union[str, int, NoneType] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __le__(self, other) from functools
| Return a <= b. Computed by @total_ordering from (a < b) or (a == b).
|
| __lt__(self, other)
| Return self<value.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_dict(dic)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| tuple
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'description': typing.Optional[str], 'major': typin...
|
| __dataclass_fields__ = {'description': Field(name='description',type=t...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __match_args__ = ('version_str', 'description', 'major', 'minor', 'pat...
|
| description = None
|
| major = None
|
| minor = None
|
| patch = None
datasets.Video
Python Library Documentation: class Video in module datasets.features.video
class Video(builtins.object)
| Video(decode: bool = True, stream_index: Optional[int] = None, dimension_order: Literal['NCHW', 'NHWC'] = 'NCHW', num_ffmpeg_threads: int = 1, device: Union[str, ForwardRef('torch.device'), NoneType] = 'cpu', seek_mode: Literal['exact', 'approximate'] = 'exact', id: Optional[str] = None) -> None
|
| Video [`Feature`] to read video data from a video file.
|
| Input: The Video feature accepts as input:
| - A `str`: Absolute path to the video file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the video file (i.e. random access is allowed).
| - A `dict` with the keys:
|
| - `path`: String with relative path of the video file in a dataset repository.
| - `bytes`: Bytes of the video file.
|
| This is useful for parquet or webdataset files which embed video files.
|
| - A `torchcodec.decoders.VideoDecoder`: torchcodec video decoder object.
|
| Output: The Video features output data as `torchcodec.decoders.VideoDecoder` objects.
|
| Args:
| decode (`bool`, defaults to `True`):
| Whether to decode the video data. If `False`,
| returns the underlying dictionary in the format `{"path": video_path, "bytes": video_bytes}`.
| stream_index (`int`, *optional*):
| The streaming index to use from the file. If `None` defaults to the "best" index.
| dimension_order (`str`, defaults to `NCHW`):
| The dimension order of the decoded frames.
| where N is the batch size, C is the number of channels,
| H is the height, and W is the width of the frames.
| num_ffmpeg_threads (`int`, defaults to `1`):
| The number of threads to use for decoding the video. (Recommended to keep this at 1)
| device (`str` or `torch.device`, defaults to `cpu`):
| The device to use for decoding the video.
| seek_mode (`str`, defaults to `exact`):
| Determines if frame access will be “exact” or “approximate”.
| Exact guarantees that requesting frame i will always return frame i, but doing so requires an initial scan of the file.
| Approximate is faster as it avoids scanning the file, but less accurate as it uses the file's metadata to calculate where i probably is.
| read more [here](https://docs.pytorch.org/torchcodec/stable/generated_examples/approximate_mode.html#sphx-glr-generated-examples-approximate-mode-py)
|
| Examples:
|
| ```py
| >>> from datasets import Dataset, Video
| >>> ds = Dataset.from_dict({"video":["path/to/Screen Recording.mov"]}).cast_column("video", Video())
| >>> ds.features["video"]
| Video(decode=True, id=None)
| >>> ds[0]["video"]
| <torchcodec.decoders._video_decoder.VideoDecoder object at 0x14a61e080>
| >>> video = ds[0]["video"]
| >>> video.get_frames_in_range(0, 10)
| FrameBatch:
| data (shape): torch.Size([10, 3, 50, 66])
| pts_seconds: tensor([0.4333, 0.4333, 0.4333, 0.4333, 0.4333, 0.4333, 0.4333, 0.4333, 0.4333,
| 0.4333], dtype=torch.float64)
| duration_seconds: tensor([0.0167, 0.0167, 0.0167, 0.0167, 0.0167, 0.0167, 0.0167, 0.0167, 0.0167,
| 0.0167], dtype=torch.float64)
| >>> ds.cast_column('video', Video(decode=False))[0]["video]
| {'bytes': None,
| 'path': 'path/to/Screen Recording.mov'}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, stream_index: Optional[int] = None, dimension_order: Literal['NCHW', 'NHWC'] = 'NCHW', num_ffmpeg_threads: int = 1, device: Union[str, ForwardRef('torch.device'), NoneType] = 'cpu', seek_mode: Literal['exact', 'approximate'] = 'exact', id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray, pyarrow.lib.ListArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Video arrow storage type.
| The Arrow types that can be converted to the Video pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.binary()` - it must contain the video bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
| - `pa.list(*)` - it must contain the video array data
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Video arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: Union[str, datasets.features.video.Example], token_per_repo_id: Optional[dict[str, Union[bool, str]]] = None) -> 'VideoDecoder'
| Decode example video file into video data.
|
| Args:
| value (`str` or `dict`):
| A string with the absolute video file path, a dictionary with
| keys:
|
| - `path`: String with absolute or relative video file path.
| - `bytes`: The bytes of the video file.
| token_per_repo_id (`dict`, *optional*):
| To access and decode
| video files from private repositories on the Hub, you can pass
| a dictionary repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `torchcodec.decoders.VideoDecoder`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed image files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the Video arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, datasets.features.video.Example, numpy.ndarray, ForwardRef('VideoDecoder')]) -> datasets.features.video.Example
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `np.ndarray`, `bytes`, `bytearray`, `VideoDecoder` or `dict`):
| Data passed as input to Video feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'stream_index', 'dimension_order', 'num_ff...
|
| decode = True
|
| device = 'cpu'
|
| dimension_order = 'NCHW'
|
| dtype = 'torchcodec.decoders.VideoDecoder'
|
| id = None
|
| num_ffmpeg_threads = 1
|
| pa_type = StructType(struct<bytes: binary, path: string>)
|
| seek_mode = 'exact'
|
| stream_index = None
datasets.are_progress_bars_disabled
Python Library Documentation: function are_progress_bars_disabled in module datasets.utils.tqdm
are_progress_bars_disabled() -> bool
Return whether progress bars are globally disabled or not.
Progress bars used in `datasets` can be enable or disabled globally using [`~utils.enable_progress_bars`]
and [`~utils.disable_progress_bars`] or by setting `HF_DATASETS_DISABLE_PROGRESS_BARS` as environment variable.
datasets.arrow_dataset
Python Library Documentation: module datasets.arrow_dataset in datasets
NAME
datasets.arrow_dataset - Simple Dataset wrapping an Arrow Table.
CLASSES
builtins.Exception(builtins.BaseException)
DatasetTransformationNotAllowedError
NonExistentDatasetError
builtins.object
DatasetInfoMixin
Dataset(DatasetInfoMixin, datasets.search.IndexableMixin, TensorflowDatasetMixin)
TensorflowDatasetMixin
collections.abc.Sequence(collections.abc.Reversible, collections.abc.Collection)
Column
class Column(collections.abc.Sequence)
| Column(source: Union[ForwardRef('Dataset'), ForwardRef('Column')], column_name: str)
|
| An iterable for a specific column of a [`Dataset`].
|
| Example:
|
| Iterate on the texts of the "text" column of a dataset:
|
| ```python
| for text in dataset["text"]:
| ...
| ```
|
| It also works with nested columns:
|
| ```python
| for source in dataset["metadata"]["source"]:
| ...
| ```
|
| Method resolution order:
| Column
| collections.abc.Sequence
| collections.abc.Reversible
| collections.abc.Collection
| collections.abc.Sized
| collections.abc.Iterable
| collections.abc.Container
| builtins.object
|
| Methods defined here:
|
| __eq__(self, value)
| Return self==value.
|
| __getitem__(self, key: Union[int, str, list[int]]) -> Any
|
| __init__(self, source: Union[ForwardRef('Dataset'), ForwardRef('Column')], column_name: str)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> collections.abc.Iterator[typing.Any]
|
| __len__(self) -> int
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __hash__ = None
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.Sequence:
|
| __contains__(self, value)
|
| __reversed__(self)
|
| count(self, value)
| S.count(value) -> integer -- return number of occurrences of value
|
| index(self, value, start=0, stop=None)
| S.index(value, [start, [stop]]) -> integer -- return first index of value.
| Raises ValueError if the value is not present.
|
| Supporting start and stop arguments is optional, but
| recommended.
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Reversible:
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
class Dataset(DatasetInfoMixin, datasets.search.IndexableMixin, TensorflowDatasetMixin)
| Dataset(arrow_table: datasets.table.Table, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, indices_table: Optional[datasets.table.Table] = None, fingerprint: Optional[str] = None)
|
| A Dataset backed by an Arrow table.
|
| Method resolution order:
| Dataset
| DatasetInfoMixin
| datasets.search.IndexableMixin
| TensorflowDatasetMixin
| builtins.object
|
| Methods defined here:
|
| __del__(self)
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| __getitem__(self, key)
| Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools).
|
| __getitems__(self, keys: list) -> list
| Can be used to get a batch using a list of integers indices.
|
| __init__(self, arrow_table: datasets.table.Table, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, indices_table: Optional[datasets.table.Table] = None, fingerprint: Optional[str] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| Iterate through the examples.
|
| If a formatting is set with [`Dataset.set_format`] rows will be returned with the
| selected format.
|
| __len__(self)
| Number of rows in the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.__len__
| <bound method Dataset.__len__ of Dataset({
| features: ['text', 'label'],
| num_rows: 1066
| })>
| ```
|
| __repr__(self)
| Return repr(self).
|
| __setstate__(self, state)
|
| add_column(self, name: str, column: Union[list, numpy.ndarray], new_fingerprint: Optional[str] = None, feature: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None)
| Add column to Dataset.
|
| <Added version="1.7"/>
|
| Args:
| name (`str`):
| Column name.
| column (`list` or `np.array`):
| Column data to be added.
| feature (`FeatureType` or `None`, defaults to `None`):
| Column datatype.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> more_text = ds["text"]
| >>> ds = ds.add_column(name="text_2", column=more_text)
| >>> ds
| Dataset({
| features: ['text', 'label', 'text_2'],
| num_rows: 1066
| })
| ```
|
| add_elasticsearch_index(self, column: str, index_name: Optional[str] = None, host: Optional[str] = None, port: Optional[int] = None, es_client: Optional[ForwardRef('elasticsearch.Elasticsearch')] = None, es_index_name: Optional[str] = None, es_index_config: Optional[dict] = None)
| Add a text index using ElasticSearch for fast retrieval. This is done in-place.
|
| Args:
| column (`str`):
| The column of the documents to add to the index.
| index_name (`str`, *optional*):
| The `index_name`/identifier of the index.
| This is the index name that is used to call [`~Dataset.get_nearest_examples`] or [`~Dataset.search`].
| By default it corresponds to `column`.
| host (`str`, *optional*, defaults to `localhost`):
| Host of where ElasticSearch is running.
| port (`str`, *optional*, defaults to `9200`):
| Port of where ElasticSearch is running.
| es_client (`elasticsearch.Elasticsearch`, *optional*):
| The elasticsearch client used to create the index if host and port are `None`.
| es_index_name (`str`, *optional*):
| The elasticsearch index name used to create the index.
| es_index_config (`dict`, *optional*):
| The configuration of the elasticsearch index.
| Default config is:
| ```
| {
| "settings": {
| "number_of_shards": 1,
| "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},
| },
| "mappings": {
| "properties": {
| "text": {
| "type": "text",
| "analyzer": "standard",
| "similarity": "BM25"
| },
| }
| },
| }
| ```
| Example:
|
| ```python
| >>> es_client = elasticsearch.Elasticsearch()
| >>> ds = datasets.load_dataset('community-datasets/crime_and_punish', split='train')
| >>> ds.add_elasticsearch_index(column='line', es_client=es_client, es_index_name="my_es_index")
| >>> scores, retrieved_examples = ds.get_nearest_examples('line', 'my new query', k=10)
| ```
|
| add_faiss_index(self, column: str, index_name: Optional[str] = None, device: Optional[int] = None, string_factory: Optional[str] = None, metric_type: Optional[int] = None, custom_index: Optional[ForwardRef('faiss.Index')] = None, batch_size: int = 1000, train_size: Optional[int] = None, faiss_verbose: bool = False, dtype=<class 'numpy.float32'>)
| Add a dense index using Faiss for fast retrieval.
| By default the index is done over the vectors of the specified column.
| You can specify `device` if you want to run it on GPU (`device` must be the GPU index).
| You can find more information about Faiss here:
|
| - For [string factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)
|
| Args:
| column (`str`):
| The column of the vectors to add to the index.
| index_name (`str`, *optional*):
| The `index_name`/identifier of the index.
| This is the `index_name` that is used to call [`~datasets.Dataset.get_nearest_examples`] or [`~datasets.Dataset.search`].
| By default it corresponds to `column`.
| device (`Union[int, List[int]]`, *optional*):
| If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs.
| If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU.
| string_factory (`str`, *optional*):
| This is passed to the index factory of Faiss to create the index.
| Default index class is `IndexFlat`.
| metric_type (`int`, *optional*):
| Type of metric. Ex: `faiss.METRIC_INNER_PRODUCT` or `faiss.METRIC_L2`.
| custom_index (`faiss.Index`, *optional*):
| Custom Faiss index that you already have instantiated and configured for your needs.
| batch_size (`int`):
| Size of the batch to use while adding vectors to the `FaissIndex`. Default value is `1000`.
| <Added version="2.4.0"/>
| train_size (`int`, *optional*):
| If the index needs a training step, specifies how many vectors will be used to train the index.
| faiss_verbose (`bool`, defaults to `False`):
| Enable the verbosity of the Faiss index.
| dtype (`data-type`):
| The dtype of the numpy arrays that are indexed.
| Default is `np.float32`.
|
| Example:
|
| ```python
| >>> ds = datasets.load_dataset('community-datasets/crime_and_punish', split='train')
| >>> ds_with_embeddings = ds.map(lambda example: {'embeddings': embed(example['line']}))
| >>> ds_with_embeddings.add_faiss_index(column='embeddings')
| >>> # query
| >>> scores, retrieved_examples = ds_with_embeddings.get_nearest_examples('embeddings', embed('my new query'), k=10)
| >>> # save index
| >>> ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')
|
| >>> ds = datasets.load_dataset('community-datasets/crime_and_punish', split='train')
| >>> # load index
| >>> ds.load_faiss_index('embeddings', 'my_index.faiss')
| >>> # query
| >>> scores, retrieved_examples = ds.get_nearest_examples('embeddings', embed('my new query'), k=10)
| ```
|
| add_faiss_index_from_external_arrays(self, external_arrays: <built-in function array>, index_name: str, device: Optional[int] = None, string_factory: Optional[str] = None, metric_type: Optional[int] = None, custom_index: Optional[ForwardRef('faiss.Index')] = None, batch_size: int = 1000, train_size: Optional[int] = None, faiss_verbose: bool = False, dtype=<class 'numpy.float32'>)
| Add a dense index using Faiss for fast retrieval.
| The index is created using the vectors of `external_arrays`.
| You can specify `device` if you want to run it on GPU (`device` must be the GPU index).
| You can find more information about Faiss here:
|
| - For [string factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)
|
| Args:
| external_arrays (`np.array`):
| If you want to use arrays from outside the lib for the index, you can set `external_arrays`.
| It will use `external_arrays` to create the Faiss index instead of the arrays in the given `column`.
| index_name (`str`):
| The `index_name`/identifier of the index.
| This is the `index_name` that is used to call [`~datasets.Dataset.get_nearest_examples`] or [`~datasets.Dataset.search`].
| device (Optional `Union[int, List[int]]`, *optional*):
| If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs.
| If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU.
| string_factory (`str`, *optional*):
| This is passed to the index factory of Faiss to create the index.
| Default index class is `IndexFlat`.
| metric_type (`int`, *optional*):
| Type of metric. Ex: `faiss.faiss.METRIC_INNER_PRODUCT` or `faiss.METRIC_L2`.
| custom_index (`faiss.Index`, *optional*):
| Custom Faiss index that you already have instantiated and configured for your needs.
| batch_size (`int`, *optional*):
| Size of the batch to use while adding vectors to the FaissIndex. Default value is 1000.
| <Added version="2.4.0"/>
| train_size (`int`, *optional*):
| If the index needs a training step, specifies how many vectors will be used to train the index.
| faiss_verbose (`bool`, defaults to False):
| Enable the verbosity of the Faiss index.
| dtype (`numpy.dtype`):
| The dtype of the numpy arrays that are indexed. Default is np.float32.
|
| add_item(self, item: dict, new_fingerprint: Optional[str] = None)
| Add item to Dataset.
|
| <Added version="1.7"/>
|
| Args:
| item (`dict`):
| Item data to be added.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> new_review = {'label': 0, 'text': 'this movie is the absolute worst thing I have ever seen'}
| >>> ds = ds.add_item(new_review)
| >>> ds[-1]
| {'label': 0, 'text': 'this movie is the absolute worst thing I have ever seen'}
| ```
|
| align_labels_with_mapping(self, label2id: dict, label_column: str) -> 'Dataset'
| Align the dataset's label ID and label name mapping to match an input `label2id` mapping.
| This is useful when you want to ensure that a model's predicted labels are aligned with the dataset.
| The alignment in done using the lowercase label names.
|
| Args:
| label2id (`dict`):
| The label name to ID mapping to align the dataset with.
| label_column (`str`):
| The column name of labels to align on.
|
| Example:
|
| ```python
| >>> # dataset with mapping {'entailment': 0, 'neutral': 1, 'contradiction': 2}
| >>> ds = load_dataset("nyu-mll/glue", "mnli", split="train")
| >>> # mapping to align with
| >>> label2id = {'CONTRADICTION': 0, 'NEUTRAL': 1, 'ENTAILMENT': 2}
| >>> ds_aligned = ds.align_labels_with_mapping(label2id, "label")
| ```
|
| batch(self, batch_size: Optional[int] = None, by_column: Union[str, list[str], NoneType] = None, drop_last_batch: bool = False, num_proc: Optional[int] = None, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Group samples from the dataset into batches.
|
| Specify either `batch_size` to make batches of `batch_size` examples.
|
| Or use `by_column=column_name` to make batches of successive examples that share
| the same value in column `column_name`.
|
| Args:
| batch_size (`int`, optional):
| The number of samples in each batch.
| by_column (`Union[str, list[str]`, optional):
| The column used to batch examples together.
| Successive examples with the same value for that column are in grouped the same batch.
| This can also be a list of columns if you want to batch by multiple columns.
| If batching by column, the batch_size is only used to control the size of the batches
| to group together or slice during acculumation.
|
| <Added version="4.9.0"/>
| drop_last_batch (`bool`, defaults to `False`):
| Whether to drop the last incomplete batch.
| num_proc (`int`, *optional*, defaults to `None`):
| Max number of processes when generating cache. Already cached shards are loaded sequentially.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A new Dataset where each item is a batch of multiple samples from the original dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> batched_ds = ds.batch(batch_size=4)
| >>> batched_ds[0]
| {'text': ['compassionately explores the seemingly irreconcilable situation...', ...], # 4 items
| 'label': [1, 1, 1, 1]}
| ```
|
| cast(self, features: datasets.features.features.Features, batch_size: Optional[int] = 1000, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, num_proc: Optional[int] = None) -> 'Dataset'
| Cast the dataset to a new set of features.
|
| Args:
| features ([`Features`]):
| New features to cast the dataset to.
| The name of the fields in the features must match the current column names.
| The type of the data must also be convertible from one type to the other.
| For non-trivial conversion, e.g. `str` <-> `ClassLabel` you should use [`~datasets.Dataset.map`] to update the Dataset.
| batch_size (`int`, defaults to `1000`):
| Number of examples per batch provided to cast.
| If `batch_size <= 0` or `batch_size == None` then provide the full dataset as a single batch to cast.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| load_from_cache_file (`bool`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_name (`str`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running [`~datasets.Dataset.map`].
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes for multiprocessing. By default it doesn't
| use multiprocessing.
|
| Returns:
| [`Dataset`]: A copy of the dataset with casted features.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel, Value
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> new_features = ds.features.copy()
| >>> new_features['label'] = ClassLabel(names=['bad', 'good'])
| >>> new_features['text'] = Value('large_string')
| >>> ds = ds.cast(new_features)
| >>> ds.features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('large_string')}
| ```
|
| cast_column(self, column: str, feature: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti], new_fingerprint: Optional[str] = None) -> 'Dataset'
| Cast column to feature for decoding.
|
| Args:
| column (`str`):
| Column name.
| feature (`FeatureType`):
| Target feature.
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> ds = ds.cast_column('label', ClassLabel(names=['bad', 'good']))
| >>> ds.features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('string')}
| ```
|
| class_encode_column(self, column: str, include_nulls: bool = False) -> 'Dataset'
| Casts the given column as [`~datasets.features.ClassLabel`] and updates the table.
|
| Args:
| column (`str`):
| The name of the column to cast (list all the column names with [`~datasets.Dataset.column_names`])
| include_nulls (`bool`, defaults to `False`):
| Whether to include null values in the class labels. If `True`, the null values will be encoded as the `"None"` class label.
|
| <Added version="1.14.2"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("google/boolq", split="validation")
| >>> ds.features
| {'answer': Value('bool'),
| 'passage': Value('string'),
| 'question': Value('string')}
| >>> ds = ds.class_encode_column('answer')
| >>> ds.features
| {'answer': ClassLabel(num_classes=2, names=['False', 'True']),
| 'passage': Value('string'),
| 'question': Value('string')}
| ```
|
| cleanup_cache_files(self) -> int
| Clean up all cache files in the dataset cache directory, excepted the currently used cache file if there is
| one.
|
| Be careful when running this command that no other process is currently using other cache files.
|
| Returns:
| `int`: Number of removed files.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.cleanup_cache_files()
| 10
| ```
|
| filter(self, function: Optional[Callable] = None, with_indices: bool = False, with_rank: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, suffix_template: str = '_{rank:05d}_of_{num_proc:05d}', new_fingerprint: Optional[str] = None, desc: Optional[str] = None) -> 'Dataset'
| Apply a filter function to all the elements in the table in batches
| and update the table so that the dataset only includes examples according to the filter function.
|
| If the function is asynchronous, then `filter` will run your function in parallel, with up to one thousand simultaneous calls (configurable).
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`): Callable with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> bool` if `batched=False` and `with_indices=False` and `with_rank=False`
| - `function(example: Dict[str, Any], *extra_args) -> bool` if `batched=False` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
| - `function(batch: Dict[str, List]) -> List[bool]` if `batched=True` and `with_indices=False` and `with_rank=False`
| - `function(batch: Dict[str, List], *extra_args) -> List[bool]` if `batched=True` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
|
| If the function is asynchronous, then `filter` will run your function in parallel.
| If no function is provided, defaults to an always `True` function: `lambda x: True`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the
| signature of `function` should be `def function(example, idx[, rank]): ...`.
| with_rank (`bool`, defaults to `False`):
| Provide process rank to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], rank): ...`.
| input_columns (`str` or `List[str]`, *optional*):
| The columns to be passed into `function` as
| positional arguments. If `None`, a `dict` mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if
| `batched = True`. If `batched = False`, one example per batch is passed to `function`.
| If `batch_size <= 0` or `batch_size == None`, provide the full dataset as a single batch to `function`.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| fn_kwargs (`dict`, *optional*):
| Keyword arguments to be passed to `function`.
| num_proc (`int`, *optional*, defaults to `None`):
| The number of processes to use for multiprocessing.
| - If `None` or `0`, no multiprocessing is used and the operation runs in the main process.
| - If greater than `1`, one or multiple worker processes are used to process data in parallel.
| Note: The function passed to `map()` must be picklable for multiprocessing to work correctly
| (i.e., prefer functions defined at the top level of a module, not inside another function or class).
| suffix_template (`str`):
| If `cache_file_name` is specified, then this suffix will be added at the end of the base name of each.
| For example, if `cache_file_name` is `"processed.arrow"`, then for `rank = 1` and `num_proc = 4`,
| the resulting file would be `"processed_00001_of_00004.arrow"` for the default suffix (default
| `_{rank:05d}_of_{num_proc:05d}`).
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
| desc (`str`, *optional*, defaults to `None`):
| Meaningful description to be displayed alongside with the progress bar while filtering examples.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.filter(lambda x: x["label"] == 1)
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 533
| })
| ```
|
| flatten(self, new_fingerprint: Optional[str] = None, max_depth=16) -> 'Dataset'
| Flatten the table.
| Each column with a struct type is flattened into one column per struct field.
| Other columns are left unchanged.
|
| Args:
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset with flattened columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("rajpurkar/squad", split="train")
| >>> ds.features
| {'id': Value('string'),
| 'title': Value('string'),
| 'context': Value('string'),
| 'question': Value('string'),
| 'answers': {'text': List(Value('string')),
| 'answer_start': List(Value('int32'))}}
| >>> ds = ds.flatten()
| >>> ds
| Dataset({
| features: ['id', 'title', 'context', 'question', 'answers.text', 'answers.answer_start'],
| num_rows: 87599
| })
| ```
|
| flatten_indices(self, keep_in_memory: bool = False, cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, features: Optional[datasets.features.features.Features] = None, disable_nullable: bool = False, num_proc: Optional[int] = None, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Create and cache a new Dataset by flattening the indices mapping.
|
| Args:
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| cache_file_name (`str`, *optional*, default `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| features (`Optional[datasets.Features]`, defaults to `None`):
| Use a specific [`Features`] to store the cache file
| instead of the automatically generated one.
| disable_nullable (`bool`, defaults to `False`):
| Allow null values in the table.
| num_proc (`int`, optional, default `None`):
| Max number of processes when generating cache. Already cached shards are loaded sequentially
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
|
| formatted_as(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| To be used in a `with` statement. Set `__getitem__` return format (type and columns).
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__`` returns python objects (default).
| columns (`List[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| iter(self, batch_size: int, drop_last_batch: bool = False)
| Iterate through the batches of size `batch_size`.
|
| If a formatting is set with [`~datasets.Dataset.set_format`] rows will be returned with the
| selected format.
|
| Args:
| batch_size (:obj:`int`): size of each batch to yield.
| drop_last_batch (:obj:`bool`, default `False`): Whether a last batch smaller than the batch_size should be
| dropped
|
| map(self, function: Optional[Callable] = None, with_indices: bool = False, with_rank: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, drop_last_batch: bool = False, remove_columns: Union[str, list[str], NoneType] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, features: Optional[datasets.features.features.Features] = None, disable_nullable: bool = False, fn_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, suffix_template: str = '_{rank:05d}_of_{num_proc:05d}', new_fingerprint: Optional[str] = None, desc: Optional[str] = None, try_original_type: Optional[bool] = True, on_mixed_types: Optional[Literal['use_json']] = 'use_json') -> 'Dataset'
| Apply a function to all the examples in the table (individually or in batches) and update the table.
| If your function returns a column that already exists, then it overwrites it.
|
| You can specify whether the function should be batched or not with the `batched` parameter:
|
| - If batched is `False`, then the function takes 1 example in and should return 1 example.
| An example is a dictionary, e.g. `{"text": "Hello there !"}`.
| - If batched is `True` and `batch_size` is 1, then the function takes a batch of 1 example as input and can return a batch with 1 or more examples.
| A batch is a dictionary, e.g. a batch of 1 example is `{"text": ["Hello there !"]}`.
| - If batched is `True` and `batch_size` is `n > 1`, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.
| Note that the last batch may have less than `n` examples.
| A batch is a dictionary, e.g. a batch of `n` examples is `{"text": ["Hello there !"] * n}`.
|
| If the function is asynchronous, then `map` will run your function in parallel, with up to one thousand simultaneous calls.
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`): Function with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> Dict[str, Any]` if `batched=False` and `with_indices=False` and `with_rank=False`
| - `function(example: Dict[str, Any], *extra_args) -> Dict[str, Any]` if `batched=False` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
| - `function(batch: Dict[str, List]) -> Dict[str, List]` if `batched=True` and `with_indices=False` and `with_rank=False`
| - `function(batch: Dict[str, List], *extra_args) -> Dict[str, List]` if `batched=True` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
|
| For advanced usage, the function can also return a `pyarrow.Table`.
| If the function is asynchronous, then `map` will run your function in parallel.
| Moreover if your function returns nothing (`None`), then `map` will run your function and return the dataset unchanged.
| If no function is provided, default to identity function: `lambda x: x`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the
| signature of `function` should be `def function(example, idx[, rank]): ...`.
| with_rank (`bool`, defaults to `False`):
| Provide process rank to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], rank): ...`.
| input_columns (`Optional[Union[str, List[str]]]`, defaults to `None`):
| The columns to be passed into `function`
| as positional arguments. If `None`, a `dict` mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| If `batch_size <= 0` or `batch_size == None`, provide the full dataset as a single batch to `function`.
| drop_last_batch (`bool`, defaults to `False`):
| Whether a last batch smaller than the batch_size should be
| dropped instead of being processed by the function.
| remove_columns (`Optional[Union[str, List[str]]]`, defaults to `None`):
| Remove a selection of columns while doing the mapping.
| Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding
| columns with names in `remove_columns`, these columns will be kept.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_name (`str`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| features (`Optional[datasets.Features]`, defaults to `None`):
| Use a specific Features to store the cache file
| instead of the automatically generated one.
| disable_nullable (`bool`, defaults to `False`):
| Disallow null values in the table.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`.
| num_proc (`int`, *optional*, defaults to `None`):
| The number of processes to use for multiprocessing.
| - If `None` or `0`, no multiprocessing is used and the operation runs in the main process.
| - If greater than `1`, one or multiple worker processes are used to process data in parallel.
| Note: The function passed to `map()` must be picklable for multiprocessing to work correctly
| (i.e., prefer functions defined at the top level of a module, not inside another function or class).
| suffix_template (`str`):
| If `cache_file_name` is specified, then this suffix
| will be added at the end of the base name of each. Defaults to `"_{rank:05d}_of_{num_proc:05d}"`. For example, if `cache_file_name` is "processed.arrow", then for
| `rank=1` and `num_proc=4`, the resulting file would be `"processed_00001_of_00004.arrow"` for the default suffix.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
| desc (`str`, *optional*, defaults to `None`):
| Meaningful description to be displayed alongside with the progress bar while mapping examples.
| try_original_type (`Optional[bool]`, defaults to `True`):
| Try to keep the types of the original columns (e.g. int32 -> int32).
| Set to False if you want to always infer new types.
| on_mixed_types (`Literal["use_json"]`, *optional*, defaults to `None`):
| If "use_json", use the Json() type for mixed-types fields,
| i.e. unstructured fields that contain data without a predefined schema.
| In this case, a field with mixed type is set to Json().
|
| This allow loading lists with a mix of strings/integers/floats
| for example, or dictionaries with arbitrary value types.
|
| <Added version="4.7.0"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> def add_prefix(example):
| ... example["text"] = "Review: " + example["text"]
| ... return example
| >>> ds = ds.map(add_prefix)
| >>> ds[0:3]["text"]
| ['Review: compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'Review: the soundtrack alone is worth the price of admission .',
| 'Review: rodriguez does a splendid job of racial profiling hollywood style--casting excellent latin actors of all ages--a trend long overdue .']
|
| # process a batch of examples
| >>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
| # set number of processors
| >>> ds = ds.map(add_prefix, num_proc=4)
| ```
|
| push_to_hub(self, repo_id: str, config_name: str = 'default', set_default: Optional[bool] = None, split: Optional[str] = None, data_dir: Optional[str] = None, commit_message: Optional[str] = None, commit_description: Optional[str] = None, private: Optional[bool] = None, token: Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = False, max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[int] = None, embed_external_files: bool = True, num_proc: Optional[int] = None) -> huggingface_hub.hf_api.CommitInfo
| Pushes the dataset to the hub as a Parquet dataset.
| The dataset is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
|
| The resulting Parquet files are self-contained by default. If your dataset contains [`Image`], [`Audio`] or [`Video`]
| data, the Parquet files will store the bytes of your images or audio files.
| You can disable this by setting `embed_external_files` to `False`.
|
| Args:
| repo_id (`str`):
| The ID of the repository to push to in the following format: `<user>/<dataset_name>` or
| `<org>/<dataset_name>`. Also accepts `<dataset_name>`, which will default to the namespace
| of the logged-in user.
|
| It could also be a location inside a bucket, e.g. `buckets/<user_or_org>/<bucket_name>/...`
| config_name (`str`, defaults to "default"):
| The configuration name (or subset) of a dataset. Defaults to "default".
| set_default (`bool`, *optional*):
| Whether to set this configuration as the default one. Otherwise, the default configuration is the one
| named "default".
| split (`str`, *optional*):
| The name of the split that will be given to that dataset. Defaults to `self.split`.
| data_dir (`str`, *optional*):
| Directory name that will contain the uploaded data files. Defaults to the `config_name` if different
| from "default", else "data".
|
| <Added version="2.17.0"/>
| commit_message (`str`, *optional*):
| Message to commit while pushing. Will default to `"Upload dataset"`.
| commit_description (`str`, *optional*):
| Description of the commit that will be created.
| Additionally, description of the PR if a PR is created (`create_pr` is True).
|
| <Added version="2.16.0"/>
| private (`bool`, *optional*):
| Whether to make the repo private. If `None` (default), the repo will be public unless the
| organization's default is private. This value is ignored if the repo already exists.
| token (`str`, *optional*):
| An optional authentication token for the Hugging Face Hub. If no token is passed, will default
| to the token saved locally when logging in with `huggingface-cli login`. Will raise an error
| if no token is passed and the user is not logged-in.
| revision (`str`, *optional*):
| Branch to push the uploaded files to. Defaults to the `"main"` branch.
|
| <Added version="2.15.0"/>
| create_pr (`bool`, *optional*, defaults to `False`):
| Whether to create a PR with the uploaded files or directly commit.
|
| <Added version="2.15.0"/>
| max_shard_size (`int` or `str`, *optional*, defaults to `"500MB"`):
| The maximum size of the dataset shards to be uploaded to the hub. If expressed as a string, needs to be digits followed by
| a unit (like `"5MB"`).
| num_shards (`int`, *optional*):
| Number of shards to write. By default, the number of shards depends on `max_shard_size`.
|
| <Added version="2.8.0"/>
| embed_external_files (`bool`, defaults to `True`):
| Whether to embed file bytes in the shards.
| In particular, this will do the following before the push for the fields of type:
|
| - [`Audio`] and [`Image`]: remove local path information and embed file content in the Parquet files.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when preparing and uploading the dataset.
| This is helpful if the dataset is made of many samples or media files to embed.
| I uses "spawn" context to work with hf_xet, the rust client for fast uploads to HF.
| Multiprocessing is disabled by default.
|
| <Added version="4.0.0"/>
|
| Return:
| huggingface_hub.CommitInfo
|
| Example:
|
| ```python
| >>> dataset.push_to_hub("<organization>/<dataset_id>")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", private=True)
| >>> dataset.push_to_hub("<organization>/<dataset_id>", max_shard_size="1GB")
| >>> dataset.push_to_hub("<organization>/<dataset_id>", num_shards=1024)
| ```
|
| If your dataset has multiple splits (e.g. train/validation/test):
|
| ```python
| >>> train_dataset.push_to_hub("<organization>/<dataset_id>", split="train")
| >>> val_dataset.push_to_hub("<organization>/<dataset_id>", split="validation")
| >>> # later
| >>> dataset = load_dataset("<organization>/<dataset_id>")
| >>> train_dataset = dataset["train"]
| >>> val_dataset = dataset["validation"]
| ```
|
| If you want to add a new configuration (or subset) to a dataset (e.g. if the dataset has multiple tasks/versions/languages):
|
| ```python
| >>> english_dataset.push_to_hub("<organization>/<dataset_id>", "en")
| >>> french_dataset.push_to_hub("<organization>/<dataset_id>", "fr")
| >>> # later
| >>> english_dataset = load_dataset("<organization>/<dataset_id>", "en")
| >>> french_dataset = load_dataset("<organization>/<dataset_id>", "fr")
| ```
|
| remove_columns(self, column_names: Union[str, list[str]], new_fingerprint: Optional[str] = None) -> 'Dataset'
| Remove one or several column(s) in the dataset and the features associated to them.
|
| You can also remove a column using [`~datasets.Dataset.map`] with `remove_columns` but the present method
| doesn't copy the data of the remaining columns and is thus faster.
|
| Args:
| column_names (`Union[str, List[str]]`):
| Name of the column(s) to remove.
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset object without the columns to remove.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.remove_columns('label')
| Dataset({
| features: ['text'],
| num_rows: 1066
| })
| >>> ds = ds.remove_columns(column_names=ds.column_names) # Removing all the columns returns an empty dataset with the `num_rows` property set to 0
| Dataset({
| features: [],
| num_rows: 0
| })
| ```
|
| rename_column(self, original_column_name: str, new_column_name: str, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Rename a column in the dataset, and move the features associated to the original column under the new column
| name.
|
| Args:
| original_column_name (`str`):
| Name of the column to rename.
| new_column_name (`str`):
| New name for the column.
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset with a renamed column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.rename_column('label', 'label_new')
| Dataset({
| features: ['text', 'label_new'],
| num_rows: 1066
| })
| ```
|
| rename_columns(self, column_mapping: dict[str, str], new_fingerprint: Optional[str] = None) -> 'Dataset'
| Rename several columns in the dataset, and move the features associated to the original columns under
| the new column names.
|
| Args:
| column_mapping (`Dict[str, str]`):
| A mapping of columns to rename to their new names
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset with renamed columns
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.rename_columns({'text': 'text_new', 'label': 'label_new'})
| Dataset({
| features: ['text_new', 'label_new'],
| num_rows: 1066
| })
| ```
|
| repeat(self, num_times: int) -> 'Dataset'
| Create a new [`Dataset`] that repeats the underlying dataset `num_times` times.
|
| Like itertools.repeat, repeating once just returns the full dataset.
|
| Args:
| num_times (`int`):
| Number of times to repeat the dataset.
|
| Example:
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> ds = ds.take(2).repeat(2)
| >>> list(ds)
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'},
| {'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| ```
|
| reset_format(self)
| Reset `__getitem__` return format to python objects and all columns.
|
| Same as `self.set_format()`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds.set_format(type='numpy', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
| >>> ds.format
| {'columns': ['input_ids', 'token_type_ids', 'attention_mask', 'label'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'numpy'}
| >>> ds.reset_format()
| >>> ds.format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': None}
| ```
|
| save_to_disk(self, dataset_path: Union[str, bytes, os.PathLike], max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[int] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None)
| Saves a dataset to a dataset directory, or in a filesystem using any implementation of `fsspec.spec.AbstractFileSystem`.
|
| For [`Image`], [`Audio`] and [`Video`] data:
|
| All the Image(), Audio() and Video() data are stored in the arrow files.
| If you want to store paths or urls, please use the Value("string") type.
|
| Args:
| dataset_path (`path-like`):
| Path (e.g. `dataset/train`) or remote URI (e.g. `s3://my-bucket/dataset/train`)
| of the dataset directory where the dataset will be saved to.
| max_shard_size (`int` or `str`, *optional*, defaults to `"500MB"`):
| The maximum size of the dataset shards to be saved to the filesystem. If expressed as a string, needs to be digits followed by a unit
| (like `"50MB"`).
| num_shards (`int`, *optional*):
| Number of shards to write. By default the number of shards depends on `max_shard_size` and `num_proc`.
|
| <Added version="2.8.0"/>
| num_proc (`int`, *optional*):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.8.0"/>
|
| Example:
|
| ```py
| >>> ds.save_to_disk("path/to/dataset/directory")
| >>> ds.save_to_disk("path/to/dataset/directory", max_shard_size="1GB")
| >>> ds.save_to_disk("path/to/dataset/directory", num_shards=1024)
| ```
|
| select(self, indices: collections.abc.Iterable, keep_in_memory: bool = False, indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Create a new dataset with rows selected following the list/array of indices.
|
| Args:
| indices (`range`, `list`, `iterable`, `ndarray` or `Series`):
| Range, list or 1D-array of integer indices for indexing.
| If the indices correspond to a contiguous range, the Arrow table is simply sliced.
| However passing a list of indices that are not contiguous creates indices mapping, which is much less efficient,
| but still faster than recreating an Arrow table made of the requested rows.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the indices mapping in memory instead of writing it to a cache file.
| indices_cache_file_name (`str`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| indices mapping instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.select(range(4))
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 4
| })
| ```
|
| select_columns(self, column_names: Union[str, list[str]], new_fingerprint: Optional[str] = None) -> 'Dataset'
| Select one or several column(s) in the dataset and the features
| associated to them.
|
| Args:
| column_names (`Union[str, List[str]]`):
| Name of the column(s) to keep.
| new_fingerprint (`str`, *optional*):
| The new fingerprint of the dataset after transform. If `None`,
| the new fingerprint is computed using a hash of the previous
| fingerprint, and the transform arguments.
|
| Returns:
| [`Dataset`]: A copy of the dataset object which only consists of
| selected columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.select_columns(['text'])
| >>> ds
| Dataset({
| features: ['text'],
| num_rows: 1066
| })
| ```
|
| set_format(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| Set `__getitem__` return format (type and columns). The data formatting is applied on-the-fly.
| The format `type` (for example "numpy") is used to format batches when using `__getitem__`.
| It's also possible to use custom transforms for formatting using [`~datasets.Dataset.set_transform`].
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`List[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| It is possible to call [`~datasets.Dataset.map`] after calling `set_format`. Since `map` may add new columns, then the list of formatted columns
| gets updated. In this case, if you apply `map` on a dataset to add a new column, then this column will be formatted as:
|
| ```
| new formatted columns = (all columns - previously unformatted columns)
| ```
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds.set_format(type='numpy', columns=['text', 'label'])
| >>> ds.format
| {'type': 'numpy',
| 'format_kwargs': {},
| 'columns': ['text', 'label'],
| 'output_all_columns': False}
| ```
|
| set_transform(self, transform: Optional[Callable], columns: Optional[list] = None, output_all_columns: bool = False)
| Set `__getitem__` return format using this transform. The transform is applied on-the-fly on batches when `__getitem__` is called.
| As [`~datasets.Dataset.set_format`], this can be reset using [`~datasets.Dataset.reset_format`].
|
| Args:
| transform (`Callable`, *optional*):
| User-defined formatting transform, replaces the format defined by [`~datasets.Dataset.set_format`].
| A formatting function is a callable that takes a batch (as a `dict`) as input and returns a batch.
| This function is applied right before returning the objects in `__getitem__`.
| columns (`List[str]`, *optional*):
| Columns to format in the output.
| If specified, then the input batch of the transform only contains those columns.
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| If set to True, then the other un-formatted columns are kept with the output of the transform.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
| >>> def encode(batch):
| ... return tokenizer(batch['text'], padding=True, truncation=True, return_tensors='pt')
| >>> ds.set_transform(encode)
| >>> ds[0]
| {'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1]),
| 'input_ids': tensor([ 101, 29353, 2135, 15102, 1996, 9428, 20868, 2890, 8663, 6895,
| 20470, 2571, 3663, 2090, 4603, 3017, 3008, 1998, 2037, 24211,
| 5637, 1998, 11690, 2336, 1012, 102]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0])}
| ```
|
| shard(self, num_shards: int, index: int, contiguous: bool = True, keep_in_memory: bool = False, indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000) -> 'Dataset'
| Return the `index`-nth shard from dataset split into `num_shards` pieces.
|
| This shards deterministically. `dataset.shard(n, i)` splits the dataset into contiguous chunks,
| so it can be easily concatenated back together after processing. If `len(dataset) % n == l`, then the
| first `l` dataset each have length `(len(dataset) // n) + 1`, and the remaining dataset have length `(len(dataset) // n)`.
| `datasets.concatenate_datasets([dset.shard(n, i) for i in range(n)])` returns a dataset with the same order as the original.
|
| Note: n should be less or equal to the number of elements in the dataset `len(dataset)`.
|
| On the other hand, `dataset.shard(n, i, contiguous=False)` contains all elements of the dataset whose index mod `n = i`.
|
| Be sure to shard before using any randomizing operator (such as `shuffle`).
| It is best if the shard operator is used early in the dataset pipeline.
|
| Args:
| num_shards (`int`):
| How many shards to split the dataset into.
| index (`int`):
| Which shard to select and return.
| contiguous: (`bool`, defaults to `True`):
| Whether to select contiguous blocks of indices for shards.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| indices_cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| indices of each shard instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| This only concerns the indices mapping.
| Number of indices per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 1066
| })
| >>> ds = ds.shard(num_shards=2, index=0)
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 533
| })
| ```
|
| shuffle(self, seed: Optional[int] = None, generator: Optional[numpy.random._generator.Generator] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Create a new Dataset where the rows are shuffled.
|
| Currently shuffling uses numpy random generators.
| You can either supply a NumPy BitGenerator to use, or a seed to initiate NumPy's default random generator (PCG64).
|
| Shuffling takes the list of indices `[0:len(my_dataset)]` and shuffles it to create an indices mapping.
| However as soon as your [`Dataset`] has an indices mapping, the speed can become 10x slower.
| This is because there is an extra step to get the row index to read using the indices mapping, and most importantly, you aren't reading contiguous chunks of data anymore.
| To restore the speed, you'd need to rewrite the entire dataset on your disk again using [`Dataset.flatten_indices`], which removes the indices mapping.
| This may take a lot of time depending of the size of your dataset though:
|
| ```python
| my_dataset[0] # fast
| my_dataset = my_dataset.shuffle(seed=42)
| my_dataset[0] # up to 10x slower
| my_dataset = my_dataset.flatten_indices() # rewrite the shuffled dataset on disk as contiguous chunks of data
| my_dataset[0] # fast again
| ```
|
| In this case, we recommend switching to an [`IterableDataset`] and leveraging its fast approximate shuffling method [`IterableDataset.shuffle`].
| It only shuffles the shards order and adds a shuffle buffer to your dataset, which keeps the speed of your dataset optimal:
|
| ```python
| my_iterable_dataset = my_dataset.to_iterable_dataset(num_shards=128)
| for example in enumerate(my_iterable_dataset): # fast
| pass
|
| shuffled_iterable_dataset = my_iterable_dataset.shuffle(seed=42, buffer_size=100)
|
| for example in enumerate(shuffled_iterable_dataset): # as fast as before
| pass
| ```
|
| Args:
| seed (`int`, *optional*):
| A seed to initialize the default BitGenerator if `generator=None`.
| If `None`, then fresh, unpredictable entropy will be pulled from the OS.
| If an `int` or `array_like[ints]` is passed, then it will be passed to SeedSequence to derive the initial BitGenerator state.
| generator (`numpy.random.Generator`, *optional*):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| keep_in_memory (`bool`, default `False`):
| Keep the shuffled indices in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the shuffled indices
| can be identified, use it instead of recomputing.
| indices_cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| shuffled indices instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
|
| # set a seed
| >>> shuffled_ds = ds.shuffle(seed=42)
| >>> shuffled_ds['label'][:10]
| [1, 0, 1, 1, 0, 0, 0, 0, 0, 0]
| ```
|
| skip(self, n: int) -> 'Dataset'
| Create a new [`Dataset`] that skips the first `n` elements.
|
| Args:
| n (`int`):
| Number of elements to skip.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| >>> ds = ds.skip(1)
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'},
| {'label': 1,
| 'text': 'if you sometimes like to go to the movies to have fun , wasabi is a good place to start .'}]
| ```
|
| sort(self, column_names: Union[str, collections.abc.Sequence[str]], reverse: Union[bool, collections.abc.Sequence[bool]] = False, null_placement: str = 'at_end', keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, new_fingerprint: Optional[str] = None) -> 'Dataset'
| Create a new dataset sorted according to a single or multiple columns.
|
| Args:
| column_names (`Union[str, Sequence[str]]`):
| Column name(s) to sort by.
| reverse (`Union[bool, Sequence[bool]]`, defaults to `False`):
| If `True`, sort by descending order rather than ascending. If a single bool is provided,
| the value is applied to the sorting of all column names. Otherwise a list of bools with the
| same length and order as column_names must be provided.
| null_placement (`str`, defaults to `at_end`):
| Put `None` values at the beginning if `at_start` or `first` or at the end if `at_end` or `last`
|
| <Added version="1.14.2"/>
| keep_in_memory (`bool`, defaults to `False`):
| Keep the sorted indices in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the sorted indices
| can be identified, use it instead of recomputing.
| indices_cache_file_name (`str`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| sorted indices instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| Higher value gives smaller cache files, lower value consume less temporary memory.
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset('cornell-movie-review-data/rotten_tomatoes', split='validation')
| >>> ds['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
| >>> sorted_ds = ds.sort('label')
| >>> sorted_ds['label'][:10]
| [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
| >>> another_sorted_ds = ds.sort(['label', 'text'], reverse=[True, False])
| >>> another_sorted_ds['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
| ```
|
| take(self, n: int) -> 'Dataset'
| Create a new [`Dataset`] with only the first `n` elements.
|
| Args:
| n (`int`):
| Number of elements to take.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> small_ds = ds.take(2)
| >>> list(small_ds)
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}]
| ```
|
| to_csv(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **to_csv_kwargs) -> int
| Exports the dataset to csv
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.csv`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.csv`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| num_proc (`int`, *optional*):
| Number of processes for multiprocessing. By default it doesn't
| use multiprocessing. `batch_size` in this case defaults to
| `datasets.config.DEFAULT_MAX_BATCH_SIZE` but feel free to make it 5x or 10x of the default
| value if you have sufficient compute power.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.19.0"/>
| **to_csv_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_csv`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html).
|
| <Changed version="2.10.0">
|
| Now, `index` defaults to `False` if not specified.
|
| If you would like to write the index, pass `index=True` and also set a name for the index column by
| passing `index_label`.
|
| </Changed>
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_csv("path/to/dataset/directory")
| ```
|
| to_dict(self, batch_size: Optional[int] = None, batched: bool = False) -> Union[dict, collections.abc.Iterator[dict]]
| Returns the dataset as a Python dict. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*): The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
|
| Returns:
| `dict` or `Iterator[dict]`
|
| Example:
|
| ```py
| >>> ds.to_dict()
| ```
|
| to_iterable_dataset(self, num_shards: Optional[int] = 1) -> 'IterableDataset'
| Get an [`datasets.IterableDataset`] from a map-style [`datasets.Dataset`].
| This is equivalent to loading a dataset in streaming mode with [`datasets.load_dataset`], but much faster since the data is streamed from local files.
|
| Contrary to map-style datasets, iterable datasets are lazy and can only be iterated over (e.g. using a for loop).
| Since they are read sequentially in training loops, iterable datasets are much faster than map-style datasets.
| All the transformations applied to iterable datasets like filtering or processing are done on-the-fly when you start iterating over the dataset.
|
| Still, it is possible to shuffle an iterable dataset using [`datasets.IterableDataset.shuffle`].
| This is a fast approximate shuffling that works best if you have multiple shards and if you specify a buffer size that is big enough.
|
| To get the best speed performance, make sure your dataset doesn't have an indices mapping.
| If this is the case, the data are not read contiguously, which can be slow sometimes.
| You can use `ds = ds.flatten_indices()` to write your dataset in contiguous chunks of data and have optimal speed before switching to an iterable dataset.
|
| Args:
| num_shards (`int`, default to `1`):
| Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly,
| and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example.
| Shards are defined using [`datasets.Dataset.shard`]: it simply slices the data without writing anything on disk.
|
| Returns:
| [`datasets.IterableDataset`]
|
| Example:
|
| Basic usage:
| ```python
| >>> ids = ds.to_iterable_dataset()
| >>> for example in ids:
| ... pass
| ```
|
| With lazy filtering and processing:
| ```python
| >>> ids = ds.to_iterable_dataset()
| >>> ids = ids.filter(filter_fn).map(process_fn) # will filter and process on-the-fly when you start iterating over the iterable dataset
| >>> for example in ids:
| ... pass
| ```
|
| With sharding to enable efficient shuffling:
| ```python
| >>> ids = ds.to_iterable_dataset(num_shards=64) # the dataset is split into 64 shards to be iterated over
| >>> ids = ids.shuffle(buffer_size=10_000) # will shuffle the shards order and use a shuffle buffer for fast approximate shuffling when you start iterating
| >>> for example in ids:
| ... pass
| ```
|
| With a PyTorch DataLoader:
| ```python
| >>> import torch
| >>> ids = ds.to_iterable_dataset(num_shards=64)
| >>> ids = ids.filter(filter_fn).map(process_fn)
| >>> dataloader = torch.utils.data.DataLoader(ids, num_workers=4) # will assign 64 / 4 = 16 shards to each worker to load, filter and process when you start iterating
| >>> for example in ids:
| ... pass
| ```
|
| With a PyTorch DataLoader and shuffling:
| ```python
| >>> import torch
| >>> ids = ds.to_iterable_dataset(num_shards=64)
| >>> ids = ids.shuffle(buffer_size=10_000) # will shuffle the shards order and use a shuffle buffer when you start iterating
| >>> dataloader = torch.utils.data.DataLoader(ids, num_workers=4) # will assign 64 / 4 = 16 shards from the shuffled list of shards to each worker when you start iterating
| >>> for example in ids:
| ... pass
| ```
|
| In a distributed setup like PyTorch DDP with a PyTorch DataLoader and shuffling
| ```python
| >>> from datasets.distributed import split_dataset_by_node
| >>> ids = ds.to_iterable_dataset(num_shards=512)
| >>> ids = ids.shuffle(buffer_size=10_000, seed=42) # will shuffle the shards order and use a shuffle buffer when you start iterating
| >>> ids = split_dataset_by_node(ds, world_size=8, rank=0) # will keep only 512 / 8 = 64 shards from the shuffled lists of shards when you start iterating
| >>> dataloader = torch.utils.data.DataLoader(ids, num_workers=4) # will assign 64 / 4 = 16 shards from this node's list of shards to each worker when you start iterating
| >>> for example in ids:
| ... pass
| ```
|
| With shuffling and multiple epochs:
| ```python
| >>> ids = ds.to_iterable_dataset(num_shards=64)
| >>> ids = ids.shuffle(buffer_size=10_000, seed=42) # will shuffle the shards order and use a shuffle buffer when you start iterating
| >>> for epoch in range(n_epochs):
| ... ids.set_epoch(epoch) # will use effective_seed = seed + epoch to shuffle the shards and for the shuffle buffer when you start iterating
| ... for example in ids:
| ... pass
| ```
| Feel free to also use [`IterableDataset.set_epoch`] when using a PyTorch DataLoader or in distributed setups.
|
| to_json(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **to_json_kwargs) -> int
| Export the dataset to JSON Lines or JSON.
|
| The default output format is [JSON Lines](https://jsonlines.org/).
| To export to [JSON](https://www.json.org), pass `lines=False` argument and the desired `orient`.
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.json`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.json`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| num_proc (`int`, *optional*):
| Number of processes for multiprocessing. By default, it doesn't
| use multiprocessing. `batch_size` in this case defaults to
| `datasets.config.DEFAULT_MAX_BATCH_SIZE` but feel free to make it 5x or 10x of the default
| value if you have sufficient compute power.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.19.0"/>
| **to_json_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_json`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html).
| Default arguments are `lines=True` and `orient="records".
|
| <Changed version="2.11.0">
|
| The parameter `index` defaults to `False` if `orient` is `"split"` or `"table"`.
|
| If you would like to write the index, pass `index=True`.
|
| </Changed>
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_json("path/to/dataset/directory/filename.jsonl")
| ```
|
| to_list(self) -> list
| Returns the dataset as a Python list.
|
| Returns:
| `list`
|
| Example:
|
| ```py
| >>> ds.to_list()
| ```
|
| to_pandas(self, batch_size: Optional[int] = None, batched: bool = False) -> Union[pandas.core.frame.DataFrame, collections.abc.Iterator[pandas.core.frame.DataFrame]]
| Returns the dataset as a `pandas.DataFrame`. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*):
| The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
|
| Returns:
| `pandas.DataFrame` or `Iterator[pandas.DataFrame]`
|
| Example:
|
| ```py
| >>> ds.to_pandas()
| ```
|
| to_parquet(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, storage_options: Optional[dict] = None, **parquet_writer_kwargs) -> int
| Exports the dataset to parquet
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.parquet`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.parquet`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| By default it aims for row groups with maximum uncompressed byte size of "100MB",
| defined by `datasets.config.MAX_ROW_GROUP_SIZE`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.19.0"/>
| **parquet_writer_kwargs (additional keyword arguments):
| Parameters to pass to PyArrow's `pyarrow.parquet.ParquetWriter`.
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_parquet("path/to/dataset/directory")
| ```
|
| to_polars(self, batch_size: Optional[int] = None, batched: bool = False, schema_overrides: Optional[dict] = None, rechunk: bool = True) -> Union[ForwardRef('pl.DataFrame'), collections.abc.Iterator['pl.DataFrame']]
| Returns the dataset as a `polars.DataFrame`. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*):
| The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
| schema_overrides (`dict`, *optional*):
| Support type specification or override of one or more columns; note that
| any dtypes inferred from the schema param will be overridden.
| rechunk (`bool`):
| Make sure that all data is in contiguous memory. Defaults to `True`.
| Returns:
| `polars.DataFrame` or `Iterator[polars.DataFrame]`
|
| Example:
|
| ```py
| >>> ds.to_polars()
| ```
|
| to_sql(self, name: str, con: Union[str, ForwardRef('sqlalchemy.engine.Connection'), ForwardRef('sqlalchemy.engine.Engine'), ForwardRef('sqlite3.Connection')], batch_size: Optional[int] = None, num_proc: Optional[int] = None, **sql_writer_kwargs) -> int
| Exports the dataset to a SQL database.
|
| Args:
| name (`str`):
| Name of SQL table.
| con (`str` or `sqlite3.Connection` or `sqlalchemy.engine.Connection` or `sqlalchemy.engine.Connection`):
| A [URI string](https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls) or a SQLite3/SQLAlchemy connection object used to write to a database.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| num_proc (`int`, *optional*):
| Number of processes for multiprocessing. By default, it doesn't
| use multiprocessing. `batch_size` in this case defaults to
| `datasets.config.DEFAULT_MAX_BATCH_SIZE` but feel free to make it 5x or 10x of the default
| value if you have sufficient compute power.
| **sql_writer_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_sql`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_sql.html).
|
| <Changed version="2.11.0">
|
| Now, `index` defaults to `False` if not specified.
|
| If you would like to write the index, pass `index=True` and also set a name for the index column by
| passing `index_label`.
|
| </Changed>
|
| Returns:
| `int`: The number of records written.
|
| Example:
|
| ```py
| >>> # con provided as a connection URI string
| >>> ds.to_sql("data", "sqlite:///my_own_db.sql")
| >>> # con provided as a sqlite3 connection object
| >>> import sqlite3
| >>> con = sqlite3.connect("my_own_db.sql")
| >>> with con:
| ... ds.to_sql("data", con)
| ```
|
| train_test_split(self, test_size: Union[float, int, NoneType] = None, train_size: Union[float, int, NoneType] = None, shuffle: bool = True, stratify_by_column: Optional[str] = None, seed: Optional[int] = None, generator: Optional[numpy.random._generator.Generator] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, train_indices_cache_file_name: Optional[str] = None, test_indices_cache_file_name: Optional[str] = None, writer_batch_size: Optional[int] = 1000, train_new_fingerprint: Optional[str] = None, test_new_fingerprint: Optional[str] = None) -> 'DatasetDict'
| Return a dictionary ([`datasets.DatasetDict`]) with two random train and test subsets (`train` and `test` `Dataset` splits).
| Splits are created from the dataset according to `test_size`, `train_size` and `shuffle`.
|
| This method is similar to scikit-learn `train_test_split`.
|
| Args:
| test_size (`Union[float, int, None]`, *optional*):
| Size of the test split
| If `float`, should be between `0.0` and `1.0` and represent the proportion of the dataset to include in the test split.
| If `int`, represents the absolute number of test samples.
| If `None`, the value is set to the complement of the train size.
| If `train_size` is also `None`, it will be set to `0.25`.
| train_size (`Union[float, int, None]`, *optional*):
| Size of the train split
| If `float`, should be between `0.0` and `1.0` and represent the proportion of the dataset to include in the train split.
| If `int`, represents the absolute number of train samples.
| If `None`, the value is automatically set to the complement of the test size.
| shuffle (`bool`, *optional*, defaults to `True`):
| Whether or not to shuffle the data before splitting.
| stratify_by_column (`str`, *optional*, defaults to `None`):
| The column name of labels to be used to perform stratified split of data.
| seed (`int`, *optional*):
| A seed to initialize the default BitGenerator if `generator=None`.
| If `None`, then fresh, unpredictable entropy will be pulled from the OS.
| If an `int` or `array_like[ints]` is passed, then it will be passed to SeedSequence to derive the initial BitGenerator state.
| generator (`numpy.random.Generator`, *optional*):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| keep_in_memory (`bool`, defaults to `False`):
| Keep the splits indices in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the splits indices
| can be identified, use it instead of recomputing.
| train_cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| train split indices instead of the automatically generated cache file name.
| test_cache_file_name (`str`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| test split indices instead of the automatically generated cache file name.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| train_new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the train set after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
| test_new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the test set after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds = ds.train_test_split(test_size=0.2, shuffle=True)
| DatasetDict({
| train: Dataset({
| features: ['text', 'label'],
| num_rows: 852
| })
| test: Dataset({
| features: ['text', 'label'],
| num_rows: 214
| })
| })
|
| # set a seed
| >>> ds = ds.train_test_split(test_size=0.2, seed=42)
|
| # stratified split
| >>> ds = load_dataset("stanfordnlp/imdb",split="train")
| Dataset({
| features: ['text', 'label'],
| num_rows: 25000
| })
| >>> ds = ds.train_test_split(test_size=0.2, stratify_by_column="label")
| DatasetDict({
| train: Dataset({
| features: ['text', 'label'],
| num_rows: 20000
| })
| test: Dataset({
| features: ['text', 'label'],
| num_rows: 5000
| })
| })
| ```
|
| unique(self, column: str) -> list
| Return a list of the unique elements in a column.
|
| This is implemented in the low-level backend and as such, very fast.
|
| Args:
| column (`str`):
| Column name (list all the column names with [`~datasets.Dataset.column_names`]).
|
| Returns:
| `list`: List of unique elements in the given column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.unique('label')
| [1, 0]
| ```
|
| with_format(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| Set `__getitem__` return format (type and columns). The data formatting is applied on-the-fly.
| The format `type` (for example "numpy") is used to format batches when using `__getitem__`.
|
| It's also possible to use custom transforms for formatting using [`~datasets.Dataset.with_transform`].
|
| Contrary to [`~datasets.Dataset.set_format`], `with_format` returns a new [`Dataset`] object.
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`List[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds.format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': None}
| >>> ds = ds.with_format("torch")
| >>> ds.format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'torch'}
| >>> ds[0]
| {'text': 'compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'label': tensor(1),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
| 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| with_transform(self, transform: Optional[Callable], columns: Optional[list] = None, output_all_columns: bool = False)
| Set `__getitem__` return format using this transform. The transform is applied on-the-fly on batches when `__getitem__` is called.
|
| As [`~datasets.Dataset.set_format`], this can be reset using [`~datasets.Dataset.reset_format`].
|
| Contrary to [`~datasets.Dataset.set_transform`], `with_transform` returns a new [`Dataset`] object.
|
| Args:
| transform (`Callable`, `optional`):
| User-defined formatting transform, replaces the format defined by [`~datasets.Dataset.set_format`].
| A formatting function is a callable that takes a batch (as a `dict`) as input and returns a batch.
| This function is applied right before returning the objects in `__getitem__`.
| columns (`List[str]`, `optional`):
| Columns to format in the output.
| If specified, then the input batch of the transform only contains those columns.
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| If set to `True`, then the other un-formatted columns are kept with the output of the transform.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> def encode(example):
| ... return tokenizer(example["text"], padding=True, truncation=True, return_tensors='pt')
| >>> ds = ds.with_transform(encode)
| >>> ds[0]
| {'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1]),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0])}
| ```
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_buffer(buffer: pyarrow.lib.Buffer, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, indices_buffer: Optional[pyarrow.lib.Buffer] = None) -> 'Dataset'
| Instantiate a Dataset backed by an Arrow buffer.
|
| Args:
| buffer (`pyarrow.Buffer`):
| Arrow buffer.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| indices_buffer (`pyarrow.Buffer`, *optional*):
| Indices Arrow buffer.
|
| Returns:
| [`Dataset`]
|
| from_dict(mapping: dict, features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, on_mixed_types: Optional[Literal['use_json']] = None) -> 'Dataset'
| Convert `dict` to a `pyarrow.Table` to create a [`Dataset`].
|
| Important: a dataset created with from_dict() lives in memory
| and therefore doesn't have an associated cache directory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. save_to_disk / load_from_disk.
|
| Args:
| mapping (`Mapping`):
| Mapping of strings to Arrays or Python lists.
| features ([`Features`], *optional*):
| Dataset features.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| on_mixed_types (`Literal["use_json"]`, *optional*, defaults to `None`):
| If "use_json", use the Json() type for mixed-types fields,
| i.e. unstructured fields that contain data without a predefined schema.
| In this case, a field with mixed type is set to Json().
|
| This allow loading lists with a mix of strings/integers/floats
| for example, or dictionaries with arbitrary value types.
|
| <Added version="4.7.0"/>
|
| Returns:
| [`Dataset`]
|
| Examples:
|
| Get a Dataset from a dictionary containing one list per column:
|
| ```py
| >>> ds = Dataset.from_dict({"text": ["hello there !", "general kenobi !"]})
| ```
|
| Pass features to set the column types, e.g. for an image dataset:
|
| ```py
| >>> features = Features({"image": Image()})
| >>> ds = Dataset.from_dict({"image": ["path/to/image.png"]}, features=features)
| ```
|
| Datasets are based on Arrow which is a columnar format, and therefore they expect every example to have the same
| type and subtypes, and dictionaries to have the same keys and values types.
| Loading a dataset errors out when fields have mismatching types, and fills missing fields in dictionaries with None so all dictionaries have the same keys and value types.
|
| To avoid this and allow mixed-types without errors, you can use `on_mixed_types="use_json"` or specify `features=` with a [`Json`] type:
|
| ```py
| >>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]})
| Traceback (most recent call last):
| ...
| File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
| pyarrow.lib.ArrowInvalid: Could not convert 'foo' with type str: tried to convert to int64
|
| >>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]}, on_mixed_types="use_json")
| >>> ds.features
| {'a': Json()}
| >>> list(ds["a"])
| [0, "foo", {"subfield": "bar"}]
|
| >>> features = Features({"a": Json()})
| >>> ds = Dataset.from_dict({"a": [0, "foo", {"subfield": "bar"}]}, features=features)
| >>> ds.features
| {'a': Json()}
| >>> list(ds["a"])
| [0, "foo", {"subfield": "bar"}]
| ```
|
| This is also useful for lists of dictionaries with arbitrary keys and values, to avoid filling missing fields with None:
|
| ```py
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]})
| >>> ds.features
| {'a': List({'b': Value('int64'), 'c': Value('int64')})}
| >>> list(ds["a"])
| [[{'b': 0, 'c': None}, {'b': None, 'c': 0}]] # missing fields are filled with None
|
| >>> features = Features({"a": List(Json())})
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, features=features)
| >>> ds.features
| {'a': List(Json())}
| >>> list(ds["a"])
| [[{'b': 0}, {'c': 0}]] # OK
|
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, on_mixed_types="use_json")
| >>> ds.features
| {'a': List(Json())}
| >>> list(ds["a"])
| [[{'b': 0}, {'c': 0}]] # OK
| ```
|
| Another example with tool calling data:
|
| ```py
| >>> messages = [
| ... {"role": "user", "content": "Turn on the living room lights and play my electronic music playlist."},
| ... {"role": "assistant", "tool_calls": [
| ... {"type": "function", "function": {
| ... "name": "control_light",
| ... "arguments": {"room": "living room", "state": "on"}
| ... }},
| ... {"type": "function", "function": {
| ... "name": "play_music",
| ... "arguments": {"playlist": "electronic"} # mixed-type here since keys ["playlist"] and ["room", "state"] are different
| ... }}]
| ... },
| ... {"role": "tool", "name": "control_light", "content": "The lights in the living room are now on."},
| ... {"role": "tool", "name": "play_music", "content": "The music is now playing."},
| ... {"role": "assistant", "content": "Done!"}
| ... ]
| >>> ds = Dataset.from_dict({"messages": [messages]}, on_mixed_types="use_json")
| >>> ds.features
| {'messages': List({'role': Value('string'), 'content': Value('string'), 'tool_calls': List(Json()), 'name': Value('string')})}
| >>> ds[0]["messages"][1]["tool_calls"][0]["function"]["arguments"]
| {"room": "living room", "state": "on"}
| ```
|
| from_file(filename: str, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, indices_filename: Optional[str] = None, in_memory: bool = False) -> 'Dataset'
| Instantiate a Dataset backed by an Arrow table at filename.
|
| Args:
| filename (`str`):
| File name of the dataset.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| indices_filename (`str`, *optional*):
| File names of the indices.
| in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
|
| Returns:
| [`Dataset`]
|
| from_list(mapping: list[dict], features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, on_mixed_types: Optional[Literal['use_json']] = None) -> 'Dataset'
| Convert a list of dicts to a `pyarrow.Table` to create a [`Dataset`]`.
|
| Note that the keys of the first entry will be used to determine the dataset columns,
| regardless of what is passed to features.
|
| Important: a dataset created with from_list() lives in memory
| and therefore doesn't have an associated cache directory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. save_to_disk / load_from_disk.
|
| Args:
| mapping (`List[dict]`): A list of mappings of strings to row values.
| features (`Features`, optional): Dataset features.
| info (`DatasetInfo`, optional): Dataset information, like description, citation, etc.
| split (`NamedSplit`, optional): Name of the dataset split.
| on_mixed_types (`Literal["use_json"]`, *optional*, defaults to `None`):
| If "use_json", use the Json() type for mixed-types fields,
| i.e. unstructured fields that contain data without a predefined schema.
| In this case, a field with mixed type is set to Json().
|
| This allow loading lists with a mix of strings/integers/floats
| for example, or dictionaries with arbitrary value types.
|
| <Added version="4.7.0"/>
|
| Returns:
| [`Dataset`]
|
| Examples:
|
| Get a Dataset from a list containing the examples:
|
| ```py
| >>> ds = Dataset.from_list([{"text": "hello there !"}, {"text": "general kenobi !"}]})
| ```
|
| Pass features to set the column types, e.g. for an image dataset:
|
| ```py
| >>> features = Features({"image": Image()})
| >>> ds = Dataset.from_list([{"image": "path/to/image.png"}], features=features)
| ```
|
| Datasets are based on Arrow which is a columnar format, and therefore they expect every example to have the same
| type and subtypes, and dictionaries to have the same keys and values types.
| Loading a dataset errors out when fields have mismatching types, and fills missing fields in dictionaries with None so all dictionaries have the same keys and value types.
|
| To avoid this and allow mixed-types without errors, you can use `on_mixed_types="use_json"` or specify `features=` with a [`Json`] type:
|
| ```py
| >>> ds = Dataset.from_list([{"a": 0}, {"a": "foo"}, {"a": {"subfield": "bar"}}])
| Traceback (most recent call last):
| ...
| File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
| pyarrow.lib.ArrowInvalid: Could not convert 'foo' with type str: tried to convert to int64
|
| >>> ds = Dataset.from_list([{"a": 0}, {"a": "foo"}, {"a": {"subfield": "bar"}}], on_mixed_types="use_json")
| >>> ds.features
| {'a': Json()}
| >>> list(ds["a"])
| [0, "foo", {"subfield": "bar"}]
|
| >>> features = Features({"a": Json()})
| >>> ds = Dataset.from_list([{"a": 0}, {"a": "foo"}, {"a": {"subfield": "bar"}}], features=features)
| >>> ds.features
| {'a': Json()}
| >>> list(ds["a"])
| [0, "foo", {"subfield": "bar"}]
| ```
|
| This is also useful for lists of dictionaries with arbitrary keys and values, to avoid filling missing fields with None:
|
| ```py
| >>> ds = Dataset.from_list([{"a": [{"b": 0}, {"c": 0}]}])
| >>> ds.features
| {'a': List({'b': Value('int64'), 'c': Value('int64')})}
| >>> list(ds["a"])
| [[{'b': 0, 'c': None}, {'b': None, 'c': 0}]] # missing fields are filled with None
|
| >>> features = Features({"a": List(Json())})
| >>> ds = Dataset.from_list([{"a": [{"b": 0}, {"c": 0}]}], features=features)
| >>> ds.features
| {'a': List(Json())}
| >>> list(ds["a"])
| [[{'b': 0}, {'c': 0}]] # OK
|
| >>> ds = Dataset.from_list([{"a": [{"b": 0}, {"c": 0}]}], on_mixed_types="use_json")
| >>> ds.features
| {'a': List(Json())}
| >>> list(ds["a"])
| [[{'b': 0}, {'c': 0}]] # OK
| ```
|
| Another example with tool calling data:
|
| ```py
| >>> messages = [
| ... {"role": "user", "content": "Turn on the living room lights and play my electronic music playlist."},
| ... {"role": "assistant", "tool_calls": [
| ... {"type": "function", "function": {
| ... "name": "control_light",
| ... "arguments": {"room": "living room", "state": "on"}
| ... }},
| ... {"type": "function", "function": {
| ... "name": "play_music",
| ... "arguments": {"playlist": "electronic"} # mixed-type here since keys ["playlist"] and ["room", "state"] are different
| ... }}]
| ... },
| ... {"role": "tool", "name": "control_light", "content": "The lights in the living room are now on."},
| ... {"role": "tool", "name": "play_music", "content": "The music is now playing."},
| ... {"role": "assistant", "content": "Done!"}
| ... ]
| >>> ds = Dataset.from_list([{"messages": messages}], on_mixed_types="use_json")
| >>> ds.features
| {'messages': List({'role': Value('string'), 'content': Value('string'), 'tool_calls': List(Json()), 'name': Value('string')})}
| >>> ds[0]["messages"][1]["tool_calls"][0]["function"]["arguments"]
| {"room": "living room", "state": "on"}
| ```
|
| from_pandas(df: pandas.core.frame.DataFrame, features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, preserve_index: Optional[bool] = None) -> 'Dataset'
| Convert `pandas.DataFrame` to a `pyarrow.Table` to create a [`Dataset`].
|
| The column types in the resulting Arrow Table are inferred from the dtypes of the `pandas.Series` in the
| DataFrame. In the case of non-object Series, the NumPy dtype is translated to its Arrow equivalent. In the
| case of `object`, we need to guess the datatype by looking at the Python objects in this Series.
|
| Be aware that Series of the `object` dtype don't carry enough information to always lead to a meaningful Arrow
| type. In the case that we cannot infer a type, e.g. because the DataFrame is of length 0 or the Series only
| contains `None/nan` objects, the type is set to `null`. This behavior can be avoided by constructing explicit
| features and passing it to this function.
|
| Important: a dataset created with from_pandas() lives in memory
| and therefore doesn't have an associated cache directory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. save_to_disk / load_from_disk.
|
| Args:
| df (`pandas.DataFrame`):
| Dataframe that contains the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| preserve_index (`bool`, *optional*):
| Whether to store the index as an additional column in the resulting Dataset.
| The default of `None` will store the index as a column, except for `RangeIndex` which is stored as metadata only.
| Use `preserve_index=True` to force it to be stored as a column.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_pandas(df)
| ```
|
| from_polars(df: Union[ForwardRef('pl.DataFrame'), ForwardRef('pl.LazyFrame')], features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None) -> 'Dataset'
| Collect the underlying arrow arrays in an Arrow Table.
|
| This operation is mostly zero copy.
|
| Data types that do copy:
| * CategoricalType
|
| Args:
| df (`polars.DataFrame`): DataFrame to convert to Arrow Table
| features (`Features`, optional): Dataset features.
| info (`DatasetInfo`, optional): Dataset information, like description, citation, etc.
| split (`NamedSplit`, optional): Name of the dataset split.
|
| Examples:
| ```py
| >>> ds = Dataset.from_polars(df)
| ```
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| from_csv(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, num_proc: Optional[int] = None, **kwargs) -> 'Dataset'
| Create Dataset from CSV file(s).
|
| Read the CSV files, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the CSV file(s).
| split ([`NamedSplit`], *optional*):
| Split name to be assigned to the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`pandas.read_csv`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_csv('path/to/dataset.csv')
| ```
|
| from_generator(generator: Callable, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, gen_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, split: datasets.splits.NamedSplit = NamedSplit('train'), fingerprint: Optional[str] = None, **kwargs) -> 'Dataset'
| Create a Dataset from a generator.
|
| Load the data from the generator, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| generator (:`Callable`):
| A generator function that `yields` examples.
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| gen_kwargs(`dict`, *optional*):
| Keyword arguments to be passed to the `generator` callable.
| You can define a sharded dataset by passing the list of shards in `gen_kwargs` and setting `num_proc` greater than 1.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
| If `num_proc` is greater than one, then all list values in `gen_kwargs` must be the same length. These values will be split between calls to the generator. The number of shards will be the minimum of the shortest list in `gen_kwargs` and `num_proc`.
|
| <Added version="2.7.0"/>
| split ([`NamedSplit`], defaults to `Split.TRAIN`):
| Split name to be assigned to the dataset.
|
| <Added version="2.21.0"/>
| fingerprint (`str`, *optional*):
| Fingerprint that will be used to generate dataset ID.
| By default `fingerprint` is generated by hashing the generator function and all the args which can be slow
| if it uses large objects like AI models.
|
| <Added version="4.3.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to :[`GeneratorConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> def gen():
| ... yield {"text": "Good", "label": 0}
| ... yield {"text": "Bad", "label": 1}
| ...
| >>> ds = Dataset.from_generator(gen)
| ```
|
| ```py
| >>> def gen(shards):
| ... for shard in shards:
| ... with open(shard) as f:
| ... for line in f:
| ... yield {"line": line}
| ...
| >>> shards = [f"data{i}.txt" for i in range(32)]
| >>> ds = Dataset.from_generator(gen, gen_kwargs={"shards": shards})
| ```
|
| from_json(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, field: Optional[str] = None, num_proc: Optional[int] = None, **kwargs) -> 'Dataset'
| Create Dataset from JSON or JSON Lines file(s).
|
| Read the JSON files, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the JSON or JSON Lines file(s).
| split ([`NamedSplit`], *optional*):
| Split name to be assigned to the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| field (`str`, *optional*):
| Field name of the JSON file where the dataset is contained in.
| num_proc (`int`, *optional* defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`JsonConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_json('path/to/dataset.json')
| ```
|
| from_parquet(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, columns: Optional[list[str]] = None, num_proc: Optional[int] = None, filters: Union[pyarrow._compute.Expression, list[tuple], list[list[tuple]], NoneType] = None, fragment_scan_options: Optional[pyarrow._dataset_parquet.ParquetFragmentScanOptions] = None, on_bad_files: Literal['error', 'warn', 'skip'] = 'error', **kwargs) -> 'Dataset'
| Create Dataset from Parquet file(s).
|
| Read the Parquet files, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the Parquet file(s).
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| columns (`List[str]`, *optional*):
| If not `None`, only these columns will be read from the file.
| A column name may be a prefix of a nested field, e.g. 'a' will select
| 'a.b', 'a.c', and 'a.d.e'.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| filters (`Union[pyarrow.dataset.Expression, list[tuple], list[list[tuple]]]`, *optional*):
| Return only the rows matching the filter.
| If possible the predicate will be pushed down to exploit the partition information
| or internal metadata found in the data source, e.g. Parquet statistics.
| Otherwise filters the loaded RecordBatches before yielding them.
| fragment_scan_options (`pyarrow.dataset.ParquetFragmentScanOptions`, *optional*)
| Scan-specific options for Parquet fragments.
| This is especially useful to configure buffering and caching.
|
| <Added version="4.2.0"/>
| on_bad_files (`Literal["error", "warn", "skip"]`, *optional*, defaults to "error")
| Specify what to do upon encountering a bad file (a file that can't be read). Allowed values are :
| * 'error', raise an Exception when a bad file is encountered.
| * 'warn', raise a warning when a bad file is encountered and skip that file.
| * 'skip', skip bad files without raising or warning when they are encountered.
|
| <Added version="4.2.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`ParquetConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_parquet('path/to/dataset.parquet')
| ```
|
| Load a subset of columns:
|
| ```python
| >>> ds = Dataset.from_parquet('path/to/dataset.parquet', columns=["col_0", "col_1"])
| ```
|
| Efficiently filter data, possibly skipping entire files or row groups:
|
| ```python
| >>> filters = [("col_0", "==", 0)]
| >>> ds = Dataset.from_parquet(parquet_files_list, filters=filters)
| ```
|
| from_spark(df: 'pyspark.sql.DataFrame', split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, cache_dir: str = None, working_dir: str = None, load_from_cache_file: bool = True, **kwargs) -> 'Dataset'
| Create a Dataset from Spark DataFrame. Dataset downloading is distributed over Spark workers.
|
| Read the Spark DataFrame, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| df (`pyspark.sql.DataFrame`):
| The DataFrame containing the desired data.
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data. When using a multi-node Spark cluster, the cache_dir must be accessible to both
| workers and the driver.
| keep_in_memory (`bool`):
| Whether to copy the data in-memory.
| working_dir (`str`, *optional*)
| Intermediate directory for each Spark worker to write data to before moving it to `cache_dir`. Setting
| a non-NFS intermediate directory may improve performance.
| load_from_cache_file (`bool`):
| Whether to load the dataset from the cache if possible.
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> df = spark.createDataFrame(
| >>> data=[[1, "Elia"], [2, "Teo"], [3, "Fang"]],
| >>> columns=["id", "name"],
| >>> )
| >>> ds = Dataset.from_spark(df)
| ```
|
| from_sql(sql: Union[str, ForwardRef('sqlalchemy.sql.Selectable')], con: Union[str, ForwardRef('sqlalchemy.engine.Connection'), ForwardRef('sqlalchemy.engine.Engine'), ForwardRef('sqlite3.Connection')], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, **kwargs) -> 'Dataset'
| Create Dataset from SQL query or database table.
|
| Query the SQL database, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| sql (`str` or `sqlalchemy.sql.Selectable`):
| SQL query to be executed or a table name.
| con (`str` or `sqlite3.Connection` or `sqlalchemy.engine.Connection` or `sqlalchemy.engine.Connection`):
| A [URI string](https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls) used to instantiate a database connection or a SQLite3/SQLAlchemy connection object.
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`SqlConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> # Fetch a database table
| >>> ds = Dataset.from_sql("test_data", "postgres:///db_name")
| >>> # Execute a SQL query on the table
| >>> ds = Dataset.from_sql("SELECT sentence FROM test_data", "postgres:///db_name")
| >>> # Use a Selectable object to specify the query
| >>> from sqlalchemy import select, text
| >>> stmt = select([text("sentence")]).select_from(text("test_data"))
| >>> ds = Dataset.from_sql(stmt, "postgres:///db_name")
| ```
|
| > [!TIP]
| > The returned dataset can only be cached if `con` is specified as URI string.
|
| from_text(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, num_proc: Optional[int] = None, keep_linebreaks: bool = False, sample_by: Literal['line', 'paragraph', 'document'] = 'line', **kwargs) -> 'Dataset'
| Create Dataset from text file(s).
|
| Read the text files, cache the data in Arrow format on disk and return the Dataset from the memory-mapped Arrow data on disk.
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the text file(s).
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| This is helpful if the dataset is made of multiple files. Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| keep_linebreaks: (`bool`, defaults to False):
| Whether to keep line breaks.
| sample_by (`Literal["line", "paragraph", "document"]`, defaults to "line"):
| Whether to load data per line, praragraph or document.
| By default one row in the dataset = one line.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`TextConfig`].
|
| Returns:
| [`Dataset`]
|
| Example:
|
| ```py
| >>> ds = Dataset.from_text('path/to/dataset.txt')
| ```
|
| load_from_disk(dataset_path: Union[str, bytes, os.PathLike], keep_in_memory: Optional[bool] = None, storage_options: Optional[dict] = None) -> 'Dataset'
| Loads a dataset that was previously saved using [`save_to_disk`] from a dataset directory, or from a
| filesystem using any implementation of `fsspec.spec.AbstractFileSystem`.
|
| Args:
| dataset_path (`path-like`):
| Path (e.g. `"dataset/train"`) or remote URI (e.g. `"s3//my-bucket/dataset/train"`)
| of the dataset directory where the dataset will be loaded from.
| keep_in_memory (`bool`, defaults to `None`):
| Whether to copy the dataset in-memory. If `None`, the
| dataset will not be copied in-memory unless explicitly enabled by setting
| `datasets.config.IN_MEMORY_MAX_SIZE` to nonzero. See more details in the
| [improve performance](../cache#improve-performance) section.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.8.0"/>
|
| Returns:
| [`Dataset`] or [`DatasetDict`]:
| - If `dataset_path` is a path of a dataset directory, the dataset requested.
| - If `dataset_path` is a path of a dataset dict directory, a `datasets.DatasetDict` with each split.
|
| Example:
|
| ```py
| >>> ds = load_from_disk("path/to/dataset/directory")
| ```
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| cache_files
| The cache files containing the Apache Arrow table backing the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.cache_files
| [{'filename': '/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46/rotten_tomatoes_movie_review-validation.arrow'}]
| ```
|
| column_names
| Names of the columns in the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.column_names
| ['text', 'label']
| ```
|
| data
| The Apache Arrow table backing the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.data
| MemoryMappedTable
| text: string
| label: int64
| ----
| text: [["compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .","the soundtrack alone is worth the price of admission .","rodriguez does a splendid job of racial profiling hollywood style--casting excellent latin actors of all ages--a trend long overdue .","beneath the film's obvious determination to shock at any cost lies considerable skill and determination , backed by sheer nerve .","bielinsky is a filmmaker of impressive talent .","so beautifully acted and directed , it's clear that washington most certainly has a new career ahead of him if he so chooses .","a visual spectacle full of stunning images and effects .","a gentle and engrossing character study .","it's enough to watch huppert scheming , with her small , intelligent eyes as steady as any noir villain , and to enjoy the perfectly pitched web of tension that chabrol spins .","an engrossing portrait of uncompromising artists trying to create something original against the backdrop of a corporate music industry that only seems to care about the bottom line .",...,"ultimately , jane learns her place as a girl , softens up and loses some of the intensity that made her an interesting character to begin with .","ah-nuld's action hero days might be over .","it's clear why deuces wild , which was shot two years ago , has been gathering dust on mgm's shelf .","feels like nothing quite so much as a middle-aged moviemaker's attempt to surround himself with beautiful , half-naked women .","when the precise nature of matthew's predicament finally comes into sharp focus , the revelation fails to justify the build-up .","this picture is murder by numbers , and as easy to be bored by as your abc's , despite a few whopping shootouts .","hilarious musical comedy though stymied by accents thick as mud .","if you are into splatter movies , then you will probably have a reasonably good time with the salton sea .","a dull , simple-minded and stereotypical tale of drugs , death and mind-numbing indifference on the inner-city streets .","the feature-length stretch . . . strains the show's concept ."]]
| label: [[1,1,1,1,1,1,1,1,1,1,...,0,0,0,0,0,0,0,0,0,0]]
| ```
|
| features
|
| format
|
| num_columns
| Number of columns in the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.num_columns
| 2
| ```
|
| num_rows
| Number of rows in the dataset (same as [`Dataset.__len__`]).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.num_rows
| 1066
| ```
|
| shape
| Shape of the dataset (number of columns, number of rows).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.shape
| (1066, 2)
| ```
|
| ----------------------------------------------------------------------
| Readonly properties inherited from DatasetInfoMixin:
|
| builder_name
|
| citation
|
| config_name
|
| dataset_size
|
| description
|
| download_checksums
|
| download_size
|
| homepage
|
| info
| [`~datasets.DatasetInfo`] object containing all the metadata in the dataset.
|
| license
|
| size_in_bytes
|
| split
| [`~datasets.NamedSplit`] object corresponding to a named dataset split.
|
| supervised_keys
|
| version
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DatasetInfoMixin:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from datasets.search.IndexableMixin:
|
| drop_index(self, index_name: str)
| Drop the index with the specified column.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index.
|
| get_index(self, index_name: str) -> datasets.search.BaseIndex
| List the `index_name`/identifiers of all the attached indexes.
|
| Args:
| index_name (`str`): Index name.
|
| Returns:
| [`BaseIndex`]
|
| get_nearest_examples(self, index_name: str, query: Union[str, <built-in function array>], k: int = 10, **kwargs) -> datasets.search.NearestExamplesResults
| Find the nearest examples in the dataset to the query.
|
| Args:
| index_name (`str`):
| The index_name/identifier of the index.
| query (`Union[str, np.ndarray]`):
| The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve.
|
| Returns:
| `(scores, examples)`:
| A tuple of `(scores, examples)` where:
| - **scores** (`List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples
| - **examples** (`dict`): the retrieved examples
|
| get_nearest_examples_batch(self, index_name: str, queries: Union[list[str], <built-in function array>], k: int = 10, **kwargs) -> datasets.search.BatchedNearestExamplesResults
| Find the nearest examples in the dataset to the query.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index.
| queries (`Union[List[str], np.ndarray]`):
| The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve per query.
|
| Returns:
| `(total_scores, total_examples)`:
| A tuple of `(total_scores, total_examples)` where:
| - **total_scores** (`List[List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples per query
| - **total_examples** (`List[dict]`): the retrieved examples per query
|
| is_index_initialized(self, index_name: str) -> bool
|
| list_indexes(self) -> list[str]
| List the `colindex_nameumns`/identifiers of all the attached indexes.
|
| load_elasticsearch_index(self, index_name: str, es_index_name: str, host: Optional[str] = None, port: Optional[int] = None, es_client: Optional[ForwardRef('Elasticsearch')] = None, es_index_config: Optional[dict] = None)
| Load an existing text index using ElasticSearch for fast retrieval.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index. This is the index name that is used to call `get_nearest` or `search`.
| es_index_name (`str`):
| The name of elasticsearch index to load.
| host (`str`, *optional*, defaults to `localhost`):
| Host of where ElasticSearch is running.
| port (`str`, *optional*, defaults to `9200`):
| Port of where ElasticSearch is running.
| es_client (`elasticsearch.Elasticsearch`, *optional*):
| The elasticsearch client used to create the index if host and port are `None`.
| es_index_config (`dict`, *optional*):
| The configuration of the elasticsearch index.
| Default config is:
| ```
| {
| "settings": {
| "number_of_shards": 1,
| "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},
| },
| "mappings": {
| "properties": {
| "text": {
| "type": "text",
| "analyzer": "standard",
| "similarity": "BM25"
| },
| }
| },
| }
| ```
|
| load_faiss_index(self, index_name: str, file: Union[str, pathlib.PurePath], device: Union[int, list[int], NoneType] = None, storage_options: Optional[dict] = None)
| Load a FaissIndex from disk.
|
| If you want to do additional configurations, you can have access to the faiss index object by doing
| `.get_index(index_name).faiss_index` to make it fit your needs.
|
| Args:
| index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to
| call `.get_nearest` or `.search`.
| file (`str`): The path to the serialized faiss index on disk or remote URI (e.g. `"s3://my-bucket/index.faiss"`).
| device (Optional `Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs.
| If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.11.0"/>
|
| save_faiss_index(self, index_name: str, file: Union[str, pathlib.PurePath], storage_options: Optional[dict] = None)
| Save a FaissIndex on disk.
|
| Args:
| index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`.
| file (`str`): The path to the serialized faiss index on disk or remote URI (e.g. `"s3://my-bucket/index.faiss"`).
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.11.0"/>
|
| search(self, index_name: str, query: Union[str, <built-in function array>], k: int = 10, **kwargs) -> datasets.search.SearchResults
| Find the nearest examples indices in the dataset to the query.
|
| Args:
| index_name (`str`):
| The name/identifier of the index.
| query (`Union[str, np.ndarray]`):
| The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve.
|
| Returns:
| `(scores, indices)`:
| A tuple of `(scores, indices)` where:
| - **scores** (`List[List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples
| - **indices** (`List[List[int]]`): the indices of the retrieved examples
|
| search_batch(self, index_name: str, queries: Union[list[str], <built-in function array>], k: int = 10, **kwargs) -> datasets.search.BatchedSearchResults
| Find the nearest examples indices in the dataset to the query.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index.
| queries (`Union[List[str], np.ndarray]`):
| The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve per query.
|
| Returns:
| `(total_scores, total_indices)`:
| A tuple of `(total_scores, total_indices)` where:
| - **total_scores** (`List[List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples per query
| - **total_indices** (`List[List[int]]`): the indices of the retrieved examples per query
|
| ----------------------------------------------------------------------
| Methods inherited from TensorflowDatasetMixin:
|
| to_tf_dataset(self, batch_size: Optional[int] = None, columns: Union[str, list[str], NoneType] = None, shuffle: bool = False, collate_fn: Optional[Callable] = None, drop_remainder: bool = False, collate_fn_args: Optional[dict[str, Any]] = None, label_cols: Union[str, list[str], NoneType] = None, prefetch: bool = True, num_workers: int = 0, num_test_batches: int = 20)
| Create a `tf.data.Dataset` from the underlying Dataset. This `tf.data.Dataset` will load and collate batches from
| the Dataset, and is suitable for passing to methods like `model.fit()` or `model.predict()`. The dataset will yield
| `dicts` for both inputs and labels unless the `dict` would contain only a single key, in which case a raw
| `tf.Tensor` is yielded instead.
|
| Args:
| batch_size (`int`, *optional*):
| Size of batches to load from the dataset. Defaults to `None`, which implies that the dataset won't be
| batched, but the returned dataset can be batched later with `tf_dataset.batch(batch_size)`.
| columns (`List[str]` or `str`, *optional*):
| Dataset column(s) to load in the `tf.data.Dataset`.
| Column names that are created by the `collate_fn` and that do not exist in the original dataset can be used.
| shuffle(`bool`, defaults to `False`):
| Shuffle the dataset order when loading. Recommended `True` for training, `False` for
| validation/evaluation.
| drop_remainder(`bool`, defaults to `False`):
| Drop the last incomplete batch when loading. Ensures
| that all batches yielded by the dataset will have the same length on the batch dimension.
| collate_fn(`Callable`, *optional*):
| A function or callable object (such as a `DataCollator`) that will collate
| lists of samples into a batch.
| collate_fn_args (`Dict`, *optional*):
| An optional `dict` of keyword arguments to be passed to the
| `collate_fn`.
| label_cols (`List[str]` or `str`, defaults to `None`):
| Dataset column(s) to load as labels.
| Note that many models compute loss internally rather than letting Keras do it, in which case
| passing the labels here is optional, as long as they're in the input `columns`.
| prefetch (`bool`, defaults to `True`):
| Whether to run the dataloader in a separate thread and maintain
| a small buffer of batches for training. Improves performance by allowing data to be loaded in the
| background while the model is training.
| num_workers (`int`, defaults to `0`):
| Number of workers to use for loading the dataset.
| num_test_batches (`int`, defaults to `20`):
| Number of batches to use to infer the output signature of the dataset.
| The higher this number, the more accurate the signature will be, but the longer it will take to
| create the dataset.
|
| Returns:
| `tf.data.Dataset`
|
| Example:
|
| ```py
| >>> ds_train = ds["train"].to_tf_dataset(
| ... columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'],
| ... shuffle=True,
| ... batch_size=16,
| ... collate_fn=data_collator,
| ... )
| ```
class DatasetInfoMixin(builtins.object)
| DatasetInfoMixin(info: datasets.info.DatasetInfo, split: Optional[datasets.splits.NamedSplit])
|
| This base class exposes some attributes of DatasetInfo
| at the base level of the Dataset for easy access.
|
| Methods defined here:
|
| __init__(self, info: datasets.info.DatasetInfo, split: Optional[datasets.splits.NamedSplit])
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| builder_name
|
| citation
|
| config_name
|
| dataset_size
|
| description
|
| download_checksums
|
| download_size
|
| features
|
| homepage
|
| info
| [`~datasets.DatasetInfo`] object containing all the metadata in the dataset.
|
| license
|
| size_in_bytes
|
| split
| [`~datasets.NamedSplit`] object corresponding to a named dataset split.
|
| supervised_keys
|
| version
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class DatasetTransformationNotAllowedError(builtins.Exception)
| Method resolution order:
| DatasetTransformationNotAllowedError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NonExistentDatasetError(builtins.Exception)
| Used when we expect the existence of a dataset
|
| Method resolution order:
| NonExistentDatasetError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class TensorflowDatasetMixin(builtins.object)
| Methods defined here:
|
| to_tf_dataset(self, batch_size: Optional[int] = None, columns: Union[str, list[str], NoneType] = None, shuffle: bool = False, collate_fn: Optional[Callable] = None, drop_remainder: bool = False, collate_fn_args: Optional[dict[str, Any]] = None, label_cols: Union[str, list[str], NoneType] = None, prefetch: bool = True, num_workers: int = 0, num_test_batches: int = 20)
| Create a `tf.data.Dataset` from the underlying Dataset. This `tf.data.Dataset` will load and collate batches from
| the Dataset, and is suitable for passing to methods like `model.fit()` or `model.predict()`. The dataset will yield
| `dicts` for both inputs and labels unless the `dict` would contain only a single key, in which case a raw
| `tf.Tensor` is yielded instead.
|
| Args:
| batch_size (`int`, *optional*):
| Size of batches to load from the dataset. Defaults to `None`, which implies that the dataset won't be
| batched, but the returned dataset can be batched later with `tf_dataset.batch(batch_size)`.
| columns (`List[str]` or `str`, *optional*):
| Dataset column(s) to load in the `tf.data.Dataset`.
| Column names that are created by the `collate_fn` and that do not exist in the original dataset can be used.
| shuffle(`bool`, defaults to `False`):
| Shuffle the dataset order when loading. Recommended `True` for training, `False` for
| validation/evaluation.
| drop_remainder(`bool`, defaults to `False`):
| Drop the last incomplete batch when loading. Ensures
| that all batches yielded by the dataset will have the same length on the batch dimension.
| collate_fn(`Callable`, *optional*):
| A function or callable object (such as a `DataCollator`) that will collate
| lists of samples into a batch.
| collate_fn_args (`Dict`, *optional*):
| An optional `dict` of keyword arguments to be passed to the
| `collate_fn`.
| label_cols (`List[str]` or `str`, defaults to `None`):
| Dataset column(s) to load as labels.
| Note that many models compute loss internally rather than letting Keras do it, in which case
| passing the labels here is optional, as long as they're in the input `columns`.
| prefetch (`bool`, defaults to `True`):
| Whether to run the dataloader in a separate thread and maintain
| a small buffer of batches for training. Improves performance by allowing data to be loaded in the
| background while the model is training.
| num_workers (`int`, defaults to `0`):
| Number of workers to use for loading the dataset.
| num_test_batches (`int`, defaults to `20`):
| Number of batches to use to infer the output signature of the dataset.
| The higher this number, the more accurate the signature will be, but the longer it will take to
| create the dataset.
|
| Returns:
| `tf.data.Dataset`
|
| Example:
|
| ```py
| >>> ds_train = ds["train"].to_tf_dataset(
| ... columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'],
| ... shuffle=True,
| ... batch_size=16,
| ... collate_fn=data_collator,
| ... )
| ```
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
async async_get_indices_from_mask_function(function: Callable, batched: bool, with_indices: bool, with_rank: bool, input_columns: Union[str, list[str], NoneType], indices_mapping: Optional[datasets.table.Table] = None, *args, **fn_kwargs)
same function but async
ceil(x, /)
Return the ceiling of x as an Integral.
This is the smallest integer >= x.
floor(x, /)
Return the floor of x as an Integral.
This is the largest integer <= x.
get_indices_from_mask_function(function: Callable, batched: bool, with_indices: bool, with_rank: bool, input_columns: Union[str, list[str], NoneType], indices_mapping: Optional[datasets.table.Table] = None, *args, **fn_kwargs)
transmit_format(func)
Wrapper for dataset transforms that recreate a new Dataset to transmit the format of the original dataset to the new dataset
update_metadata_with_features(table: datasets.table.Table, features: datasets.features.features.Features)
To be used in dataset transforms that modify the features of the dataset, in order to update the features stored in the metadata of its schema.
DATA
Callable = typing.Callable
Deprecated alias to collections.abc.Callable.
Callable[[int], str] signifies a function that takes a single
parameter of type int and returns a str.
The subscription syntax must always be used with exactly two
values: the argument list and the return type.
The argument list must be a list of types, a ParamSpec,
Concatenate or ellipsis. The return type must be a single type.
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
FeatureType = typing.Union[dict, list, tuple, datasets.feature....feat...
ListLike = typing.Union[list[~T], tuple[~T, ...]]
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
PUSH_TO_HUB_WITHOUT_METADATA_CONFIGS_SPLIT_PATTERN_SHARDED = 'data/{sp...
PathLike = typing.Union[str, bytes, os.PathLike]
TYPE_CHECKING = False
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.arrow_dataset (WARNING)>
VERSION
5.0.1
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\arrow_dataset.py
datasets.arrow_reader
Python Library Documentation: module datasets.arrow_reader in datasets
NAME
datasets.arrow_reader - Arrow ArrowReader.
CLASSES
builtins.ConnectionError(builtins.OSError)
DatasetNotOnHfGcsError
MissingFilesOnHfGcsError
builtins.object
BaseReader
ArrowReader
ParquetReader
FileInstructions
ReadInstruction
class ArrowReader(BaseReader)
| ArrowReader(path: str, info: Optional[ForwardRef('DatasetInfo')])
|
| Build a Dataset object out of Instruction instance(s).
| This Reader uses either memory mapping or file descriptors (in-memory) on arrow files.
|
| Method resolution order:
| ArrowReader
| BaseReader
| builtins.object
|
| Methods defined here:
|
| __init__(self, path: str, info: Optional[ForwardRef('DatasetInfo')])
| Initializes ArrowReader.
|
| Args:
| path (str): path where Arrow files are stored.
| info (DatasetInfo): info about the dataset.
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| read_table(filename, in_memory=False) -> datasets.table.Table
| Read table from file.
|
| Args:
| filename (str): File name of the table.
| in_memory (bool, default=False): Whether to copy the data in-memory.
|
| Returns:
| pyarrow.Table
|
| ----------------------------------------------------------------------
| Methods inherited from BaseReader:
|
| get_file_instructions(self, name, instruction, split_infos)
| Return list of dict {'filename': str, 'skip': int, 'take': int}
|
| read(self, name, instructions, split_infos, in_memory=False)
| Returns Dataset instance(s).
|
| Args:
| name (str): name of the dataset.
| instructions (ReadInstruction): instructions to read.
| Instruction can be string and will then be passed to the Instruction
| constructor as it.
| split_infos (list of SplitInfo proto): the available splits for dataset.
| in_memory (bool, default False): Whether to copy the data in-memory.
|
| Returns:
| kwargs to build a single Dataset instance.
|
| read_files(self, files: list[dict], original_instructions: Union[NoneType, ForwardRef('ReadInstruction'), ForwardRef('Split')] = None, in_memory=False)
| Returns single Dataset instance for the set of file instructions.
|
| Args:
| files: List[dict(filename, skip, take)], the files information.
| The filenames contains the relative path, not absolute.
| skip/take indicates which example read in the file: `ds.skip().take()`
| original_instructions: store the original instructions used to build the dataset split in the dataset.
| in_memory (bool, default False): Whether to copy the data in-memory.
|
| Returns:
| kwargs to build a Dataset instance.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseReader:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class BaseReader(builtins.object)
| BaseReader(path: str, info: Optional[ForwardRef('DatasetInfo')])
|
| Build a Dataset object out of Instruction instance(s).
|
| Methods defined here:
|
| __init__(self, path: str, info: Optional[ForwardRef('DatasetInfo')])
| Initializes ArrowReader.
|
| Args:
| path (str): path where tfrecords are stored.
| info (DatasetInfo): info about the dataset.
|
| get_file_instructions(self, name, instruction, split_infos)
| Return list of dict {'filename': str, 'skip': int, 'take': int}
|
| read(self, name, instructions, split_infos, in_memory=False)
| Returns Dataset instance(s).
|
| Args:
| name (str): name of the dataset.
| instructions (ReadInstruction): instructions to read.
| Instruction can be string and will then be passed to the Instruction
| constructor as it.
| split_infos (list of SplitInfo proto): the available splits for dataset.
| in_memory (bool, default False): Whether to copy the data in-memory.
|
| Returns:
| kwargs to build a single Dataset instance.
|
| read_files(self, files: list[dict], original_instructions: Union[NoneType, ForwardRef('ReadInstruction'), ForwardRef('Split')] = None, in_memory=False)
| Returns single Dataset instance for the set of file instructions.
|
| Args:
| files: List[dict(filename, skip, take)], the files information.
| The filenames contains the relative path, not absolute.
| skip/take indicates which example read in the file: `ds.skip().take()`
| original_instructions: store the original instructions used to build the dataset split in the dataset.
| in_memory (bool, default False): Whether to copy the data in-memory.
|
| Returns:
| kwargs to build a Dataset instance.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class DatasetNotOnHfGcsError(builtins.ConnectionError)
| When you can't get the dataset from the Hf google cloud storage
|
| Method resolution order:
| DatasetNotOnHfGcsError
| builtins.ConnectionError
| builtins.OSError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.ConnectionError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.OSError:
|
| __reduce__(...)
| Helper for pickle.
|
| __str__(self, /)
| Return str(self).
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.OSError:
|
| __new__(*args, **kwargs) class method of builtins.OSError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.OSError:
|
| characters_written
|
| errno
| POSIX exception code
|
| filename
| exception filename
|
| filename2
| second exception filename
|
| strerror
| exception strerror
|
| winerror
| Win32 exception code
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class FileInstructions(builtins.object)
| FileInstructions(num_examples: int, file_instructions: list[dict]) -> None
|
| The file instructions associated with a split ReadInstruction.
|
| Attributes:
| num_examples: `int`, The total number of examples
| file_instructions: List[dict(filename, skip, take)], the files information.
| The filenames contains the relative path, not absolute.
| skip/take indicates which example read in the file: `ds.slice(skip, take)`
|
| Methods defined here:
|
| __delattr__(self, name)
| Implement delattr(self, name).
|
| __eq__(self, other)
| Return self==value.
|
| __hash__(self)
| Return hash(self).
|
| __init__(self, num_examples: int, file_instructions: list[dict]) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| __setattr__(self, name, value)
| Implement setattr(self, name, value).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'file_instructions': list[dict], 'num_examples': <c...
|
| __dataclass_fields__ = {'file_instructions': Field(name='file_instruct...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __match_args__ = ('num_examples', 'file_instructions')
class MissingFilesOnHfGcsError(builtins.ConnectionError)
| When some files are missing on the Hf oogle cloud storage
|
| Method resolution order:
| MissingFilesOnHfGcsError
| builtins.ConnectionError
| builtins.OSError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.ConnectionError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.OSError:
|
| __reduce__(...)
| Helper for pickle.
|
| __str__(self, /)
| Return str(self).
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.OSError:
|
| __new__(*args, **kwargs) class method of builtins.OSError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.OSError:
|
| characters_written
|
| errno
| POSIX exception code
|
| filename
| exception filename
|
| filename2
| second exception filename
|
| strerror
| exception strerror
|
| winerror
| Win32 exception code
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ParquetReader(BaseReader)
| ParquetReader(path: str, info: Optional[ForwardRef('DatasetInfo')])
|
| Build a Dataset object out of Instruction instance(s).
| This Reader uses memory mapping on parquet files.
|
| Method resolution order:
| ParquetReader
| BaseReader
| builtins.object
|
| Methods defined here:
|
| __init__(self, path: str, info: Optional[ForwardRef('DatasetInfo')])
| Initializes ParquetReader.
|
| Args:
| path (str): path where tfrecords are stored.
| info (DatasetInfo): info about the dataset.
|
| ----------------------------------------------------------------------
| Methods inherited from BaseReader:
|
| get_file_instructions(self, name, instruction, split_infos)
| Return list of dict {'filename': str, 'skip': int, 'take': int}
|
| read(self, name, instructions, split_infos, in_memory=False)
| Returns Dataset instance(s).
|
| Args:
| name (str): name of the dataset.
| instructions (ReadInstruction): instructions to read.
| Instruction can be string and will then be passed to the Instruction
| constructor as it.
| split_infos (list of SplitInfo proto): the available splits for dataset.
| in_memory (bool, default False): Whether to copy the data in-memory.
|
| Returns:
| kwargs to build a single Dataset instance.
|
| read_files(self, files: list[dict], original_instructions: Union[NoneType, ForwardRef('ReadInstruction'), ForwardRef('Split')] = None, in_memory=False)
| Returns single Dataset instance for the set of file instructions.
|
| Args:
| files: List[dict(filename, skip, take)], the files information.
| The filenames contains the relative path, not absolute.
| skip/take indicates which example read in the file: `ds.skip().take()`
| original_instructions: store the original instructions used to build the dataset split in the dataset.
| in_memory (bool, default False): Whether to copy the data in-memory.
|
| Returns:
| kwargs to build a Dataset instance.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseReader:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class ReadInstruction(builtins.object)
| ReadInstruction(split_name, rounding=None, from_=None, to=None, unit=None)
|
| Reading instruction for a dataset.
|
| Examples::
|
| # The following lines are equivalent:
| ds = datasets.load_dataset('ylecun/mnist', split='test[:33%]')
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction.from_spec('test[:33%]'))
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction('test', to=33, unit='%'))
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction(
| 'test', from_=0, to=33, unit='%'))
|
| # The following lines are equivalent:
| ds = datasets.load_dataset('ylecun/mnist', split='test[:33%]+train[1:-1]')
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction.from_spec(
| 'test[:33%]+train[1:-1]'))
| ds = datasets.load_dataset('ylecun/mnist', split=(
| datasets.ReadInstruction('test', to=33, unit='%') +
| datasets.ReadInstruction('train', from_=1, to=-1, unit='abs')))
|
| # The following lines are equivalent:
| ds = datasets.load_dataset('ylecun/mnist', split='test[:33%](pct1_dropremainder)')
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction.from_spec(
| 'test[:33%](pct1_dropremainder)'))
| ds = datasets.load_dataset('ylecun/mnist', split=datasets.ReadInstruction(
| 'test', from_=0, to=33, unit='%', rounding="pct1_dropremainder"))
|
| # 10-fold validation:
| tests = datasets.load_dataset(
| 'ylecun/mnist',
| [datasets.ReadInstruction('train', from_=k, to=k+10, unit='%')
| for k in range(0, 100, 10)])
| trains = datasets.load_dataset(
| 'ylecun/mnist',
| [datasets.ReadInstruction('train', to=k, unit='%') + datasets.ReadInstruction('train', from_=k+10, unit='%')
| for k in range(0, 100, 10)])
|
| Methods defined here:
|
| __add__(self, other)
| Returns a new ReadInstruction obj, result of appending other to self.
|
| __init__(self, split_name, rounding=None, from_=None, to=None, unit=None)
| Initialize ReadInstruction.
|
| Args:
| split_name (str): name of the split to read. Eg: 'train'.
| rounding (str, optional): The rounding behaviour to use when percent slicing is
| used. Ignored when slicing with absolute indices.
| Possible values:
| - 'closest' (default): The specified percentages are rounded to the
| closest value. Use this if you want specified percents to be as
| much exact as possible.
| - 'pct1_dropremainder': the specified percentages are treated as
| multiple of 1%. Use this option if you want consistency. Eg:
| len(5%) == 5 * len(1%).
| Using this option, one might not be able to use the full set of
| examples, if the number of those is not a multiple of 100.
| from_ (int):
| to (int): alternative way of specifying slicing boundaries. If any of
| {from_, to, unit} argument is used, slicing cannot be specified as
| string.
| unit (str): optional, one of:
| '%': to set the slicing unit as percents of the split size.
| 'abs': to set the slicing unit as absolute numbers.
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| to_absolute(self, name2len)
| Translate instruction into a list of absolute instructions.
|
| Those absolute instructions are then to be added together.
|
| Args:
| name2len (`dict`):
| Associating split names to number of examples.
|
| Returns:
| list of _AbsoluteInstruction instances (corresponds to the + in spec).
|
| to_spec(self)
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_spec(spec)
| Creates a `ReadInstruction` instance out of a string spec.
|
| Args:
| spec (`str`):
| Split(s) + optional slice(s) to read + optional rounding
| if percents are used as the slicing unit. A slice can be specified,
| using absolute numbers (`int`) or percentages (`int`).
|
| Examples:
|
| ```
| test: test split.
| test + validation: test split + validation split.
| test[10:]: test split, minus its first 10 records.
| test[:10%]: first 10% records of test split.
| test[:20%](pct1_dropremainder): first 10% records, rounded with the pct1_dropremainder rounding.
| test[:-5%]+train[40%:60%]: first 95% of test + middle 20% of train.
| ```
|
| Returns:
| ReadInstruction instance.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
make_file_instructions(name: str, split_infos: list['SplitInfo'], instruction: Union[str, ForwardRef('ReadInstruction')], filetype_suffix: Optional[str] = None, prefix_path: Optional[str] = None) -> datasets.arrow_reader.FileInstructions
Returns instructions of the split dict.
Args:
name (`str`): Name of the dataset.
split_infos (`list` of `[SplitInfo]`): Dataset splits information.
instruction ([`ReadInstruction`] or `str`): Reading instruction for a dataset.
filetype_suffix (`str`, *optional*): Suffix of dataset files, e.g. 'arrow' or 'parquet'.
prefix_path (`str`, *optional*): Prefix of dataset files, e.g. directory name.
Returns:
[`FileInstructions`]
DATA
HF_GCP_BASE_URL = 'https://storage.googleapis.com/huggingface-nlp/cach...
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
TYPE_CHECKING = False
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.arrow_reader (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\arrow_reader.py
datasets.arrow_writer
Python Library Documentation: module datasets.arrow_writer in datasets
NAME
datasets.arrow_writer - To write records into Parquet files.
CLASSES
builtins.ValueError(builtins.Exception)
SchemaInferenceError
builtins.object
ArrowWriter
ParquetWriter
TypedSequence
OptimizedTypedSequence
class ArrowWriter(builtins.object)
| ArrowWriter(schema: Optional[pyarrow.lib.Schema] = None, features: Optional[datasets.features.features.Features] = None, path: Optional[str] = None, stream: Optional[pyarrow.lib.NativeFile] = None, fingerprint: Optional[str] = None, writer_batch_size: Optional[int] = None, disable_nullable: bool = False, update_features: bool = False, on_mixed_types: Optional[Literal['use_json']] = 'use_json', with_metadata: bool = True, unit: str = 'examples', embed_local_files: bool = False, storage_options: Optional[dict] = None)
|
| Shuffles and writes Examples to Arrow files.
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| __init__(self, schema: Optional[pyarrow.lib.Schema] = None, features: Optional[datasets.features.features.Features] = None, path: Optional[str] = None, stream: Optional[pyarrow.lib.NativeFile] = None, fingerprint: Optional[str] = None, writer_batch_size: Optional[int] = None, disable_nullable: bool = False, update_features: bool = False, on_mixed_types: Optional[Literal['use_json']] = 'use_json', with_metadata: bool = True, unit: str = 'examples', embed_local_files: bool = False, storage_options: Optional[dict] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __len__(self)
| Return the number of writed and staged examples
|
| close(self)
|
| finalize(self, close_stream=True)
|
| write(self, example: dict[str, typing.Any], writer_batch_size: Optional[int] = None)
| Add a given (Example,Key) pair to the write-pool of examples which is written to file.
|
| Args:
| example: the Example to add.
|
| write_batch(self, batch_examples: dict[str, list], writer_batch_size: Optional[int] = None, try_original_type: Optional[bool] = True)
| Write a batch of Example to file.
| Ignores the batch if it appears to be empty,
| preventing a potential schema update of unknown types.
|
| Args:
| batch_examples: the batch of examples to add.
| try_original_type: use `try_type` when instantiating OptimizedTypedSequence if `True`, otherwise `try_type = None`.
|
| write_examples_on_file(self)
| Write stored examples from the write-pool of examples. It makes a table out of the examples and write it.
|
| write_row(self, row: pyarrow.lib.Table, writer_batch_size: Optional[int] = None)
| Add a given single-row Table to the write-pool of rows which is written to file.
|
| Args:
| row: the row to add.
|
| write_rows_on_file(self)
| Write stored rows from the write-pool of rows. It concatenates the single-row tables and it writes the resulting table.
|
| write_table(self, pa_table: pyarrow.lib.Table, writer_batch_size: Optional[int] = None)
| Write a Table to file.
|
| Args:
| example: the Table to add.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| schema
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class OptimizedTypedSequence(TypedSequence)
| OptimizedTypedSequence(data, type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, try_type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, col: Optional[str] = None, optimized_int_type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, on_mixed_types: Optional[Literal['use_json']] = None)
|
| Method resolution order:
| OptimizedTypedSequence
| TypedSequence
| builtins.object
|
| Methods defined here:
|
| __init__(self, data, type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, try_type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, col: Optional[str] = None, optimized_int_type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, on_mixed_types: Optional[Literal['use_json']] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from TypedSequence:
|
| __arrow_array__(self, type: Optional[pyarrow.lib.DataType] = None)
|
| get_inferred_type(self) -> Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti]
| Return the inferred feature type.
| This is done by converting the sequence to an Arrow array, and getting the corresponding
| feature type.
|
| Since building the Arrow array can be expensive, the value of the inferred type is cached
| as soon as pa.array is called on the typed sequence.
|
| Returns:
| FeatureType: inferred feature type of the sequence.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from TypedSequence:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class ParquetWriter(ArrowWriter)
| ParquetWriter(*args, use_content_defined_chunking=True, write_page_index=True, **kwargs)
|
| Method resolution order:
| ParquetWriter
| ArrowWriter
| builtins.object
|
| Methods defined here:
|
| __init__(self, *args, use_content_defined_chunking=True, write_page_index=True, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from ArrowWriter:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| __len__(self)
| Return the number of writed and staged examples
|
| close(self)
|
| finalize(self, close_stream=True)
|
| write(self, example: dict[str, typing.Any], writer_batch_size: Optional[int] = None)
| Add a given (Example,Key) pair to the write-pool of examples which is written to file.
|
| Args:
| example: the Example to add.
|
| write_batch(self, batch_examples: dict[str, list], writer_batch_size: Optional[int] = None, try_original_type: Optional[bool] = True)
| Write a batch of Example to file.
| Ignores the batch if it appears to be empty,
| preventing a potential schema update of unknown types.
|
| Args:
| batch_examples: the batch of examples to add.
| try_original_type: use `try_type` when instantiating OptimizedTypedSequence if `True`, otherwise `try_type = None`.
|
| write_examples_on_file(self)
| Write stored examples from the write-pool of examples. It makes a table out of the examples and write it.
|
| write_row(self, row: pyarrow.lib.Table, writer_batch_size: Optional[int] = None)
| Add a given single-row Table to the write-pool of rows which is written to file.
|
| Args:
| row: the row to add.
|
| write_rows_on_file(self)
| Write stored rows from the write-pool of rows. It concatenates the single-row tables and it writes the resulting table.
|
| write_table(self, pa_table: pyarrow.lib.Table, writer_batch_size: Optional[int] = None)
| Write a Table to file.
|
| Args:
| example: the Table to add.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from ArrowWriter:
|
| schema
|
| ----------------------------------------------------------------------
| Data descriptors inherited from ArrowWriter:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class SchemaInferenceError(builtins.ValueError)
| Method resolution order:
| SchemaInferenceError
| builtins.ValueError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.ValueError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.ValueError:
|
| __new__(*args, **kwargs) class method of builtins.ValueError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class TypedSequence(builtins.object)
| TypedSequence(data: collections.abc.Iterable, type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, try_type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, optimized_int_type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, on_mixed_types: Optional[Literal['use_json']] = None)
|
| This data container generalizes the typing when instantiating pyarrow arrays, tables or batches.
|
| More specifically it adds several features:
| - Support extension types like ``datasets.features.Array2DExtensionType``:
| By default pyarrow arrays don't return extension arrays. One has to call
| ``pa.ExtensionArray.from_storage(type, pa.array(data, type.storage_type))``
| in order to get an extension array.
| - Support for ``try_type`` parameter that can be used instead of ``type``:
| When an array is transformed, we like to keep the same type as before if possible.
| For example when calling :func:`datasets.Dataset.map`, we don't want to change the type
| of each column by default.
| - Better error message when a pyarrow array overflows.
|
| Example::
|
| from datasets.features import Array2D, Array2DExtensionType, Value
| from datasets.arrow_writer import TypedSequence
| import pyarrow as pa
|
| arr = pa.array(TypedSequence([1, 2, 3], type=Value("int32")))
| assert arr.type == pa.int32()
|
| arr = pa.array(TypedSequence([1, 2, 3], try_type=Value("int32")))
| assert arr.type == pa.int32()
|
| arr = pa.array(TypedSequence(["foo", "bar"], try_type=Value("int32")))
| assert arr.type == pa.string()
|
| arr = pa.array(TypedSequence([[[1, 2, 3]]], type=Array2D((1, 3), "int64")))
| assert arr.type == Array2DExtensionType((1, 3), "int64")
|
| table = pa.Table.from_pydict({
| "image": TypedSequence([[[1, 2, 3]]], type=Array2D((1, 3), "int64"))
| })
| assert table["image"].type == Array2DExtensionType((1, 3), "int64")
|
| Methods defined here:
|
| __arrow_array__(self, type: Optional[pyarrow.lib.DataType] = None)
|
| __init__(self, data: collections.abc.Iterable, type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, try_type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, optimized_int_type: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti, NoneType] = None, on_mixed_types: Optional[Literal['use_json']] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| get_inferred_type(self) -> Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti]
| Return the inferred feature type.
| This is done by converting the sequence to an Arrow array, and getting the corresponding
| feature type.
|
| Since building the Arrow array can be expensive, the value of the inferred type is cached
| as soon as pa.array is called on the typed sequence.
|
| Returns:
| FeatureType: inferred feature type of the sequence.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
get_arrow_writer_batch_size_from_features(features: Optional[datasets.features.features.Features]) -> Optional[int]
Get the writer_batch_size that defines the maximum record batch size in the arrow files based on configuration values.
The default value is 100 for image/audio datasets and 10 for videos.
This allows to avoid overflows in arrow buffers.
Args:
features (`datasets.Features` or `None`):
Dataset Features from `datasets`.
Returns:
writer_batch_size (`Optional[int]`):
Writer batch size to pass to a dataset builder.
If `None`, then it will use the `datasets` default, i.e. `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
get_writer_batch_size_from_data_size(num_rows: int, num_bytes: int) -> int
Get the writer_batch_size that defines the maximum row group size in the parquet files.
The default in `datasets` is aiming for row groups of maximum 100MB uncompressed.
This allows to optimize random access to parquet file, since accessing 1 row requires
to read its entire row group.
This can be improved to get optimized size for querying/iterating
but at least it matches the dataset viewer expectations on HF.
Args:
num_rows (`int`):
Number of rows in the dataset.
num_bytes (`int`):
Number of bytes in the dataset.
For dataset with external files to embed (image, audio, videos), this can also be an
estimate from `dataset._estimate_nbytes()`.
Returns:
writer_batch_size (`Optional[int]`):
Writer batch size to pass to a parquet writer.
get_writer_batch_size_from_features(features: Optional[datasets.features.features.Features]) -> Optional[int]
Get the writer_batch_size that defines the maximum row group size in the parquet files based on configuration values.
By default these are not set, but it can be helpful to hard set those values in some cases.
This allows to optimize random access to parquet file, since accessing 1 row requires
to read its entire row group.
Args:
features (`datasets.Features` or `None`):
Dataset Features from `datasets`.
Returns:
writer_batch_size (`Optional[int]`):
Writer batch size to pass to a parquet writer.
If `None`, then it will use the `datasets` default, i.e. aiming for row groups of 100MB.
DATA
FeatureType = typing.Union[dict, list, tuple, datasets.feature....feat...
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
logger = <Logger datasets.arrow_writer (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\arrow_writer.py
datasets.builder
Python Library Documentation: module datasets.builder in datasets
NAME
datasets.builder - DatasetBuilder base class.
CLASSES
builtins.ValueError(builtins.Exception)
InvalidConfigName
builtins.object
BuilderConfig
DatasetBuilder
ArrowBasedBuilder
GeneratorBasedBuilder
Key
class ArrowBasedBuilder(DatasetBuilder)
| ArrowBasedBuilder(cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
|
| Base class for datasets with data generation based on Arrow loading functions (CSV/JSON/Parquet).
|
| Method resolution order:
| ArrowBasedBuilder
| DatasetBuilder
| builtins.object
|
| Methods inherited from DatasetBuilder:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setstate__(self, d)
|
| as_dataset(self, split: Union[str, datasets.splits.Split, list[str], list[datasets.splits.Split], NoneType] = None, in_memory=False) -> Union[datasets.arrow_dataset.Dataset, datasets.dataset_dict.DatasetDict]
| Return a Dataset for the specified split.
|
| Args:
| split (`datasets.Split`):
| Which subset of the data to return.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the
| downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
|
| Returns:
| datasets.Dataset
|
| Example:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')
| >>> builder.download_and_prepare()
| >>> ds = builder.as_dataset(split='train')
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 8530
| })
| ```
|
| as_streaming_dataset(self, split: Optional[str] = None, base_path: Optional[str] = None) -> Union[dict[str, datasets.iterable_dataset.IterableDataset], datasets.iterable_dataset.IterableDataset]
|
| download_and_prepare(self, output_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, verification_mode: Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None, dl_manager: Optional[datasets.download.download_manager.DownloadManager] = None, base_path: Optional[str] = None, file_format: str = 'arrow', max_shard_size: Union[str, int, NoneType] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **download_and_prepare_kwargs)
| Downloads and prepares dataset for reading.
|
| Args:
| output_dir (`str`, *optional*):
| Output directory for the dataset.
| Default to this builder's `cache_dir`, which is inside `~/.cache/huggingface/datasets` by default.
|
| <Added version="2.5.0"/>
| download_config (`DownloadConfig`, *optional*):
| Specific download configuration parameters.
| download_mode ([`DownloadMode`] or `str`, *optional*):
| Select the download/generate mode, default to `REUSE_DATASET_IF_EXISTS`.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| dl_manager (`DownloadManager`, *optional*):
| Specific `DownloadManger` to use.
| base_path (`str`, *optional*):
| Base path for relative paths that are used to download files. This can be a remote url.
| If not specified, the value of the `base_path` attribute (`self.base_path`) will be used instead.
| file_format (`str`, *optional*):
| Format of the data files in which the dataset will be written.
| Supported formats: "arrow", "parquet". Default to "arrow" format.
| If the format is "parquet", then image and audio data are embedded into the Parquet files instead of pointing to local files.
|
| <Added version="2.5.0"/>
| max_shard_size (`Union[str, int]`, *optional*):
| Maximum number of bytes written per shard, default is "500MB".
| The size is based on uncompressed data size, so in practice your shard files may be smaller than
| `max_shard_size` thanks to Parquet compression for example.
|
| <Added version="2.5.0"/>
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.7.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the caching file-system backend, if any.
|
| <Added version="2.5.0"/>
| **download_and_prepare_kwargs (additional keyword arguments): Keyword arguments.
|
| Example:
|
| Download and prepare the dataset as Arrow files that can be loaded as a Dataset using `builder.as_dataset()`:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare()
| ```
|
| Download and prepare the dataset as sharded Parquet files locally:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("./output_dir", file_format="parquet")
| ```
|
| Download and prepare the dataset as sharded Parquet files in a cloud storage:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> storage_options = {"key": aws_access_key_id, "secret": aws_secret_access_key}
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("s3://my-bucket/my_rotten_tomatoes", storage_options=storage_options, file_format="parquet")
| ```
|
| ----------------------------------------------------------------------
| Class methods inherited from DatasetBuilder:
|
| get_imported_module_dir()
| Return the path of the module of this class or subclass.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from DatasetBuilder:
|
| builder_configs
| Dictionary of pre-defined configurations for this builder class.
|
| cache_dir
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DatasetBuilder:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from DatasetBuilder:
|
| BUILDER_CONFIGS = []
|
| BUILDER_CONFIG_CLASS = <class 'datasets.builder.BuilderConfig'>
| Base class for `DatasetBuilder` data configuration.
|
| `DatasetBuilder` subclasses with data configuration options should subclass
| `BuilderConfig` and add their own properties.
|
| Attributes:
| name (`str`, defaults to `default`):
| The name of the configuration.
| version (`Version` or `str`, defaults to `0.0.0`):
| The version of the configuration.
| data_dir (`str`, *optional*):
| Path to the directory containing the source data.
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| description (`str`, *optional*):
| A human description of the configuration.
|
|
| DEFAULT_CONFIG_NAME = None
|
| DEFAULT_WRITER_BATCH_SIZE = None
|
| SLEEP_ON_THREADS_SHUTDOWNS = False
|
| VERSION = None
class BuilderConfig(builtins.object)
| BuilderConfig(name: str = 'default', version: Union[datasets.utils.version.Version, str, NoneType] = 0.0.0, data_dir: Optional[str] = None, data_files: Union[datasets.data_files.DataFilesDict, datasets.data_files.DataFilesPatternsDict, NoneType] = None, description: Optional[str] = None) -> None
|
| Base class for `DatasetBuilder` data configuration.
|
| `DatasetBuilder` subclasses with data configuration options should subclass
| `BuilderConfig` and add their own properties.
|
| Attributes:
| name (`str`, defaults to `default`):
| The name of the configuration.
| version (`Version` or `str`, defaults to `0.0.0`):
| The version of the configuration.
| data_dir (`str`, *optional*):
| Path to the directory containing the source data.
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| description (`str`, *optional*):
| A human description of the configuration.
|
| Methods defined here:
|
| __eq__(self, o)
| Return self==value.
|
| __init__(self, name: str = 'default', version: Union[datasets.utils.version.Version, str, NoneType] = 0.0.0, data_dir: Optional[str] = None, data_files: Union[datasets.data_files.DataFilesDict, datasets.data_files.DataFilesPatternsDict, NoneType] = None, description: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| create_config_id(self, config_kwargs: dict, custom_features: Optional[datasets.features.features.Features] = None) -> str
| The config id is used to build the cache directory.
| By default it is equal to the config name.
| However the name of a config is not sufficient to have a unique identifier for the dataset being generated
| since it doesn't take into account:
| - the config kwargs that can be used to overwrite attributes
| - the custom features used to write the dataset
| - the data_files for json/text/csv/pandas datasets
|
| Therefore the config id is just the config name with an optional suffix based on these.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'data_dir': typing.Optional[str], 'data_files': typ...
|
| __dataclass_fields__ = {'data_dir': Field(name='data_dir',type=typing....
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('name', 'version', 'data_dir', 'data_files', 'descri...
|
| data_dir = None
|
| data_files = None
|
| description = None
|
| name = 'default'
|
| version = 0.0.0
class DatasetBuilder(builtins.object)
| DatasetBuilder(cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
|
| Abstract base class for all datasets.
|
| `DatasetBuilder` has 3 key methods:
|
| - [`DatasetBuilder.info`]: Documents the dataset, including feature
| names, types, shapes, version, splits, citation, etc.
| - [`DatasetBuilder.download_and_prepare`]: Downloads the source data
| and writes it to disk.
| - [`DatasetBuilder.as_dataset`]: Generates a [`Dataset`].
|
| Some `DatasetBuilder`s expose multiple variants of the
| dataset by defining a [`BuilderConfig`] subclass and accepting a
| config object (or name) on construction. Configurable datasets expose a
| pre-defined set of configurations in [`DatasetBuilder.builder_configs`].
|
| Args:
| cache_dir (`str`, *optional*):
| Directory to cache data. Defaults to `"~/.cache/huggingface/datasets"`.
| dataset_name (`str`, *optional*):
| Name of the dataset, if different from the builder name. Useful for packaged builders
| like csv, imagefolder, audiofolder, etc. to reflect the difference between datasets
| that use the same packaged builder.
| config_name (`str`, *optional*):
| Name of the dataset configuration.
| It affects the data generated on disk. Different configurations will have their own subdirectories and
| versions.
| If not provided, the default configuration is used (if it exists).
|
| <Added version="2.3.0">
|
| Parameter `name` was renamed to `config_name`.
|
| </Added>
| hash (`str`, *optional*):
| Hash specific to the dataset builder code. Used to update the caching directory when the
| dataset builder code is updated (to avoid reusing old data).
| The typical caching directory (defined in `self._relative_data_dir`) is `name/version/hash/`.
| base_path (`str`, *optional*):
| Base path for relative paths that are used to download files.
| This can be a remote URL.
| features ([`Features`], *optional*):
| Features types to use with this dataset.
| It can be used to change the [`Features`] types of a dataset, for example.
| token (`str` or `bool`, *optional*):
| String or boolean to use as Bearer token for remote files on the
| Datasets Hub. If `True`, will get token from `"~/.huggingface"`.
| repo_id (`str`, *optional*):
| ID of the dataset repository.
| Used to distinguish builders with the same name but not coming from the same namespace, for example "rajpurkar/squad"
| and "lhoestq/squad" repo IDs. In the latter, the builder name would be "lhoestq___squad".
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| For builders like "csv" or "json" that need the user to specify data files. They can be either
| local or remote files. For convenience, you can use a `DataFilesDict`.
| data_dir (`str`, *optional*):
| Path to directory containing source data file(s).
| Use only if `data_files` is not passed, in which case it is equivalent to passing
| `os.path.join(data_dir, "**")` as `data_files`.
| For builders that require manual download, it must be the path to the local directory containing the
| manually downloaded data.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the dataset file-system backend, if any.
| writer_batch_size (`int`, *optional*):
| Batch size used by the ArrowWriter.
| It defines the number of samples that are kept in memory before writing them
| and also the length of the arrow chunks.
| None means that the ArrowWriter will use its default value.
| **config_kwargs (additional keyword arguments): Keyword arguments to be passed to the corresponding builder
| configuration class, set on the class attribute [`DatasetBuilder.BUILDER_CONFIG_CLASS`]. The builder
| configuration class is [`BuilderConfig`] or a subclass of it.
|
| Methods defined here:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setstate__(self, d)
|
| as_dataset(self, split: Union[str, datasets.splits.Split, list[str], list[datasets.splits.Split], NoneType] = None, in_memory=False) -> Union[datasets.arrow_dataset.Dataset, datasets.dataset_dict.DatasetDict]
| Return a Dataset for the specified split.
|
| Args:
| split (`datasets.Split`):
| Which subset of the data to return.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the
| downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
|
| Returns:
| datasets.Dataset
|
| Example:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')
| >>> builder.download_and_prepare()
| >>> ds = builder.as_dataset(split='train')
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 8530
| })
| ```
|
| as_streaming_dataset(self, split: Optional[str] = None, base_path: Optional[str] = None) -> Union[dict[str, datasets.iterable_dataset.IterableDataset], datasets.iterable_dataset.IterableDataset]
|
| download_and_prepare(self, output_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, verification_mode: Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None, dl_manager: Optional[datasets.download.download_manager.DownloadManager] = None, base_path: Optional[str] = None, file_format: str = 'arrow', max_shard_size: Union[str, int, NoneType] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **download_and_prepare_kwargs)
| Downloads and prepares dataset for reading.
|
| Args:
| output_dir (`str`, *optional*):
| Output directory for the dataset.
| Default to this builder's `cache_dir`, which is inside `~/.cache/huggingface/datasets` by default.
|
| <Added version="2.5.0"/>
| download_config (`DownloadConfig`, *optional*):
| Specific download configuration parameters.
| download_mode ([`DownloadMode`] or `str`, *optional*):
| Select the download/generate mode, default to `REUSE_DATASET_IF_EXISTS`.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| dl_manager (`DownloadManager`, *optional*):
| Specific `DownloadManger` to use.
| base_path (`str`, *optional*):
| Base path for relative paths that are used to download files. This can be a remote url.
| If not specified, the value of the `base_path` attribute (`self.base_path`) will be used instead.
| file_format (`str`, *optional*):
| Format of the data files in which the dataset will be written.
| Supported formats: "arrow", "parquet". Default to "arrow" format.
| If the format is "parquet", then image and audio data are embedded into the Parquet files instead of pointing to local files.
|
| <Added version="2.5.0"/>
| max_shard_size (`Union[str, int]`, *optional*):
| Maximum number of bytes written per shard, default is "500MB".
| The size is based on uncompressed data size, so in practice your shard files may be smaller than
| `max_shard_size` thanks to Parquet compression for example.
|
| <Added version="2.5.0"/>
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.7.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the caching file-system backend, if any.
|
| <Added version="2.5.0"/>
| **download_and_prepare_kwargs (additional keyword arguments): Keyword arguments.
|
| Example:
|
| Download and prepare the dataset as Arrow files that can be loaded as a Dataset using `builder.as_dataset()`:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare()
| ```
|
| Download and prepare the dataset as sharded Parquet files locally:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("./output_dir", file_format="parquet")
| ```
|
| Download and prepare the dataset as sharded Parquet files in a cloud storage:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> storage_options = {"key": aws_access_key_id, "secret": aws_secret_access_key}
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("s3://my-bucket/my_rotten_tomatoes", storage_options=storage_options, file_format="parquet")
| ```
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| get_imported_module_dir()
| Return the path of the module of this class or subclass.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| builder_configs
| Dictionary of pre-defined configurations for this builder class.
|
| cache_dir
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| BUILDER_CONFIGS = []
|
| BUILDER_CONFIG_CLASS = <class 'datasets.builder.BuilderConfig'>
| Base class for `DatasetBuilder` data configuration.
|
| `DatasetBuilder` subclasses with data configuration options should subclass
| `BuilderConfig` and add their own properties.
|
| Attributes:
| name (`str`, defaults to `default`):
| The name of the configuration.
| version (`Version` or `str`, defaults to `0.0.0`):
| The version of the configuration.
| data_dir (`str`, *optional*):
| Path to the directory containing the source data.
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| description (`str`, *optional*):
| A human description of the configuration.
|
|
| DEFAULT_CONFIG_NAME = None
|
| DEFAULT_WRITER_BATCH_SIZE = None
|
| SLEEP_ON_THREADS_SHUTDOWNS = False
|
| VERSION = None
class GeneratorBasedBuilder(DatasetBuilder)
| GeneratorBasedBuilder(cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
|
| Base class for datasets with data generation based on dict generators.
|
| `GeneratorBasedBuilder` is a convenience class that abstracts away much
| of the data writing and reading of `DatasetBuilder`. It expects subclasses to
| implement generators of feature dictionaries across the dataset splits
| (`_split_generators`). See the method docstrings for details.
|
| Method resolution order:
| GeneratorBasedBuilder
| DatasetBuilder
| builtins.object
|
| Methods inherited from DatasetBuilder:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, cache_dir: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, hash: Optional[str] = None, base_path: Optional[str] = None, info: Optional[datasets.info.DatasetInfo] = None, features: Optional[datasets.features.features.Features] = None, token: Union[bool, str, NoneType] = None, repo_id: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, data_dir: Optional[str] = None, storage_options: Optional[dict] = None, writer_batch_size: Optional[int] = None, config_id: Optional[str] = None, **config_kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setstate__(self, d)
|
| as_dataset(self, split: Union[str, datasets.splits.Split, list[str], list[datasets.splits.Split], NoneType] = None, in_memory=False) -> Union[datasets.arrow_dataset.Dataset, datasets.dataset_dict.DatasetDict]
| Return a Dataset for the specified split.
|
| Args:
| split (`datasets.Split`):
| Which subset of the data to return.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the
| downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
|
| Returns:
| datasets.Dataset
|
| Example:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')
| >>> builder.download_and_prepare()
| >>> ds = builder.as_dataset(split='train')
| >>> ds
| Dataset({
| features: ['text', 'label'],
| num_rows: 8530
| })
| ```
|
| as_streaming_dataset(self, split: Optional[str] = None, base_path: Optional[str] = None) -> Union[dict[str, datasets.iterable_dataset.IterableDataset], datasets.iterable_dataset.IterableDataset]
|
| download_and_prepare(self, output_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, verification_mode: Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None, dl_manager: Optional[datasets.download.download_manager.DownloadManager] = None, base_path: Optional[str] = None, file_format: str = 'arrow', max_shard_size: Union[str, int, NoneType] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **download_and_prepare_kwargs)
| Downloads and prepares dataset for reading.
|
| Args:
| output_dir (`str`, *optional*):
| Output directory for the dataset.
| Default to this builder's `cache_dir`, which is inside `~/.cache/huggingface/datasets` by default.
|
| <Added version="2.5.0"/>
| download_config (`DownloadConfig`, *optional*):
| Specific download configuration parameters.
| download_mode ([`DownloadMode`] or `str`, *optional*):
| Select the download/generate mode, default to `REUSE_DATASET_IF_EXISTS`.
| verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
| Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/...).
|
| <Added version="2.9.1"/>
| dl_manager (`DownloadManager`, *optional*):
| Specific `DownloadManger` to use.
| base_path (`str`, *optional*):
| Base path for relative paths that are used to download files. This can be a remote url.
| If not specified, the value of the `base_path` attribute (`self.base_path`) will be used instead.
| file_format (`str`, *optional*):
| Format of the data files in which the dataset will be written.
| Supported formats: "arrow", "parquet". Default to "arrow" format.
| If the format is "parquet", then image and audio data are embedded into the Parquet files instead of pointing to local files.
|
| <Added version="2.5.0"/>
| max_shard_size (`Union[str, int]`, *optional*):
| Maximum number of bytes written per shard, default is "500MB".
| The size is based on uncompressed data size, so in practice your shard files may be smaller than
| `max_shard_size` thanks to Parquet compression for example.
|
| <Added version="2.5.0"/>
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.7.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the caching file-system backend, if any.
|
| <Added version="2.5.0"/>
| **download_and_prepare_kwargs (additional keyword arguments): Keyword arguments.
|
| Example:
|
| Download and prepare the dataset as Arrow files that can be loaded as a Dataset using `builder.as_dataset()`:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare()
| ```
|
| Download and prepare the dataset as sharded Parquet files locally:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("./output_dir", file_format="parquet")
| ```
|
| Download and prepare the dataset as sharded Parquet files in a cloud storage:
|
| ```py
| >>> from datasets import load_dataset_builder
| >>> storage_options = {"key": aws_access_key_id, "secret": aws_secret_access_key}
| >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
| >>> builder.download_and_prepare("s3://my-bucket/my_rotten_tomatoes", storage_options=storage_options, file_format="parquet")
| ```
|
| ----------------------------------------------------------------------
| Class methods inherited from DatasetBuilder:
|
| get_imported_module_dir()
| Return the path of the module of this class or subclass.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from DatasetBuilder:
|
| builder_configs
| Dictionary of pre-defined configurations for this builder class.
|
| cache_dir
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DatasetBuilder:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from DatasetBuilder:
|
| BUILDER_CONFIGS = []
|
| BUILDER_CONFIG_CLASS = <class 'datasets.builder.BuilderConfig'>
| Base class for `DatasetBuilder` data configuration.
|
| `DatasetBuilder` subclasses with data configuration options should subclass
| `BuilderConfig` and add their own properties.
|
| Attributes:
| name (`str`, defaults to `default`):
| The name of the configuration.
| version (`Version` or `str`, defaults to `0.0.0`):
| The version of the configuration.
| data_dir (`str`, *optional*):
| Path to the directory containing the source data.
| data_files (`str` or `Sequence` or `Mapping`, *optional*):
| Path(s) to source data file(s).
| description (`str`, *optional*):
| A human description of the configuration.
|
|
| DEFAULT_CONFIG_NAME = None
|
| DEFAULT_WRITER_BATCH_SIZE = None
|
| SLEEP_ON_THREADS_SHUTDOWNS = False
|
| VERSION = None
class InvalidConfigName(builtins.ValueError)
| Method resolution order:
| InvalidConfigName
| builtins.ValueError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.ValueError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.ValueError:
|
| __new__(*args, **kwargs) class method of builtins.ValueError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class Key(builtins.object)
| Key(original_shard_id: int, item_or_batch_id: int) -> None
|
| Key(original_shard_id: int, item_or_batch_id: int)
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, original_shard_id: int, item_or_batch_id: int) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'item_or_batch_id': <class 'int'>, 'original_shard_...
|
| __dataclass_fields__ = {'item_or_batch_id': Field(name='item_or_batch_...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('original_shard_id', 'item_or_batch_id')
DATA
INVALID_WINDOWS_CHARACTERS_IN_PATH = r'<>:/\|?*'
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
TYPE_CHECKING = False
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.builder (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\builder.py
datasets.combine
Python Library Documentation: module datasets.combine in datasets
NAME
datasets.combine
FUNCTIONS
concatenate_datasets(dsets: list[~DatasetType], info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, axis: int = 0) -> ~DatasetType
Concatenate several datasets (sources) into a single dataset.
Use axis=0 to concatenate vertically (default), or axis=1 to concatenate horizontally.
Note for iterable datasets:
* if axis=0, the resulting dataset's `num_shards` is the sum of each dataset's `num_shards`.
* if axis=1, the resulting dataset has one (1) shard to not misalign data.
Args:
dsets (`List[datasets.Dataset]` or `List[datasets.IterableDataset]`):
List of Datasets to concatenate.
info (`DatasetInfo`, *optional*):
Dataset information, like description, citation, etc.
split (`NamedSplit`, *optional*):
Name of the dataset split.
axis (`{0, 1}`, defaults to `0`):
Axis to concatenate over, where `0` means over rows (vertically) and `1` means over columns
(horizontally).
<Added version="1.6.0"/>
Example:
```py
>>> ds3 = concatenate_datasets([ds1, ds2])
```
interleave_datasets(datasets: list[~DatasetType], probabilities: Optional[list[float]] = None, seed: Optional[int] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, stopping_strategy: Literal['first_exhausted', 'all_exhausted', 'all_exhausted_without_replacement'] = 'first_exhausted') -> ~DatasetType
Interleave several datasets (sources) into a single dataset.
The new dataset is constructed by alternating between the sources to get the examples.
You can use this function on a list of [`Dataset`] objects, or on a list of [`IterableDataset`] objects.
- If `probabilities` is `None` (default) the new dataset is constructed by cycling between each source to get the examples.
- If `probabilities` is not `None`, the new dataset is constructed by getting examples from a random source at a time according to the provided probabilities.
The resulting dataset ends when one of the source datasets runs out of examples except when `oversampling` is `True`,
in which case, the resulting dataset ends when all datasets have ran out of examples at least one time.
Note for iterable datasets:
* The resulting dataset's `num_shards` is the minimum of each dataset's `num_shards` to ensure good parallelism.
If some of your datasets have a very low number of shards, you may use [`IterableDataset.reshard`].
* In a distributed setup or in PyTorch DataLoader workers, the stopping strategy is applied per process.
Therefore the "first_exhausted" strategy on an sharded iterable dataset can generate less samples in total (up to 1 missing sample per subdataset per worker).
Args:
datasets (`List[Dataset]` or `List[IterableDataset]`):
List of datasets to interleave.
probabilities (`List[float]`, *optional*, defaults to `None`):
If specified, the new dataset is constructed by sampling
examples from one source at a time according to these probabilities.
seed (`int`, *optional*, defaults to `None`):
The random seed used to choose a source for each example.
info ([`DatasetInfo`], *optional*):
Dataset information, like description, citation, etc.
<Added version="2.4.0"/>
split ([`NamedSplit`], *optional*):
Name of the dataset split.
<Added version="2.4.0"/>
stopping_strategy (`str`, defaults to `first_exhausted`):
Three strategies are proposed right now, `first_exhausted`, `all_exhausted` and `all_exhausted_without_replacement`.
By default, `first_exhausted` is an undersampling strategy, i.e the dataset construction is stopped as soon as one dataset has ran out of samples.
If the strategy is `all_exhausted`, we use an oversampling strategy, i.e the dataset construction is stopped as soon as every samples of every dataset has been added at least once.
When strategy is `all_exhausted_without_replacement` we make sure that each sample in each dataset is sampled only once.
Note that if the strategy is `all_exhausted`, the interleaved dataset size can get enormous:
- with no probabilities, the resulting dataset will have `max_length_datasets*nb_dataset` samples.
- with given probabilities, the resulting dataset will have more samples if some datasets have really low probability of visiting.
Returns:
[`Dataset`] or [`IterableDataset`]: Return type depends on the input `datasets`
parameter. `Dataset` if the input is a list of `Dataset`, `IterableDataset` if the input is a list of
`IterableDataset`.
Example:
For regular datasets (map-style):
```python
>>> from datasets import Dataset, interleave_datasets
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42, stopping_strategy="all_exhausted")
>>> dataset["a"]
[10, 0, 11, 1, 2, 20, 12, 10, 0, 1, 2, 21, 0, 11, 1, 2, 0, 1, 12, 2, 10, 0, 22]
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42)
>>> dataset["a"]
[10, 0, 11, 1, 2]
>>> dataset = interleave_datasets([d1, d2, d3])
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22, 23, 24]})
>>> dataset = interleave_datasets([d1, d2, d3])
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22, 0, 13, 23, 1, 10, 24]
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42)
>>> dataset["a"]
[10, 0, 11, 1, 2]
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42, stopping_strategy="all_exhausted")
>>> dataset["a"]
[10, 0, 11, 1, 2, 20, 12, 13, ..., 0, 1, 2, 0, 24]
For datasets in streaming mode (iterable):
>>> from datasets import interleave_datasets
>>> d1 = load_dataset('allenai/c4', 'es', split='train', streaming=True)
>>> d2 = load_dataset('allenai/c4', 'fr', split='train', streaming=True)
>>> dataset = interleave_datasets([d1, d2])
>>> iterator = iter(dataset)
>>> next(iterator)
{'text': 'Comprar Zapatillas para niña en chancla con goma por...'}
>>> next(iterator)
{'text': 'Le sacre de philippe ier, 23 mai 1059 - Compte Rendu...'
```
DATA
DatasetType = ~DatasetType
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
logger = <Logger datasets.combine (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\combine.py
datasets.concatenate_datasets
Python Library Documentation: function concatenate_datasets in module datasets.combine
concatenate_datasets(dsets: list[~DatasetType], info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, axis: int = 0) -> ~DatasetType
Concatenate several datasets (sources) into a single dataset.
Use axis=0 to concatenate vertically (default), or axis=1 to concatenate horizontally.
Note for iterable datasets:
* if axis=0, the resulting dataset's `num_shards` is the sum of each dataset's `num_shards`.
* if axis=1, the resulting dataset has one (1) shard to not misalign data.
Args:
dsets (`List[datasets.Dataset]` or `List[datasets.IterableDataset]`):
List of Datasets to concatenate.
info (`DatasetInfo`, *optional*):
Dataset information, like description, citation, etc.
split (`NamedSplit`, *optional*):
Name of the dataset split.
axis (`{0, 1}`, defaults to `0`):
Axis to concatenate over, where `0` means over rows (vertically) and `1` means over columns
(horizontally).
<Added version="1.6.0"/>
Example:
```py
>>> ds3 = concatenate_datasets([ds1, ds2])
```
datasets.config
Python Library Documentation: module datasets.config in datasets
NAME
datasets.config
DATA
ARCHIVES_MAX_NUMBER_FOR_MODULE_INFERENCE = 10
ARROW_READER_BATCH_SIZE_IN_DATASET_ITER = 10
ARROW_RECORD_BATCH_SIZE_FOR_AUDIO_DATASETS = 100
ARROW_RECORD_BATCH_SIZE_FOR_BINARY_DATASETS = 100
ARROW_RECORD_BATCH_SIZE_FOR_IMAGE_DATASETS = 100
ARROW_RECORD_BATCH_SIZE_FOR_VIDEO_DATASETS = 10
CLOUDFRONT_DATASETS_DISTRIB_PREFIX = 'https://cdn-datasets.huggingface...
DATASETDICT_INFOS_FILENAME = 'dataset_infos.json'
DATASETDICT_JSON_FILENAME = 'dataset_dict.json'
DATASET_ARROW_FILENAME = 'dataset.arrow'
DATASET_INDICES_FILENAME = 'indices.arrow'
DATASET_INFO_FILENAME = 'dataset_info.json'
DATASET_STATE_JSON_FILENAME = 'state.json'
DEFAULT_CDC_OPTIONS = {'max_chunk_size': 1048576, 'min_chunk_size': 26...
DEFAULT_DOWNLOADED_DATASETS_PATH = r'C:\Users\Youse\.cache\huggingface...
DEFAULT_EXTRACTED_DATASETS_PATH = r'C:\Users\Youse\.cache\huggingface\...
DEFAULT_HF_CACHE_HOME = r'~/.cache\huggingface'
DEFAULT_HF_DATASETS_CACHE = r'C:\Users\Youse/.cache\huggingface\datase...
DEFAULT_HF_MODULES_CACHE = r'C:\Users\Youse/.cache\huggingface\modules...
DEFAULT_IN_MEMORY_MAX_SIZE = 0
DEFAULT_MAX_BATCH_SIZE = 1000
DEFAULT_XDG_CACHE_HOME = '~/.cache'
DILL_VERSION = <Version('0.4.1')>
DOWNLOADED_DATASETS_DIR = 'downloads'
DOWNLOADED_DATASETS_PATH = WindowsPath('C:/Users/Youse/.cache/huggingf...
DUCKDB_AVAILABLE = True
DUCKDB_VERSION = <Version('1.5.5')>
ENV_VARS_FALSE_AND_AUTO_VALUES = {'0', 'AUTO', 'FALSE', 'NO', 'OFF'}
ENV_VARS_FALSE_VALUES = {'0', 'FALSE', 'NO', 'OFF'}
ENV_VARS_TRUE_AND_AUTO_VALUES = {'1', 'AUTO', 'ON', 'TRUE', 'YES'}
ENV_VARS_TRUE_VALUES = {'1', 'ON', 'TRUE', 'YES'}
EXTRACTED_DATASETS_DIR = 'extracted'
EXTRACTED_DATASETS_PATH = WindowsPath('C:/Users/Youse/.cache/huggingfa...
FSSPEC_VERSION = <Version('2026.6.0')>
HF_CACHE_HOME = r'C:\Users\Youse/.cache\huggingface'
HF_DATASETS_CACHE = WindowsPath('C:/Users/Youse/.cache/huggingface/dat...
HF_DATASETS_DISABLE_PROGRESS_BARS = None
HF_DATASETS_MULTITHREADING_MAX_WORKERS = 16
HF_DATASETS_OFFLINE = False
HF_ENDPOINT = 'https://huggingface.co'
HF_HUB_OFFLINE = False
HF_HUB_VERSION = <Version('1.26.0')>
HF_MODULES_CACHE = WindowsPath('C:/Users/Youse/.cache/huggingface/modu...
HF_UPDATE_DOWNLOAD_COUNTS = True
HUB_DATASETS_HFFS_URL = 'hf://datasets/{repo_id}@{revision}/{path}'
HUB_DATASETS_URL = 'https://huggingface.co/datasets/{repo_id}/resolve/...
HUB_DEFAULT_VERSION = 'main'
IN_MEMORY_MAX_SIZE = 0.0
IS_MP3_SUPPORTED = True
IS_OPUS_SUPPORTED = True
JAX_AVAILABLE = False
JAX_VERSION = 'N/A'
LICENSE_FILENAME = 'LICENSE'
LZ4_AVAILABLE = False
MAX_DATASET_CONFIG_ID_READABLE_LENGTH = 255
MAX_NUM_RUNNING_ASYNC_MAP_FUNCTIONS_IN_PARALLEL = 1000
MAX_ROW_GROUP_SIZE = '100MB'
MAX_SHARD_SIZE = '500MB'
MAX_TABLE_NBYTES_FOR_PICKLING = 4294967296
METADATA_CONFIGS_FIELD = 'configs'
MODULE_NAME_FOR_DYNAMIC_MODULES = 'datasets_modules'
NIBABEL_AVAILABLE = False
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
PANDAS_VERSION = <Version('2.2.2')>
PARQUET_ROW_GROUP_SIZE_FOR_AUDIO_DATASETS = None
PARQUET_ROW_GROUP_SIZE_FOR_BINARY_DATASETS = None
PARQUET_ROW_GROUP_SIZE_FOR_IMAGE_DATASETS = None
PARQUET_ROW_GROUP_SIZE_FOR_VIDEO_DATASETS = None
PBAR_REFRESH_TIME_INTERVAL = 0.05
PDFPLUMBER_AVAILABLE = True
PIL_AVAILABLE = True
POLARS_AVAILABLE = True
POLARS_VERSION = <Version('1.43.1')>
PY7ZR_AVAILABLE = False
PYARROW_VERSION = <Version('24.0.0')>
PY_VERSION = <Version('3.12.6')>
RARFILE_AVAILABLE = False
REPOCARD_FILENAME = 'README.md'
REPOYAML_FILENAME = '.huggingface.yaml'
REPO_DATASETS_URL = 'https://raw.githubusercontent.com/huggingface/dat...
S3_DATASETS_BUCKET_PREFIX = 'https://s3.amazonaws.com/datasets.hugging...
SAVE_ORIGINAL_SHARD_LENGTHS = False
SLEEP_TIME_ON_THREADS_SHUTDOWN = 5
SQLALCHEMY_AVAILABLE = True
STREAMING_OPEN_MAX_RETRIES = 20
STREAMING_OPEN_RETRY_INTERVAL = 5
STREAMING_READ_MAX_RETRIES = 20
STREAMING_READ_RATE_LIMIT_RETRY_INTERVAL = 60
STREAMING_READ_RETRY_INTERVAL = 5
STREAMING_READ_SERVER_UNAVAILABLE_RETRY_INTERVAL = 20
TEICH_AVAILABLE = False
TEMP_CACHE_DIR_PREFIX = 'hf_datasets-'
TF_AVAILABLE = True
TF_VERSION = <Version('2.17.0')>
TORCHCODEC_AVAILABLE = False
TORCHVISION_AVAILABLE = False
TORCH_AVAILABLE = True
TORCH_VERSION = <Version('2.13.0')>
TRIMESH_AVAILABLE = True
UPLOADS_MAX_NUMBER_PER_COMMIT = 50
USE_JAX = 'AUTO'
USE_PARQUET_EXPORT = True
USE_TF = 'AUTO'
USE_TORCH = 'AUTO'
XDG_CACHE_HOME = '~/.cache'
ZSTANDARD_AVAILABLE = True
__annotations__ = {'HF_DATASETS_DISABLE_PROGRESS_BARS': typing.Optiona...
logger = <Logger datasets (WARNING)>
package = 'tensorflow'
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\config.py
datasets.data_files
Python Library Documentation: module datasets.data_files in datasets
NAME
datasets.data_files
CLASSES
builtins.FileNotFoundError(builtins.OSError)
EmptyDatasetError
builtins.dict(builtins.object)
DataFilesDict
DataFilesPatternsDict
builtins.list(builtins.object)
DataFilesList
DataFilesPatternsList
builtins.str(builtins.object)
Url
class DataFilesDict(builtins.dict)
| Dict of split_name -> list of data files (absolute local paths or URLs).
| It has two construction methods given the user's data files patterns :
| - ``from_hf_repo``: resolve patterns inside a dataset repository
| - ``from_local_or_remote``: resolve patterns from a local path
|
| Moreover, each list is a DataFilesList. It is possible to hash the dictionary
| and get a different hash if and only if at least one file changed.
| For more info, see [`DataFilesList`].
|
| This is useful for caching Dataset objects that are obtained from a list of data files.
|
| Changing the order of the keys of this dictionary also doesn't change its hash.
|
| Method resolution order:
| DataFilesDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| filter(self, *, extensions: Optional[list[str]] = None, file_names: Optional[list[str]] = None) -> 'DataFilesDict'
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_hf_repo(patterns: dict[str, typing.Union[list[str], datasets.data_files.DataFilesList]], dataset_info: huggingface_hub.hf_api.DatasetInfo, base_path: Optional[str] = None, allowed_extensions: Optional[list[str]] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'DataFilesDict'
|
| from_local_or_remote(patterns: dict[str, typing.Union[list[str], datasets.data_files.DataFilesList]], base_path: Optional[str] = None, allowed_extensions: Optional[list[str]] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'DataFilesDict'
|
| from_patterns(patterns: dict[str, typing.Union[list[str], datasets.data_files.DataFilesList]], base_path: Optional[str] = None, allowed_extensions: Optional[list[str]] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'DataFilesDict'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (dict[str, datasets.data_files.DataFilesList],)
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class DataFilesList(builtins.list)
| DataFilesList(data_files: list[str], origin_metadata: list[typing.Union[tuple[str, str], tuple[str], tuple[()]]]) -> None
|
| List of data files (absolute local paths or URLs).
| It has two construction methods given the user's data files patterns:
| - ``from_hf_repo``: resolve patterns inside a dataset repository
| - ``from_local_or_remote``: resolve patterns from a local path
|
| Moreover, DataFilesList has an additional attribute ``origin_metadata``.
| It can store:
| - the last modified time of local files
| - ETag of remote files
| - commit sha of a dataset repository
|
| Thanks to this additional attribute, it is possible to hash the list
| and get a different hash if and only if at least one file changed.
| This is useful for caching Dataset objects that are obtained from a list of data files.
|
| Method resolution order:
| DataFilesList
| builtins.list
| builtins.object
|
| Methods defined here:
|
| __add__(self, other: 'DataFilesList') -> 'DataFilesList'
| Return self+value.
|
| __init__(self, data_files: list[str], origin_metadata: list[typing.Union[tuple[str, str], tuple[str], tuple[()]]]) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| filter(self, *, extensions: Optional[list[str]] = None, file_names: Optional[list[str]] = None) -> 'DataFilesList'
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_hf_repo(patterns: list[str], dataset_info: huggingface_hub.hf_api.DatasetInfo, base_path: Optional[str] = None, allowed_extensions: Optional[list[str]] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'DataFilesList'
|
| from_local_or_remote(patterns: list[str], base_path: Optional[str] = None, allowed_extensions: Optional[list[str]] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'DataFilesList'
|
| from_patterns(patterns: list[str], base_path: Optional[str] = None, allowed_extensions: Optional[list[str]] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'DataFilesList'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (list[str],)
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.list:
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, index, /)
| Return self[index].
|
| __gt__(self, value, /)
| Return self>value.
|
| __iadd__(self, value, /)
| Implement self+=value.
|
| __imul__(self, value, /)
| Implement self*=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the list.
|
| __rmul__(self, value, /)
| Return value*self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(self, /)
| Return the size of the list in memory, in bytes.
|
| append(self, object, /)
| Append object to the end of the list.
|
| clear(self, /)
| Remove all items from list.
|
| copy(self, /)
| Return a shallow copy of the list.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| extend(self, iterable, /)
| Extend list by appending elements from the iterable.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| insert(self, index, object, /)
| Insert object before index.
|
| pop(self, index=-1, /)
| Remove and return item at index (default last).
|
| Raises IndexError if list is empty or index is out of range.
|
| remove(self, value, /)
| Remove first occurrence of value.
|
| Raises ValueError if the value is not present.
|
| reverse(self, /)
| Reverse *IN PLACE*.
|
| sort(self, /, *, key=None, reverse=False)
| Sort the list in ascending order and return None.
|
| The sort is in-place (i.e. the list itself is modified) and stable (i.e. the
| order of two equal elements is maintained).
|
| If a key function is given, apply it once to each list item and sort them,
| ascending or descending, according to their function values.
|
| The reverse flag can be set to sort in descending order.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.list:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.list:
|
| __new__(*args, **kwargs) class method of builtins.list
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.list:
|
| __hash__ = None
class DataFilesPatternsDict(builtins.dict)
| Dict of split_name -> list of data files patterns (absolute local paths or URLs).
|
| Method resolution order:
| DataFilesPatternsDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| filter_extensions(self, extensions: list[str]) -> 'DataFilesPatternsDict'
|
| resolve(self, base_path: str, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'DataFilesDict'
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_patterns(patterns: dict[str, list[str]], allowed_extensions: Optional[list[str]] = None) -> 'DataFilesPatternsDict'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (dict[str, datasets.data_files.DataFilesPatternsList]...
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class DataFilesPatternsList(builtins.list)
| DataFilesPatternsList(patterns: list[str], allowed_extensions: list[typing.Optional[list[str]]])
|
| List of data files patterns (absolute local paths or URLs).
| For each pattern there should also be a list of allowed extensions
| to keep, or a None ot keep all the files for the pattern.
|
| Method resolution order:
| DataFilesPatternsList
| builtins.list
| builtins.object
|
| Methods defined here:
|
| __add__(self, other)
| Return self+value.
|
| __init__(self, patterns: list[str], allowed_extensions: list[typing.Optional[list[str]]])
| Initialize self. See help(type(self)) for accurate signature.
|
| filter_extensions(self, extensions: list[str]) -> 'DataFilesPatternsList'
|
| resolve(self, base_path: str, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'DataFilesList'
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_patterns(patterns: list[str], allowed_extensions: Optional[list[str]] = None) -> 'DataFilesPatternsList'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (list[str],)
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.list:
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, index, /)
| Return self[index].
|
| __gt__(self, value, /)
| Return self>value.
|
| __iadd__(self, value, /)
| Implement self+=value.
|
| __imul__(self, value, /)
| Implement self*=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the list.
|
| __rmul__(self, value, /)
| Return value*self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(self, /)
| Return the size of the list in memory, in bytes.
|
| append(self, object, /)
| Append object to the end of the list.
|
| clear(self, /)
| Remove all items from list.
|
| copy(self, /)
| Return a shallow copy of the list.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| extend(self, iterable, /)
| Extend list by appending elements from the iterable.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| insert(self, index, object, /)
| Insert object before index.
|
| pop(self, index=-1, /)
| Remove and return item at index (default last).
|
| Raises IndexError if list is empty or index is out of range.
|
| remove(self, value, /)
| Remove first occurrence of value.
|
| Raises ValueError if the value is not present.
|
| reverse(self, /)
| Reverse *IN PLACE*.
|
| sort(self, /, *, key=None, reverse=False)
| Sort the list in ascending order and return None.
|
| The sort is in-place (i.e. the list itself is modified) and stable (i.e. the
| order of two equal elements is maintained).
|
| If a key function is given, apply it once to each list item and sort them,
| ascending or descending, according to their function values.
|
| The reverse flag can be set to sort in descending order.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.list:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.list:
|
| __new__(*args, **kwargs) class method of builtins.list
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.list:
|
| __hash__ = None
class EmptyDatasetError(builtins.FileNotFoundError)
| Method resolution order:
| EmptyDatasetError
| builtins.FileNotFoundError
| builtins.OSError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.FileNotFoundError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.OSError:
|
| __reduce__(...)
| Helper for pickle.
|
| __str__(self, /)
| Return str(self).
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.OSError:
|
| __new__(*args, **kwargs) class method of builtins.OSError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.OSError:
|
| characters_written
|
| errno
| POSIX exception code
|
| filename
| exception filename
|
| filename2
| second exception filename
|
| strerror
| exception strerror
|
| winerror
| Win32 exception code
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class Url(builtins.str)
| Method resolution order:
| Url
| builtins.str
| builtins.object
|
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.str:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __format__(self, format_spec, /)
| Return a formatted version of the string as described by format_spec.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(...)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mod__(self, value, /)
| Return self%value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __repr__(self, /)
| Return repr(self).
|
| __rmod__(self, value, /)
| Return value%self.
|
| __rmul__(self, value, /)
| Return value*self.
|
| __sizeof__(self, /)
| Return the size of the string in memory, in bytes.
|
| __str__(self, /)
| Return str(self).
|
| capitalize(self, /)
| Return a capitalized version of the string.
|
| More specifically, make the first character have upper case and the rest lower
| case.
|
| casefold(self, /)
| Return a version of the string suitable for caseless comparisons.
|
| center(self, width, fillchar=' ', /)
| Return a centered string of length width.
|
| Padding is done using the specified fill character (default is a space).
|
| count(...)
| S.count(sub[, start[, end]]) -> int
|
| Return the number of non-overlapping occurrences of substring sub in
| string S[start:end]. Optional arguments start and end are
| interpreted as in slice notation.
|
| encode(self, /, encoding='utf-8', errors='strict')
| Encode the string using the codec registered for encoding.
|
| encoding
| The encoding in which to encode the string.
| errors
| The error handling scheme to use for encoding errors.
| The default is 'strict' meaning that encoding errors raise a
| UnicodeEncodeError. Other possible values are 'ignore', 'replace' and
| 'xmlcharrefreplace' as well as any other name registered with
| codecs.register_error that can handle UnicodeEncodeErrors.
|
| endswith(...)
| S.endswith(suffix[, start[, end]]) -> bool
|
| Return True if S ends with the specified suffix, False otherwise.
| With optional start, test S beginning at that position.
| With optional end, stop comparing S at that position.
| suffix can also be a tuple of strings to try.
|
| expandtabs(self, /, tabsize=8)
| Return a copy where all tab characters are expanded using spaces.
|
| If tabsize is not given, a tab size of 8 characters is assumed.
|
| find(...)
| S.find(sub[, start[, end]]) -> int
|
| Return the lowest index in S where substring sub is found,
| such that sub is contained within S[start:end]. Optional
| arguments start and end are interpreted as in slice notation.
|
| Return -1 on failure.
|
| format(...)
| S.format(*args, **kwargs) -> str
|
| Return a formatted version of S, using substitutions from args and kwargs.
| The substitutions are identified by braces ('{' and '}').
|
| format_map(...)
| S.format_map(mapping) -> str
|
| Return a formatted version of S, using substitutions from mapping.
| The substitutions are identified by braces ('{' and '}').
|
| index(...)
| S.index(sub[, start[, end]]) -> int
|
| Return the lowest index in S where substring sub is found,
| such that sub is contained within S[start:end]. Optional
| arguments start and end are interpreted as in slice notation.
|
| Raises ValueError when the substring is not found.
|
| isalnum(self, /)
| Return True if the string is an alpha-numeric string, False otherwise.
|
| A string is alpha-numeric if all characters in the string are alpha-numeric and
| there is at least one character in the string.
|
| isalpha(self, /)
| Return True if the string is an alphabetic string, False otherwise.
|
| A string is alphabetic if all characters in the string are alphabetic and there
| is at least one character in the string.
|
| isascii(self, /)
| Return True if all characters in the string are ASCII, False otherwise.
|
| ASCII characters have code points in the range U+0000-U+007F.
| Empty string is ASCII too.
|
| isdecimal(self, /)
| Return True if the string is a decimal string, False otherwise.
|
| A string is a decimal string if all characters in the string are decimal and
| there is at least one character in the string.
|
| isdigit(self, /)
| Return True if the string is a digit string, False otherwise.
|
| A string is a digit string if all characters in the string are digits and there
| is at least one character in the string.
|
| isidentifier(self, /)
| Return True if the string is a valid Python identifier, False otherwise.
|
| Call keyword.iskeyword(s) to test whether string s is a reserved identifier,
| such as "def" or "class".
|
| islower(self, /)
| Return True if the string is a lowercase string, False otherwise.
|
| A string is lowercase if all cased characters in the string are lowercase and
| there is at least one cased character in the string.
|
| isnumeric(self, /)
| Return True if the string is a numeric string, False otherwise.
|
| A string is numeric if all characters in the string are numeric and there is at
| least one character in the string.
|
| isprintable(self, /)
| Return True if the string is printable, False otherwise.
|
| A string is printable if all of its characters are considered printable in
| repr() or if it is empty.
|
| isspace(self, /)
| Return True if the string is a whitespace string, False otherwise.
|
| A string is whitespace if all characters in the string are whitespace and there
| is at least one character in the string.
|
| istitle(self, /)
| Return True if the string is a title-cased string, False otherwise.
|
| In a title-cased string, upper- and title-case characters may only
| follow uncased characters and lowercase characters only cased ones.
|
| isupper(self, /)
| Return True if the string is an uppercase string, False otherwise.
|
| A string is uppercase if all cased characters in the string are uppercase and
| there is at least one cased character in the string.
|
| join(self, iterable, /)
| Concatenate any number of strings.
|
| The string whose method is called is inserted in between each given string.
| The result is returned as a new string.
|
| Example: '.'.join(['ab', 'pq', 'rs']) -> 'ab.pq.rs'
|
| ljust(self, width, fillchar=' ', /)
| Return a left-justified string of length width.
|
| Padding is done using the specified fill character (default is a space).
|
| lower(self, /)
| Return a copy of the string converted to lowercase.
|
| lstrip(self, chars=None, /)
| Return a copy of the string with leading whitespace removed.
|
| If chars is given and not None, remove characters in chars instead.
|
| partition(self, sep, /)
| Partition the string into three parts using the given separator.
|
| This will search for the separator in the string. If the separator is found,
| returns a 3-tuple containing the part before the separator, the separator
| itself, and the part after it.
|
| If the separator is not found, returns a 3-tuple containing the original string
| and two empty strings.
|
| removeprefix(self, prefix, /)
| Return a str with the given prefix string removed if present.
|
| If the string starts with the prefix string, return string[len(prefix):].
| Otherwise, return a copy of the original string.
|
| removesuffix(self, suffix, /)
| Return a str with the given suffix string removed if present.
|
| If the string ends with the suffix string and that suffix is not empty,
| return string[:-len(suffix)]. Otherwise, return a copy of the original
| string.
|
| replace(self, old, new, count=-1, /)
| Return a copy with all occurrences of substring old replaced by new.
|
| count
| Maximum number of occurrences to replace.
| -1 (the default value) means replace all occurrences.
|
| If the optional argument count is given, only the first count occurrences are
| replaced.
|
| rfind(...)
| S.rfind(sub[, start[, end]]) -> int
|
| Return the highest index in S where substring sub is found,
| such that sub is contained within S[start:end]. Optional
| arguments start and end are interpreted as in slice notation.
|
| Return -1 on failure.
|
| rindex(...)
| S.rindex(sub[, start[, end]]) -> int
|
| Return the highest index in S where substring sub is found,
| such that sub is contained within S[start:end]. Optional
| arguments start and end are interpreted as in slice notation.
|
| Raises ValueError when the substring is not found.
|
| rjust(self, width, fillchar=' ', /)
| Return a right-justified string of length width.
|
| Padding is done using the specified fill character (default is a space).
|
| rpartition(self, sep, /)
| Partition the string into three parts using the given separator.
|
| This will search for the separator in the string, starting at the end. If
| the separator is found, returns a 3-tuple containing the part before the
| separator, the separator itself, and the part after it.
|
| If the separator is not found, returns a 3-tuple containing two empty strings
| and the original string.
|
| rsplit(self, /, sep=None, maxsplit=-1)
| Return a list of the substrings in the string, using sep as the separator string.
|
| sep
| The separator used to split the string.
|
| When set to None (the default value), will split on any whitespace
| character (including \n \r \t \f and spaces) and will discard
| empty strings from the result.
| maxsplit
| Maximum number of splits.
| -1 (the default value) means no limit.
|
| Splitting starts at the end of the string and works to the front.
|
| rstrip(self, chars=None, /)
| Return a copy of the string with trailing whitespace removed.
|
| If chars is given and not None, remove characters in chars instead.
|
| split(self, /, sep=None, maxsplit=-1)
| Return a list of the substrings in the string, using sep as the separator string.
|
| sep
| The separator used to split the string.
|
| When set to None (the default value), will split on any whitespace
| character (including \n \r \t \f and spaces) and will discard
| empty strings from the result.
| maxsplit
| Maximum number of splits.
| -1 (the default value) means no limit.
|
| Splitting starts at the front of the string and works to the end.
|
| Note, str.split() is mainly useful for data that has been intentionally
| delimited. With natural text that includes punctuation, consider using
| the regular expression module.
|
| splitlines(self, /, keepends=False)
| Return a list of the lines in the string, breaking at line boundaries.
|
| Line breaks are not included in the resulting list unless keepends is given and
| true.
|
| startswith(...)
| S.startswith(prefix[, start[, end]]) -> bool
|
| Return True if S starts with the specified prefix, False otherwise.
| With optional start, test S beginning at that position.
| With optional end, stop comparing S at that position.
| prefix can also be a tuple of strings to try.
|
| strip(self, chars=None, /)
| Return a copy of the string with leading and trailing whitespace removed.
|
| If chars is given and not None, remove characters in chars instead.
|
| swapcase(self, /)
| Convert uppercase characters to lowercase and lowercase characters to uppercase.
|
| title(self, /)
| Return a version of the string where each word is titlecased.
|
| More specifically, words start with uppercased characters and all remaining
| cased characters have lower case.
|
| translate(self, table, /)
| Replace each character in the string using the given translation table.
|
| table
| Translation table, which must be a mapping of Unicode ordinals to
| Unicode ordinals, strings, or None.
|
| The table must implement lookup/indexing via __getitem__, for instance a
| dictionary or list. If this operation raises LookupError, the character is
| left untouched. Characters mapped to None are deleted.
|
| upper(self, /)
| Return a copy of the string converted to uppercase.
|
| zfill(self, width, /)
| Pad a numeric string with zeros on the left, to fill a field of the given width.
|
| The string is never truncated.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.str:
|
| __new__(*args, **kwargs) class method of builtins.str
| Create and return a new object. See help(type) for accurate signature.
|
| maketrans(...)
| Return a translation table usable for str.translate().
|
| If there is only one argument, it must be a dictionary mapping Unicode
| ordinals (integers) or characters to Unicode ordinals, strings or None.
| Character keys will be then converted to ordinals.
| If there are two arguments, they must be strings of equal length, and
| in the resulting dictionary, each character in x will be mapped to the
| character at the same position in y. If there is a third argument, it
| must be a string, whose characters will be mapped to None in the result.
FUNCTIONS
contains_wildcards(pattern: str) -> bool
get_data_patterns(base_path: str, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> dict[str, list[str]]
Get the default pattern from a directory testing all the supported patterns.
The first patterns to return a non-empty list of data files is returned.
Some examples of supported patterns:
Input:
my_dataset_repository/
├── README.md
└── dataset.csv
Output:
{'train': ['**']}
Input:
my_dataset_repository/
├── README.md
├── train.csv
└── test.csv
my_dataset_repository/
├── README.md
└── data/
├── train.csv
└── test.csv
my_dataset_repository/
├── README.md
├── train_0.csv
├── train_1.csv
├── train_2.csv
├── train_3.csv
├── test_0.csv
└── test_1.csv
Output:
{'train': ['**/train[-._ 0-9]*', '**/*[-._ 0-9]train[-._ 0-9]*', '**/training[-._ 0-9]*', '**/*[-._ 0-9]training[-._ 0-9]*'],
'test': ['**/test[-._ 0-9]*', '**/*[-._ 0-9]test[-._ 0-9]*', '**/testing[-._ 0-9]*', '**/*[-._ 0-9]testing[-._ 0-9]*', ...]}
Input:
my_dataset_repository/
├── README.md
└── data/
├── train/
│ ├── shard_0.csv
│ ├── shard_1.csv
│ ├── shard_2.csv
│ └── shard_3.csv
└── test/
├── shard_0.csv
└── shard_1.csv
Output:
{'train': ['**/train/**', '**/train[-._ 0-9]*/**', '**/*[-._ 0-9]train/**', '**/*[-._ 0-9]train[-._ 0-9]*/**', ...],
'test': ['**/test/**', '**/test[-._ 0-9]*/**', '**/*[-._ 0-9]test/**', '**/*[-._ 0-9]test[-._ 0-9]*/**', ...]}
Input:
my_dataset_repository/
├── README.md
└── data/
├── train-00000-of-00003.csv
├── train-00001-of-00003.csv
├── train-00002-of-00003.csv
├── test-00000-of-00001.csv
├── random-00000-of-00003.csv
├── random-00001-of-00003.csv
└── random-00002-of-00003.csv
Output:
{'train': ['data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'],
'test': ['data/test-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'],
'random': ['data/random-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*']}
In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS.
resolve_pattern(pattern: str, base_path: str, allowed_extensions: Optional[list[str]] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> list[str]
Resolve the paths and URLs of the data files from the pattern passed by the user.
You can use patterns to resolve multiple local files. Here are a few examples:
- *.csv to match all the CSV files at the first level
- **.csv to match all the CSV files at any level
- data/* to match all the files inside "data"
- data/** to match all the files inside "data" and its subdirectories
The patterns are resolved using the fsspec glob. In fsspec>=2023.12.0 this is equivalent to
Python's glob.glob, Path.glob, Path.match and fnmatch where ** is unsupported with a prefix/suffix
other than a forward slash /.
More generally:
- '*' matches any character except a forward-slash (to match just the file or directory name)
- '**' matches any character including a forward-slash /
Hidden files and directories (i.e. whose names start with a dot) are ignored, unless they are explicitly requested.
The same applies to special directories that start with a double underscore like "__pycache__".
You can still include one if the pattern explicitly mentions it:
- to include a hidden file: "*/.hidden.txt" or "*/.*"
- to include a hidden directory: ".hidden/*" or ".*/*"
- to include a special directory: "__special__/*" or "__*/*"
Example::
>>> from datasets.data_files import resolve_pattern
>>> base_path = "."
>>> resolve_pattern("docs/**/*.py", base_path)
[/Users/mariosasko/Desktop/projects/datasets/docs/source/_config.py']
Args:
pattern (str): Unix pattern or paths or URLs of the data files to resolve.
The paths can be absolute or relative to base_path.
Remote filesystems using fsspec are supported, e.g. with the hf:// protocol.
base_path (str): Base path to use when resolving relative paths.
allowed_extensions (Optional[list], optional): White-list of file extensions to use. Defaults to None (all extensions).
For example: allowed_extensions=[".csv", ".json", ".txt", ".parquet"]
download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters.
Returns:
List[str]: List of paths or URLs to the local or remote files that match the patterns.
sanitize_patterns(patterns: Union[dict, list, str]) -> dict[str, typing.Union[list[str], ForwardRef('DataFilesList')]]
Take the data_files patterns from the user, and format them into a dictionary.
Each key is the name of the split, and each value is a list of data files patterns (paths or urls).
The default split is "train".
Returns:
patterns: dictionary of split_name -> list of patterns
DATA
ALL_DEFAULT_PATTERNS = [{'logs': ['**/*.eval']}, {NamedSplit('test'): ...
ALL_SPLIT_PATTERNS = ['data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9]...
Callable = typing.Callable
Deprecated alias to collections.abc.Callable.
Callable[[int], str] signifies a function that takes a single
parameter of type int and returns a str.
The subscription syntax must always be used with exactly two
values: the argument list and the return type.
The argument list must be a list of types, a ParamSpec,
Concatenate or ellipsis. The return type must be a single type.
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
DEFAULT_PATTERNS_ALL = {NamedSplit('train'): ['**']}
DEFAULT_PATTERNS_LOGS = {'logs': ['**/*.eval']}
DEFAULT_PATTERNS_SPLIT_IN_DIR_NAME = {NamedSplit('test'): ['**/test/**...
DEFAULT_PATTERNS_SPLIT_IN_FILENAME = {NamedSplit('test'): ['**/test[-....
DEFAULT_SPLITS = [NamedSplit('train'), NamedSplit('validation'), Named...
FILES_TO_IGNORE = ['README.md', 'config.json', 'dataset_info.json', 'd...
KEYWORDS_IN_DIR_NAME_BASE_PATTERNS = ['**/{keyword}/**', '**/{keyword}...
KEYWORDS_IN_FILENAME_BASE_PATTERNS = ['**/{keyword}[{sep}]*', '**/*[{s...
NON_WORDS_CHARS = '-._ 0-9'
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
SANITIZED_DEFAULT_SPLIT = 'train'
SPLIT_KEYWORDS = {NamedSplit('test'): ['test', 'testing', 'eval', 'eva...
SPLIT_PATTERN_SHARDED = 'data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-...
SingleOriginMetadata = typing.Union[tuple[str, str], tuple[str], tuple...
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
WILDCARD_CHARACTERS = '*[]'
logger = <Logger datasets.data_files (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\data_files.py
datasets.dataset_dict
Python Library Documentation: module datasets.dataset_dict in datasets
NAME
datasets.dataset_dict
CLASSES
builtins.dict(builtins.object)
DatasetDict
IterableDatasetDict
functools.partial(builtins.object)
bind
class DatasetDict(builtins.dict)
| A dictionary (dict of str: datasets.Dataset) with dataset transforms methods (map, filter, etc.)
|
| Method resolution order:
| DatasetDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_val, exc_tb)
|
| __getitem__(self, k) -> datasets.arrow_dataset.Dataset
| Return self[key].
|
| __repr__(self)
| Return repr(self).
|
| align_labels_with_mapping(self, label2id: dict, label_column: str) -> 'DatasetDict'
| Align the dataset's label ID and label name mapping to match an input `label2id` mapping.
| This is useful when you want to ensure that a model's predicted labels are aligned with the dataset.
| The alignment in done using the lowercase label names.
|
| Args:
| label2id (`dict`):
| The label name to ID mapping to align the dataset with.
| label_column (`str`):
| The column name of labels to align on.
|
| Example:
|
| ```python
| >>> # dataset with mapping {'entailment': 0, 'neutral': 1, 'contradiction': 2}
| >>> ds = load_dataset("nyu-mll/glue", "mnli", split="train")
| >>> # mapping to align with
| >>> label2id = {'CONTRADICTION': 0, 'NEUTRAL': 1, 'ENTAILMENT': 2}
| >>> ds_aligned = ds.align_labels_with_mapping(label2id, "label")
| ```
|
| cast(self, features: datasets.features.features.Features) -> 'DatasetDict'
| Cast the dataset to a new set of features.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Args:
| features ([`Features`]):
| New features to cast the dataset to.
| The name and order of the fields in the features must match the current column names.
| The type of the data must also be convertible from one type to the other.
| For non-trivial conversion, e.g. `string` <-> `ClassLabel` you should use [`~DatasetDict.map`] to update the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel, Value
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds["train"].features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> new_features = ds["train"].features.copy()
| >>> new_features['label'] = ClassLabel(names=['bad', 'good'])
| >>> new_features['text'] = Value('large_string')
| >>> ds = ds.cast(new_features)
| >>> ds["train"].features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('large_string')}
| ```
|
| cast_column(self, column: str, feature) -> 'DatasetDict'
| Cast column to feature for decoding.
|
| Args:
| column (`str`):
| Column name.
| feature ([`Feature`]):
| Target feature.
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds["train"].features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> ds = ds.cast_column('label', ClassLabel(names=['bad', 'good']))
| >>> ds["train"].features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('string')}
| ```
|
| class_encode_column(self, column: str, include_nulls: bool = False) -> 'DatasetDict'
| Casts the given column as [`~datasets.features.ClassLabel`] and updates the tables.
|
| Args:
| column (`str`):
| The name of the column to cast.
| include_nulls (`bool`, defaults to `False`):
| Whether to include null values in the class labels. If `True`, the null values will be encoded as the `"None"` class label.
|
| <Added version="1.14.2"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("google/boolq")
| >>> ds["train"].features
| {'answer': Value('bool'),
| 'passage': Value('string'),
| 'question': Value('string')}
| >>> ds = ds.class_encode_column("answer")
| >>> ds["train"].features
| {'answer': ClassLabel(num_classes=2, names=['False', 'True']),
| 'passage': Value('string'),
| 'question': Value('string')}
| ```
|
| cleanup_cache_files(self) -> dict[str, int]
| Clean up all cache files in the dataset cache directory, excepted the currently used cache file if there is one.
| Be careful when running this command that no other process is currently using other cache files.
|
| Return:
| `Dict` with the number of removed files for each split
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.cleanup_cache_files()
| {'test': 0, 'train': 0, 'validation': 0}
| ```
|
| filter(self, function: Optional[Callable] = None, with_indices: bool = False, with_rank: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, desc: Optional[str] = None) -> 'DatasetDict'
| Apply a filter function to all the elements in the table in batches
| and update the table so that the dataset only includes examples according to the filter function.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Args:
| function (`Callable`): Callable with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> bool` if `batched=False` and `with_indices=False` and `with_rank=False`
| - `function(example: Dict[str, Any], *extra_args) -> bool` if `batched=False` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
| - `function(batch: Dict[str, list]) -> list[bool]` if `batched=True` and `with_indices=False` and `with_rank=False`
| - `function(batch: Dict[str, list], *extra_args) -> list[bool]` if `batched=True` and `with_indices=True` and/or `with_rank=True` (one extra arg for each)
|
| If no function is provided, defaults to an always `True` function: `lambda x: True`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the
| signature of `function` should be `def function(example, idx[, rank]): ...`.
| with_rank (`bool`, defaults to `False`):
| Provide process rank to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], rank): ...`.
| input_columns (`[Union[str, list[str]]]`, *optional*, defaults to `None`):
| The columns to be passed into `function` as
| positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`
| `batch_size <= 0` or `batch_size == None` then provide the full dataset as a single batch to `function`.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_names (`[Dict[str, str]]`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`
| num_proc (`int`, *optional*, defaults to `None`):
| The number of processes to use for multiprocessing.
| - If `None` or `0`, no multiprocessing is used and the operation runs in the main process.
| - If greater than `1`, one or multiple worker processes are used to process data in parallel.
| Note: The function passed to `map()` must be picklable for multiprocessing to work correctly
| (i.e., prefer functions defined at the top level of a module, not inside another function or class).
| desc (`str`, *optional*, defaults to `None`):
| Meaningful description to be displayed alongside with the progress bar while filtering examples.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.filter(lambda x: x["label"] == 1)
| DatasetDict({
| train: Dataset({
| features: ['text', 'label'],
| num_rows: 4265
| })
| validation: Dataset({
| features: ['text', 'label'],
| num_rows: 533
| })
| test: Dataset({
| features: ['text', 'label'],
| num_rows: 533
| })
| })
| ```
|
| flatten(self, max_depth=16) -> 'DatasetDict'
| Flatten the Apache Arrow Table of each split (nested features are flatten).
| Each column with a struct type is flattened into one column per struct field.
| Other columns are left unchanged.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("rajpurkar/squad")
| >>> ds["train"].features
| {'id': Value('string'),
| 'title': Value('string'),
| 'context': Value('string'),
| 'question': Value('string'),
| 'answers.text': List(Value('string')),
| 'answers.answer_start': List(Value('int32'))}
| >>> ds.flatten()
| DatasetDict({
| train: Dataset({
| features: ['id', 'title', 'context', 'question', 'answers.text', 'answers.answer_start'],
| num_rows: 87599
| })
| validation: Dataset({
| features: ['id', 'title', 'context', 'question', 'answers.text', 'answers.answer_start'],
| num_rows: 10570
| })
| })
| ```
|
| flatten_indices(self, keep_in_memory: bool = False, cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000, features: Optional[datasets.features.features.Features] = None, disable_nullable: bool = False, num_proc: Optional[int] = None, new_fingerprint: Optional[str] = None) -> 'DatasetDict'
| Create and cache a new Dataset by flattening the indices mapping.
|
| Args:
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| cache_file_names (`Dict[str, str]`, *optional*, default `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| features (`Optional[datasets.Features]`, defaults to `None`):
| Use a specific [`Features`] to store the cache file
| instead of the automatically generated one.
| disable_nullable (`bool`, defaults to `False`):
| Allow null values in the table.
| num_proc (`int`, optional, default `None`):
| Max number of processes when generating cache. Already cached shards are loaded sequentially
| new_fingerprint (`str`, *optional*, defaults to `None`):
| The new fingerprint of the dataset after transform.
| If `None`, the new fingerprint is computed using a hash of the previous fingerprint, and the transform arguments
|
| formatted_as(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| To be used in a `with` statement. Set `__getitem__` return format (type and columns).
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`list[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to False):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| map(self, function: Optional[Callable] = None, with_indices: bool = False, with_rank: bool = False, with_split: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, drop_last_batch: bool = False, remove_columns: Union[str, list[str], NoneType] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000, features: Optional[datasets.features.features.Features] = None, disable_nullable: bool = False, fn_kwargs: Optional[dict] = None, num_proc: Optional[int] = None, desc: Optional[str] = None, try_original_type: Optional[bool] = True, on_mixed_types: Optional[Literal['use_json']] = None) -> 'DatasetDict'
| Apply a function to all the examples in the table (individually or in batches) and update the table.
| If your function returns a column that already exists, then it overwrites it.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| You can specify whether the function should be batched or not with the `batched` parameter:
|
| - If batched is `False`, then the function takes 1 example in and should return 1 example.
| An example is a dictionary, e.g. `{"text": "Hello there !"}`.
| - If batched is `True` and `batch_size` is 1, then the function takes a batch of 1 example as input and can return a batch with 1 or more examples.
| A batch is a dictionary, e.g. a batch of 1 example is `{"text": ["Hello there !"]}`.
| - If batched is `True` and `batch_size` is `n > 1`, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.
| Note that the last batch may have less than `n` examples.
| A batch is a dictionary, e.g. a batch of `n` examples is `{"text": ["Hello there !"] * n}`.
|
| If the function is asynchronous, then `map` will run your function in parallel, with up to one thousand simultaneous calls.
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`callable`): with one of the following signature:
| - `function(example: Dict[str, Any]) -> Dict[str, Any]` if `batched=False` and `with_indices=False`
| - `function(example: Dict[str, Any], indices: int) -> Dict[str, Any]` if `batched=False` and `with_indices=True`
| - `function(batch: Dict[str, list]) -> Dict[str, list]` if `batched=True` and `with_indices=False`
| - `function(batch: Dict[str, list], indices: list[int]) -> Dict[str, list]` if `batched=True` and `with_indices=True`
|
| For advanced usage, the function can also return a `pyarrow.Table`.
| If the function is asynchronous, then `map` will run your function in parallel.
| Moreover if your function returns nothing (`None`), then `map` will run your function and return the dataset unchanged.
| If no function is provided, default to identity function: `lambda x: x`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx): ...`.
| with_rank (`bool`, defaults to `False`):
| Provide process rank to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], rank): ...`.
| with_split (`bool`, defaults to `False`):
| Provide process split to `function`. Note that in this case the
| signature of `function` should be `def function(example[, idx], split): ...`.
| input_columns (`[Union[str, list[str]]]`, *optional*, defaults to `None`):
| The columns to be passed into `function` as
| positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`,
| `batch_size <= 0` or `batch_size == None` then provide the full dataset as a single batch to `function`.
| drop_last_batch (`bool`, defaults to `False`):
| Whether a last batch smaller than the batch_size should be
| dropped instead of being processed by the function.
| remove_columns (`[Union[str, list[str]]]`, *optional*, defaults to `None`):
| Remove a selection of columns while doing the mapping.
| Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding
| columns with names in `remove_columns`, these columns will be kept.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| cache_file_names (`[Dict[str, str]]`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| results of the computation instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, default `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
| features (`[datasets.Features]`, *optional*, defaults to `None`):
| Use a specific [`Features`] to store the cache file
| instead of the automatically generated one.
| disable_nullable (`bool`, defaults to `False`):
| Disallow null values in the table.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`
| num_proc (`int`, *optional*, defaults to `None`):
| The number of processes to use for multiprocessing.
| - If `None` or `0`, no multiprocessing is used and the operation runs in the main process.
| - If greater than `1`, one or multiple worker processes are used to process data in parallel.
| Note: The function passed to `map()` must be picklable for multiprocessing to work correctly
| (i.e., prefer functions defined at the top level of a module, not inside another function or class).
| desc (`str`, *optional*, defaults to `None`):
| Meaningful description to be displayed alongside with the progress bar while mapping examples.
| try_original_type (`Optional[bool]`, defaults to `True`):
| Try to keep the types of the original columns (e.g. int32 -> int32).
| Set to False if you want to always infer new types.
| on_mixed_types (`Literal["use_json"]`, *optional*, defaults to `None`):
| If "use_json", use the Json() type for mixed-types fields,
| i.e. unstructured fields that contain data without a predefined schema.
| In this case, a field with mixed type is set to Json().
|
| This allow loading lists with a mix of strings/integers/floats
| for example, or dictionaries with arbitrary value types.
|
| <Added version="4.7.0"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> def add_prefix(example):
| ... example["text"] = "Review: " + example["text"]
| ... return example
| >>> ds = ds.map(add_prefix)
| >>> ds["train"][0:3]["text"]
| ['Review: the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',
| 'Review: the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .',
| 'Review: effective but too-tepid biopic']
|
| # process a batch of examples
| >>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
| # set number of processors
| >>> ds = ds.map(add_prefix, num_proc=4)
| ```
|
| push_to_hub(self, repo_id: str, config_name: str = 'default', set_default: Optional[bool] = None, data_dir: Optional[str] = None, commit_message: Optional[str] = None, commit_description: Optional[str] = None, private: Optional[bool] = None, token: Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = False, max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[dict[str, int]] = None, embed_external_files: bool = True, num_proc: Optional[int] = None) -> Optional[huggingface_hub.hf_api.CommitInfo]
| Pushes the [`DatasetDict`] to the hub as a Parquet dataset.
| The [`DatasetDict`] is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
|
| Each dataset split will be pushed independently. The pushed dataset will keep the original split names.
|
| The resulting Parquet files are self-contained by default: if your dataset contains [`Image`] or [`Audio`]
| data, the Parquet files will store the bytes of your images or audio files.
| You can disable this by setting `embed_external_files` to False.
|
| Args:
| repo_id (`str`):
| The ID of the repository to push to in the following format: `<user>/<dataset_name>` or
| `<org>/<dataset_name>`. Also accepts `<dataset_name>`, which will default to the namespace
| of the logged-in user.
|
| It could also be a location inside a bucket, e.g. `buckets/<user_or_org>/<bucket_name>/...`
| config_name (`str`):
| Configuration name of a dataset. Defaults to "default".
| set_default (`bool`, *optional*):
| Whether to set this configuration as the default one. Otherwise, the default configuration is the one
| named "default".
| data_dir (`str`, *optional*):
| Directory name that will contain the uploaded data files. Defaults to the `config_name` if different
| from "default", else "data".
|
| <Added version="2.17.0"/>
| commit_message (`str`, *optional*):
| Message to commit while pushing. Will default to `"Upload dataset"`.
| commit_description (`str`, *optional*):
| Description of the commit that will be created.
| Additionally, description of the PR if a PR is created (`create_pr` is True).
|
| <Added version="2.16.0"/>
| private (`bool`, *optional*):
| Whether to make the repo private. If `None` (default), the repo will be public unless the
| organization's default is private. This value is ignored if the repo already exists.
| token (`str`, *optional*):
| An optional authentication token for the Hugging Face Hub. If no token is passed, will default
| to the token saved locally when logging in with `huggingface-cli login`. Will raise an error
| if no token is passed and the user is not logged-in.
| revision (`str`, *optional*):
| Branch to push the uploaded files to. Defaults to the `"main"` branch.
|
| <Added version="2.15.0"/>
| create_pr (`bool`, *optional*, defaults to `False`):
| Whether to create a PR with the uploaded files or directly commit.
|
| <Added version="2.15.0"/>
| max_shard_size (`int` or `str`, *optional*, defaults to `"500MB"`):
| The maximum size of the dataset shards to be uploaded to the hub. If expressed as a string, needs to be digits followed by a unit
| (like `"500MB"` or `"1GB"`).
| num_shards (`Dict[str, int]`, *optional*):
| Number of shards to write. By default, the number of shards depends on `max_shard_size`.
| Use a dictionary to define a different num_shards for each split.
|
| <Added version="2.8.0"/>
| embed_external_files (`bool`, defaults to `True`):
| Whether to embed file bytes in the shards.
| In particular, this will do the following before the push for the fields of type:
|
| - [`Audio`] and [`Image`] removes local path information and embed file content in the Parquet files.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when preparing and uploading the dataset.
| This is helpful if the dataset is made of many samples or media files to embed.
| I uses "spawn" context to work with hf_xet, the rust client for fast uploads to HF.
| Multiprocessing is disabled by default.
|
| <Added version="4.0.0"/>
|
| Return:
| huggingface_hub.CommitInfo
|
| Example:
|
| ```python
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", private=True)
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", max_shard_size="1GB")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", num_shards={"train": 1024, "test": 8})
| ```
|
| If you want to add a new configuration (or subset) to a dataset (e.g. if the dataset has multiple tasks/versions/languages):
|
| ```python
| >>> english_dataset.push_to_hub("<organization>/<dataset_id>", "en")
| >>> french_dataset.push_to_hub("<organization>/<dataset_id>", "fr")
| >>> # later
| >>> english_dataset = load_dataset("<organization>/<dataset_id>", "en")
| >>> french_dataset = load_dataset("<organization>/<dataset_id>", "fr")
| ```
|
| remove_columns(self, column_names: Union[str, list[str]]) -> 'DatasetDict'
| Remove one or several column(s) from each split in the dataset
| and the features associated to the column(s).
|
| The transformation is applied to all the splits of the dataset dictionary.
|
| You can also remove a column using [`~DatasetDict.map`] with `remove_columns` but the present method
| doesn't copy the data of the remaining columns and is thus faster.
|
| Args:
| column_names (`Union[str, list[str]]`):
| Name of the column(s) to remove.
|
| Returns:
| [`DatasetDict`]: A copy of the dataset object without the columns to remove.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds = ds.remove_columns("label")
| DatasetDict({
| train: Dataset({
| features: ['text'],
| num_rows: 8530
| })
| validation: Dataset({
| features: ['text'],
| num_rows: 1066
| })
| test: Dataset({
| features: ['text'],
| num_rows: 1066
| })
| })
| ```
|
| rename_column(self, original_column_name: str, new_column_name: str) -> 'DatasetDict'
| Rename a column in the dataset and move the features associated to the original column under the new column name.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| You can also rename a column using [`~DatasetDict.map`] with `remove_columns` but the present method:
| - takes care of moving the original features under the new column name.
| - doesn't copy the data to a new dataset and is thus much faster.
|
| Args:
| original_column_name (`str`):
| Name of the column to rename.
| new_column_name (`str`):
| New name for the column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds = ds.rename_column("label", "label_new")
| DatasetDict({
| train: Dataset({
| features: ['text', 'label_new'],
| num_rows: 8530
| })
| validation: Dataset({
| features: ['text', 'label_new'],
| num_rows: 1066
| })
| test: Dataset({
| features: ['text', 'label_new'],
| num_rows: 1066
| })
| })
| ```
|
| rename_columns(self, column_mapping: dict[str, str]) -> 'DatasetDict'
| Rename several columns in the dataset, and move the features associated to the original columns under
| the new column names.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Args:
| column_mapping (`Dict[str, str]`):
| A mapping of columns to rename to their new names.
|
| Returns:
| [`DatasetDict`]: A copy of the dataset with renamed columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.rename_columns({'text': 'text_new', 'label': 'label_new'})
| DatasetDict({
| train: Dataset({
| features: ['text_new', 'label_new'],
| num_rows: 8530
| })
| validation: Dataset({
| features: ['text_new', 'label_new'],
| num_rows: 1066
| })
| test: Dataset({
| features: ['text_new', 'label_new'],
| num_rows: 1066
| })
| })
| ```
|
| reset_format(self)
| Reset `__getitem__` return format to python objects and all columns.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Same as `self.set_format()`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x["text"], truncation=True, padding=True), batched=True)
| >>> ds.set_format(type="numpy", columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
| >>> ds["train"].format
| {'columns': ['input_ids', 'token_type_ids', 'attention_mask', 'label'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'numpy'}
| >>> ds.reset_format()
| >>> ds["train"].format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': None}
| ```
|
| save_to_disk(self, dataset_dict_path: Union[str, bytes, os.PathLike], max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[dict[str, int]] = None, num_proc: Optional[int] = None, storage_options: Optional[dict] = None)
| Saves a dataset dict to a filesystem using `fsspec.spec.AbstractFileSystem`.
|
| For [`Image`], [`Audio`] and [`Video`] data:
|
| All the Image(), Audio() and Video() data are stored in the arrow files.
| If you want to store paths or urls, please use the Value("string") type.
|
| Args:
| dataset_dict_path (`path-like`):
| Path (e.g. `dataset/train`) or remote URI (e.g. `s3://my-bucket/dataset/train`)
| of the dataset dict directory where the dataset dict will be saved to.
| max_shard_size (`int` or `str`, *optional*, defaults to `"500MB"`):
| The maximum size of the dataset shards to be saved to the filesystem. If expressed as a string, needs to be digits followed by a unit
| (like `"50MB"`).
| num_shards (`Dict[str, int]`, *optional*):
| Number of shards to write. By default the number of shards depends on `max_shard_size` and `num_proc`.
| You need to provide the number of shards for each dataset in the dataset dictionary.
| Use a dictionary to define a different num_shards for each split.
|
| <Added version="2.8.0"/>
| num_proc (`int`, *optional*, default `None`):
| Number of processes when downloading and generating the dataset locally.
| Multiprocessing is disabled by default.
|
| <Added version="2.8.0"/>
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.8.0"/>
|
| Example:
|
| ```python
| >>> dataset_dict.save_to_disk("path/to/dataset/directory")
| >>> dataset_dict.save_to_disk("path/to/dataset/directory", max_shard_size="1GB")
| >>> dataset_dict.save_to_disk("path/to/dataset/directory", num_shards={"train": 1024, "test": 8})
| ```
|
| select_columns(self, column_names: Union[str, list[str]]) -> 'DatasetDict'
| Select one or several column(s) from each split in the dataset and
| the features associated to the column(s).
|
| The transformation is applied to all the splits of the dataset
| dictionary.
|
| Args:
| column_names (`Union[str, list[str]]`):
| Name of the column(s) to keep.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.select_columns("text")
| DatasetDict({
| train: Dataset({
| features: ['text'],
| num_rows: 8530
| })
| validation: Dataset({
| features: ['text'],
| num_rows: 1066
| })
| test: Dataset({
| features: ['text'],
| num_rows: 1066
| })
| })
| ```
|
| set_format(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs)
| Set `__getitem__` return format (type and columns).
| The format is set for every dataset in the dataset dictionary.
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`list[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to False):
| Keep un-formatted columns as well in the output (as python objects),
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| It is possible to call `map` after calling `set_format`. Since `map` may add new columns, then the list of formatted columns
| gets updated. In this case, if you apply `map` on a dataset to add a new column, then this column will be formatted:
|
| `new formatted columns = (all columns - previously unformatted columns)`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x["text"], truncation=True, padding=True), batched=True)
| >>> ds.set_format(type="numpy", columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
| >>> ds["train"].format
| {'columns': ['input_ids', 'token_type_ids', 'attention_mask', 'label'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'numpy'}
| ```
|
| set_transform(self, transform: Optional[Callable], columns: Optional[list] = None, output_all_columns: bool = False)
| Set ``__getitem__`` return format using this transform. The transform is applied on-the-fly on batches when ``__getitem__`` is called.
| The transform is set for every dataset in the dataset dictionary
| As :func:`datasets.Dataset.set_format`, this can be reset using :func:`datasets.Dataset.reset_format`
|
| Args:
| transform (`Callable`, optional): user-defined formatting transform, replaces the format defined by :func:`datasets.Dataset.set_format`
| A formatting function is a callable that takes a batch (as a dict) as input and returns a batch.
| This function is applied right before returning the objects in ``__getitem__``.
| columns (`list[str]`, optional): columns to format in the output
| If specified, then the input batch of the transform only contains those columns.
| output_all_columns (`bool`, default to False): keep un-formatted columns as well in the output (as python objects)
| If set to True, then the other un-formatted columns are kept with the output of the transform.
|
| shuffle(self, seeds: Union[int, dict[str, Optional[int]], NoneType] = None, seed: Optional[int] = None, generators: Optional[dict[str, numpy.random._generator.Generator]] = None, keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, indices_cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000) -> 'DatasetDict'
| Create a new Dataset where the rows are shuffled.
|
| The transformation is applied to all the datasets of the dataset dictionary.
|
| Currently shuffling uses numpy random generators.
| You can either supply a NumPy BitGenerator to use, or a seed to initiate NumPy's default random generator (PCG64).
|
| Args:
| seeds (`Dict[str, int]` or `int`, *optional*):
| A seed to initialize the default BitGenerator if `generator=None`.
| If `None`, then fresh, unpredictable entropy will be pulled from the OS.
| If an `int` or `array_like[ints]` is passed, then it will be passed to SeedSequence to derive the initial BitGenerator state.
| You can provide one `seed` per dataset in the dataset dictionary.
| seed (`int`, *optional*):
| A seed to initialize the default BitGenerator if `generator=None`. Alias for seeds (a `ValueError` is raised if both are provided).
| generators (`Dict[str, *optional*, np.random.Generator]`):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| You have to provide one `generator` per dataset in the dataset dictionary.
| keep_in_memory (`bool`, defaults to `False`):
| Keep the dataset in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the current computation from `function`
| can be identified, use it instead of recomputing.
| indices_cache_file_names (`Dict[str, str]`, *optional*):
| Provide the name of a path for the cache file. It is used to store the
| indices mappings instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| This value is a good trade-off between memory usage during the processing, and processing speed.
| Higher value makes the processing do fewer lookups, lower value consume less temporary memory while running `map`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds["train"]["label"][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
|
| # set a seed
| >>> shuffled_ds = ds.shuffle(seed=42)
| >>> shuffled_ds["train"]["label"][:10]
| [0, 1, 0, 1, 0, 0, 0, 0, 0, 0]
| ```
|
| sort(self, column_names: Union[str, collections.abc.Sequence[str]], reverse: Union[bool, collections.abc.Sequence[bool]] = False, null_placement: str = 'at_end', keep_in_memory: bool = False, load_from_cache_file: Optional[bool] = None, indices_cache_file_names: Optional[dict[str, Optional[str]]] = None, writer_batch_size: Optional[int] = 1000) -> 'DatasetDict'
| Create a new dataset sorted according to a single or multiple columns.
|
| Args:
| column_names (`Union[str, Sequence[str]]`):
| Column name(s) to sort by.
| reverse (`Union[bool, Sequence[bool]]`, defaults to `False`):
| If `True`, sort by descending order rather than ascending. If a single bool is provided,
| the value is applied to the sorting of all column names. Otherwise a list of bools with the
| same length and order as column_names must be provided.
| null_placement (`str`, defaults to `at_end`):
| Put `None` values at the beginning if `at_start` or `first` or at the end if `at_end` or `last`
| keep_in_memory (`bool`, defaults to `False`):
| Keep the sorted indices in memory instead of writing it to a cache file.
| load_from_cache_file (`Optional[bool]`, defaults to `True` if caching is enabled):
| If a cache file storing the sorted indices
| can be identified, use it instead of recomputing.
| indices_cache_file_names (`[Dict[str, str]]`, *optional*, defaults to `None`):
| Provide the name of a path for the cache file. It is used to store the
| indices mapping instead of the automatically generated cache file name.
| You have to provide one `cache_file_name` per dataset in the dataset dictionary.
| writer_batch_size (`int`, defaults to `1000`):
| Number of rows per write operation for the cache file writer.
| Higher value gives smaller cache files, lower value consume less temporary memory.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset('cornell-movie-review-data/rotten_tomatoes')
| >>> ds['train']['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
| >>> sorted_ds = ds.sort('label')
| >>> sorted_ds['train']['label'][:10]
| [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
| >>> another_sorted_ds = ds.sort(['label', 'text'], reverse=[True, False])
| >>> another_sorted_ds['train']['label'][:10]
| [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
| ```
|
| unique(self, column: str) -> dict[str, list]
| Return a list of the unique elements in a column for each split.
|
| This is implemented in the low-level backend and as such, very fast.
|
| Args:
| column (`str`):
| column name (list all the column names with [`~datasets.DatasetDict.column_names`])
|
| Returns:
| Dict[`str`, `list`]: Dictionary of unique elements in the given column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.unique("label")
| {'test': [1, 0], 'train': [1, 0], 'validation': [1, 0]}
| ```
|
| with_format(self, type: Optional[str] = None, columns: Optional[list] = None, output_all_columns: bool = False, **format_kwargs) -> 'DatasetDict'
| Set `__getitem__` return format (type and columns). The data formatting is applied on-the-fly.
| The format `type` (for example "numpy") is used to format batches when using `__getitem__`.
| The format is set for every dataset in the dataset dictionary.
|
| It's also possible to use custom transforms for formatting using [`~datasets.Dataset.with_transform`].
|
| Contrary to [`~datasets.DatasetDict.set_format`], `with_format` returns a new [`DatasetDict`] object with new [`Dataset`] objects.
|
| Args:
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means `__getitem__` returns python objects (default).
| columns (`list[str]`, *optional*):
| Columns to format in the output.
| `None` means `__getitem__` returns all columns (default).
| output_all_columns (`bool`, defaults to `False`):
| Keep un-formatted columns as well in the output (as python objects).
| **format_kwargs (additional keyword arguments):
| Keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds["train"].format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': None}
| >>> ds = ds.with_format("torch")
| >>> ds["train"].format
| {'columns': ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
| 'format_kwargs': {},
| 'output_all_columns': False,
| 'type': 'torch'}
| >>> ds["train"][0]
| {'text': 'compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'label': tensor(1),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
| 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| with_transform(self, transform: Optional[Callable], columns: Optional[list] = None, output_all_columns: bool = False) -> 'DatasetDict'
| Set `__getitem__` return format using this transform. The transform is applied on-the-fly on batches when `__getitem__` is called.
| The transform is set for every dataset in the dataset dictionary
|
| As [`~datasets.Dataset.set_format`], this can be reset using [`~datasets.Dataset.reset_format`].
|
| Contrary to [`~datasets.DatasetDict.set_transform`], `with_transform` returns a new [`DatasetDict`] object with new [`Dataset`] objects.
|
| Args:
| transform (`Callable`, *optional*):
| User-defined formatting transform, replaces the format defined by [`~datasets.Dataset.set_format`].
| A formatting function is a callable that takes a batch (as a dict) as input and returns a batch.
| This function is applied right before returning the objects in `__getitem__`.
| columns (`list[str]`, *optional*):
| Columns to format in the output.
| If specified, then the input batch of the transform only contains those columns.
| output_all_columns (`bool`, defaults to False):
| Keep un-formatted columns as well in the output (as python objects).
| If set to `True`, then the other un-formatted columns are kept with the output of the transform.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> def encode(example):
| ... return tokenizer(example['text'], truncation=True, padding=True, return_tensors="pt")
| >>> ds = ds.with_transform(encode)
| >>> ds["train"][0]
| {'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 1, 1, 1, 1]),
| 'input_ids': tensor([ 101, 1103, 2067, 1110, 17348, 1106, 1129, 1103, 6880, 1432,
| 112, 188, 1207, 107, 14255, 1389, 107, 1105, 1115, 1119,
| 112, 188, 1280, 1106, 1294, 170, 24194, 1256, 3407, 1190,
| 170, 11791, 5253, 188, 1732, 7200, 10947, 12606, 2895, 117,
| 179, 7766, 118, 172, 15554, 1181, 3498, 6961, 3263, 1137,
| 188, 1566, 7912, 14516, 6997, 119, 102]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| from_csv(path_or_paths: dict[str, typing.Union[str, bytes, os.PathLike]], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, **kwargs) -> 'DatasetDict'
| Create [`DatasetDict`] from CSV file(s).
|
| Args:
| path_or_paths (`dict` of path-like):
| Path(s) of the CSV file(s).
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (str, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`pandas.read_csv`].
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import DatasetDict
| >>> ds = DatasetDict.from_csv({'train': 'path/to/dataset.csv'})
| ```
|
| from_json(path_or_paths: dict[str, typing.Union[str, bytes, os.PathLike]], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, **kwargs) -> 'DatasetDict'
| Create [`DatasetDict`] from JSON Lines file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the JSON Lines file(s).
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (str, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`JsonConfig`].
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import DatasetDict
| >>> ds = DatasetDict.from_json({'train': 'path/to/dataset.json'})
| ```
|
| from_parquet(path_or_paths: dict[str, typing.Union[str, bytes, os.PathLike]], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, columns: Optional[list[str]] = None, **kwargs) -> 'DatasetDict'
| Create [`DatasetDict`] from Parquet file(s).
|
| Args:
| path_or_paths (`dict` of path-like):
| Path(s) of the CSV file(s).
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| columns (`list[str]`, *optional*):
| If not `None`, only these columns will be read from the file.
| A column name may be a prefix of a nested field, e.g. 'a' will select
| 'a.b', 'a.c', and 'a.d.e'.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`ParquetConfig`].
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import DatasetDict
| >>> ds = DatasetDict.from_parquet({'train': 'path/to/dataset/parquet'})
| ```
|
| from_text(path_or_paths: dict[str, typing.Union[str, bytes, os.PathLike]], features: Optional[datasets.features.features.Features] = None, cache_dir: str = None, keep_in_memory: bool = False, **kwargs) -> 'DatasetDict'
| Create [`DatasetDict`] from text file(s).
|
| Args:
| path_or_paths (`dict` of path-like):
| Path(s) of the text file(s).
| features ([`Features`], *optional*):
| Dataset features.
| cache_dir (`str`, *optional*, defaults to `"~/.cache/huggingface/datasets"`):
| Directory to cache data.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`TextConfig`].
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import DatasetDict
| >>> ds = DatasetDict.from_text({'train': 'path/to/dataset.txt'})
| ```
|
| load_from_disk(dataset_dict_path: Union[str, bytes, os.PathLike], keep_in_memory: Optional[bool] = None, storage_options: Optional[dict] = None) -> 'DatasetDict'
| Load a dataset that was previously saved using [`save_to_disk`] from a filesystem using `fsspec.spec.AbstractFileSystem`.
|
| Args:
| dataset_dict_path (`path-like`):
| Path (e.g. `"dataset/train"`) or remote URI (e.g. `"s3//my-bucket/dataset/train"`)
| of the dataset dict directory where the dataset dict will be loaded from.
| keep_in_memory (`bool`, defaults to `None`):
| Whether to copy the dataset in-memory. If `None`, the
| dataset will not be copied in-memory unless explicitly enabled by setting
| `datasets.config.IN_MEMORY_MAX_SIZE` to nonzero. See more details in the
| [improve performance](../cache#improve-performance) section.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.8.0"/>
|
| Returns:
| [`DatasetDict`]
|
| Example:
|
| ```py
| >>> ds = load_from_disk('path/to/dataset/directory')
| ```
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| cache_files
| The cache files containing the Apache Arrow table backing each split.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.cache_files
| {'test': [{'filename': '/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46/rotten_tomatoes_movie_review-test.arrow'}],
| 'train': [{'filename': '/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46/rotten_tomatoes_movie_review-train.arrow'}],
| 'validation': [{'filename': '/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46/rotten_tomatoes_movie_review-validation.arrow'}]}
| ```
|
| column_names
| Names of the columns in each split of the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.column_names
| {'test': ['text', 'label'],
| 'train': ['text', 'label'],
| 'validation': ['text', 'label']}
| ```
|
| data
| The Apache Arrow tables backing each split.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.data
| ```
|
| num_columns
| Number of columns in each split of the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.num_columns
| {'test': 2, 'train': 2, 'validation': 2}
| ```
|
| num_rows
| Number of rows in each split of the dataset.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.num_rows
| {'test': 1066, 'train': 8530, 'validation': 1066}
| ```
|
| shape
| Shape of each split of the dataset (number of rows, number of columns).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.shape
| {'test': (1066, 2), 'train': (8530, 2), 'validation': (1066, 2)}
| ```
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (dict[typing.Union[str, datasets.splits.NamedSplit], ...
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __gt__(self, value, /)
| Return self>value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class IterableDatasetDict(builtins.dict)
| Method resolution order:
| IterableDatasetDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __repr__(self)
| Return repr(self).
|
| cast(self, features: datasets.features.features.Features) -> 'IterableDatasetDict'
| Cast the dataset to a new set of features.
| The type casting is applied to all the datasets of the dataset dictionary.
|
| Args:
| features (`Features`):
| New features to cast the dataset to.
| The name of the fields in the features must match the current column names.
| The type of the data must also be convertible from one type to the other.
| For non-trivial conversion, e.g. `string` <-> `ClassLabel` you should use [`map`] to update the Dataset.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset with casted features.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds["train"].features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> new_features = ds["train"].features.copy()
| >>> new_features['label'] = ClassLabel(names=['bad', 'good'])
| >>> new_features['text'] = Value('large_string')
| >>> ds = ds.cast(new_features)
| >>> ds["train"].features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('large_string')}
| ```
|
| cast_column(self, column: str, feature: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti]) -> 'IterableDatasetDict'
| Cast column to feature for decoding.
| The type casting is applied to all the datasets of the dataset dictionary.
|
| Args:
| column (`str`):
| Column name.
| feature ([`Feature`]):
| Target feature.
|
| Returns:
| [`IterableDatasetDict`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds["train"].features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> ds = ds.cast_column('label', ClassLabel(names=['bad', 'good']))
| >>> ds["train"].features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('string')}
| ```
|
| filter(self, function: Optional[Callable] = None, with_indices=False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None) -> 'IterableDatasetDict'
| Apply a filter function to all the elements so that the dataset only includes examples according to the filter function.
| The filtering is done on-the-fly when iterating over the dataset.
| The filtering is applied to all the datasets of the dataset dictionary.
|
| Args:
| function (`Callable`):
| Callable with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> bool` if `with_indices=False, batched=False`
| - `function(example: Dict[str, Any], indices: int) -> bool` if `with_indices=True, batched=False`
| - `function(example: Dict[str, list]) -> list[bool]` if `with_indices=False, batched=True`
| - `function(example: Dict[str, list], indices: list[int]) -> list[bool]` if `with_indices=True, batched=True`
|
| If no function is provided, defaults to an always True function: `lambda x: True`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx): ...`.
| input_columns (`str` or `list[str]`, *optional*):
| The columns to be passed into `function` as
| positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.filter(lambda x: x["label"] == 0)
| >>> list(ds["train"].take(3))
| [{'label': 0, 'text': 'Review: simplistic , silly and tedious .'},
| {'label': 0,
| 'text': "Review: it's so laddish and juvenile , only teenage boys could possibly find it funny ."},
| {'label': 0,
| 'text': 'Review: exploitative and largely devoid of the depth or sophistication that would make watching such a graphic treatment of the crimes bearable .'}]
| ```
|
| map(self, function: Optional[Callable] = None, with_indices: bool = False, with_split: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: int = 1000, drop_last_batch: bool = False, remove_columns: Union[str, list[str], NoneType] = None, fn_kwargs: Optional[dict] = None) -> 'IterableDatasetDict'
| Apply a function to all the examples in the iterable dataset (individually or in batches) and update them.
| If your function returns a column that already exists, then it overwrites it.
| The function is applied on-the-fly on the examples when iterating over the dataset.
| The transformation is applied to all the datasets of the dataset dictionary.
|
| You can specify whether the function should be batched or not with the `batched` parameter:
|
| - If batched is `False`, then the function takes 1 example in and should return 1 example.
| An example is a dictionary, e.g. `{"text": "Hello there !"}`.
| - If batched is `True` and `batch_size` is 1, then the function takes a batch of 1 example as input and can return a batch with 1 or more examples.
| A batch is a dictionary, e.g. a batch of 1 example is `{"text": ["Hello there !"]}`.
| - If batched is `True` and `batch_size` is `n` > 1, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.
| Note that the last batch may have less than `n` examples.
| A batch is a dictionary, e.g. a batch of `n` examples is `{"text": ["Hello there !"] * n}`.
|
| If the function is asynchronous, then `map` will run your function in parallel, with up to one thousand simultaneous calls.
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`, *optional*, defaults to `None`):
| Function applied on-the-fly on the examples when you iterate on the dataset.
| It must have one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> Dict[str, Any]` if `batched=False` and `with_indices=False`
| - `function(example: Dict[str, Any], idx: int) -> Dict[str, Any]` if `batched=False` and `with_indices=True`
| - `function(batch: Dict[str, list]) -> Dict[str, list]` if `batched=True` and `with_indices=False`
| - `function(batch: Dict[str, list], indices: list[int]) -> Dict[str, list]` if `batched=True` and `with_indices=True`
|
| For advanced usage, the function can also return a `pyarrow.Table`.
| If the function is asynchronous, then `map` will run your function in parallel.
| Moreover if your function returns nothing (`None`), then `map` will run your function and return the dataset unchanged.
| If no function is provided, default to identity function: `lambda x: x`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx[, rank]): ...`.
| input_columns (`[Union[str, list[str]]]`, *optional*, defaults to `None`):
| The columns to be passed into `function`
| as positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| drop_last_batch (`bool`, defaults to `False`):
| Whether a last batch smaller than the `batch_size` should be
| dropped instead of being processed by the function.
| remove_columns (`[list[str]]`, *optional*, defaults to `None`):
| Remove a selection of columns while doing the mapping.
| Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding
| columns with names in `remove_columns`, these columns will be kept.
| fn_kwargs (`Dict`, *optional*, defaults to `None`):
| Keyword arguments to be passed to `function`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> def add_prefix(example):
| ... example["text"] = "Review: " + example["text"]
| ... return example
| >>> ds = ds.map(add_prefix)
| >>> next(iter(ds["train"]))
| {'label': 1,
| 'text': 'Review: the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| push_to_hub(self, repo_id: str, config_name: str = 'default', set_default: Optional[bool] = None, data_dir: Optional[str] = None, commit_message: Optional[str] = None, commit_description: Optional[str] = None, private: Optional[bool] = None, token: Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = False, max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[dict[str, int]] = None, embed_external_files: bool = True, num_proc: Optional[int] = None) -> huggingface_hub.hf_api.CommitInfo
| Pushes the [`IterableDatasetDict`] to the hub as a Parquet dataset.
| The [`IterableDatasetDict`] is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
|
| Each dataset split will be pushed independently. The pushed dataset will keep the original split names.
|
| The resulting Parquet files are self-contained by default: if your dataset contains [`Image`] or [`Audio`]
| data, the Parquet files will store the bytes of your images or audio files.
| You can disable this by setting `embed_external_files` to False.
|
| Args:
| repo_id (`str`):
| The ID of the repository to push to in the following format: `<user>/<dataset_name>` or
| `<org>/<dataset_name>`. Also accepts `<dataset_name>`, which will default to the namespace
| of the logged-in user.
|
| It could also be a location inside a bucket, e.g. `buckets/<user_or_org>/<bucket_name>/...`
| config_name (`str`):
| Configuration name of a dataset. Defaults to "default".
| set_default (`bool`, *optional*):
| Whether to set this configuration as the default one. Otherwise, the default configuration is the one
| named "default".
| data_dir (`str`, *optional*):
| Directory name that will contain the uploaded data files. Defaults to the `config_name` if different
| from "default", else "data".
|
| <Added version="2.17.0"/>
| commit_message (`str`, *optional*):
| Message to commit while pushing. Will default to `"Upload dataset"`.
| commit_description (`str`, *optional*):
| Description of the commit that will be created.
| Additionally, description of the PR if a PR is created (`create_pr` is True).
|
| <Added version="2.16.0"/>
| private (`bool`, *optional*):
| Whether to make the repo private. If `None` (default), the repo will be public unless the
| organization's default is private. This value is ignored if the repo already exists.
| token (`str`, *optional*):
| An optional authentication token for the Hugging Face Hub. If no token is passed, will default
| to the token saved locally when logging in with `huggingface-cli login`. Will raise an error
| if no token is passed and the user is not logged-in.
| revision (`str`, *optional*):
| Branch to push the uploaded files to. Defaults to the `"main"` branch.
| create_pr (`bool`, *optional*, defaults to `False`):
| Whether to create a PR with the uploaded files or directly commit.
| max_shard_size (`int` or `str`, *optional*):
| Optional maximum size of the dataset shards to be uploaded to the hub. If expressed as a string, needs to be digits followed
| by a unit (like `"500MB"` or `"1GB"`). If not provided, each split keeps its dataset-native shard count.
| num_shards (`Dict[str, int]`, *optional*):
| Number of shards to write. Use a dictionary to define a different `num_shards` for each split.
| If `max_shard_size` is provided and a split's `num_shards` is not, then the number of shards for that split is estimated
| from `max_shard_size`.
| embed_external_files (`bool`, defaults to `True`):
| Whether to embed file bytes in the shards.
| In particular, this will do the following before the push for the fields of type:
|
| - [`Audio`] and [`Image`] removes local path information and embed file content in the Parquet files.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when preparing and uploading the dataset.
| This is helpful if the dataset is made of many samples or media files to embed.
| I uses "spawn" context to work with hf_xet, the rust client for fast uploads to HF.
| Multiprocessing is disabled by default.
|
| <Added version="4.0.0"/>
|
| Return:
| huggingface_hub.CommitInfo
|
| Example:
|
| ```python
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", private=True)
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", max_shard_size="1GB")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", num_shards={"train": 1024, "test": 8})
| ```
|
| If you want to add a new configuration (or subset) to a dataset (e.g. if the dataset has multiple tasks/versions/languages):
|
| ```python
| >>> english_dataset.push_to_hub("<organization>/<dataset_id>", "en")
| >>> french_dataset.push_to_hub("<organization>/<dataset_id>", "fr")
| >>> # later
| >>> english_dataset = load_dataset("<organization>/<dataset_id>", "en")
| >>> french_dataset = load_dataset("<organization>/<dataset_id>", "fr")
| ```
|
| remove_columns(self, column_names: Union[str, list[str]]) -> 'IterableDatasetDict'
| Remove one or several column(s) in the dataset and the features associated to them.
| The removal is done on-the-fly on the examples when iterating over the dataset.
| The removal is applied to all the datasets of the dataset dictionary.
|
|
| Args:
| column_names (`Union[str, list[str]]`):
| Name of the column(s) to remove.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset object without the columns to remove.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.remove_columns("label")
| >>> next(iter(ds["train"]))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| rename_column(self, original_column_name: str, new_column_name: str) -> 'IterableDatasetDict'
| Rename a column in the dataset, and move the features associated to the original column under the new column
| name.
| The renaming is applied to all the datasets of the dataset dictionary.
|
| Args:
| original_column_name (`str`):
| Name of the column to rename.
| new_column_name (`str`):
| New name for the column.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset with a renamed column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.rename_column("text", "movie_review")
| >>> next(iter(ds["train"]))
| {'label': 1,
| 'movie_review': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| rename_columns(self, column_mapping: dict[str, str]) -> 'IterableDatasetDict'
| Rename several columns in the dataset, and move the features associated to the original columns under
| the new column names.
| The renaming is applied to all the datasets of the dataset dictionary.
|
| Args:
| column_mapping (`Dict[str, str]`):
| A mapping of columns to rename to their new names.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset with renamed columns
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.rename_columns({"text": "movie_review", "label": "rating"})
| >>> next(iter(ds["train"]))
| {'movie_review': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',
| 'rating': 1}
| ```
|
| select_columns(self, column_names: Union[str, list[str]]) -> 'IterableDatasetDict'
| Select one or several column(s) in the dataset and the features
| associated to them. The selection is done on-the-fly on the examples
| when iterating over the dataset. The selection is applied to all the
| datasets of the dataset dictionary.
|
|
| Args:
| column_names (`Union[str, list[str]]`):
| Name of the column(s) to keep.
|
| Returns:
| [`IterableDatasetDict`]: A copy of the dataset object with only selected columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> ds = ds.select("text")
| >>> next(iter(ds["train"]))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| shuffle(self, seed=None, generator: Optional[numpy.random._generator.Generator] = None, buffer_size: int = 1000) -> 'IterableDatasetDict'
| Randomly shuffles the elements of this dataset.
| The shuffling is applied to all the datasets of the dataset dictionary.
|
| This dataset fills a buffer with buffer_size elements, then randomly samples elements from this buffer,
| replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or
| equal to the full size of the dataset is required.
|
| For instance, if your dataset contains 10,000 elements but `buffer_size` is set to 1000, then `shuffle` will
| initially select a random element from only the first 1000 elements in the buffer. Once an element is
| selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element,
| maintaining the 1000 element buffer.
|
| If the dataset is made of several shards, it also does `shuffle` the order of the shards.
| However if the order has been fixed by using [`~datasets.IterableDataset.skip`] or [`~datasets.IterableDataset.take`]
| then the order of the shards is kept unchanged.
|
| Args:
| seed (`int`, *optional*, defaults to `None`):
| Random seed that will be used to shuffle the dataset.
| It is used to sample from the shuffle buffer and also to shuffle the data shards.
| generator (`numpy.random.Generator`, *optional*):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| buffer_size (`int`, defaults to `1000`):
| Size of the buffer.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", streaming=True)
| >>> list(ds["train"].take(3))
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| >>> ds = ds.shuffle(seed=42)
| >>> list(ds["train"].take(3))
| [{'label': 1,
| 'text': "a sports movie with action that's exciting on the field and a story you care about off it ."},
| {'label': 1,
| 'text': 'at its best , the good girl is a refreshingly adult take on adultery . . .'},
| {'label': 1,
| 'text': "sam jones became a very lucky filmmaker the day wilco got dropped from their record label , proving that one man's ruin may be another's fortune ."}]
| ```
|
| with_format(self, type: Optional[str] = None) -> 'IterableDatasetDict'
| Return a dataset with the specified format.
|
| Args:
|
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means it returns python objects (default).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation", streaming=True)
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds = ds.with_format("torch")
| >>> next(iter(ds))
| {'text': 'compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'label': tensor(1),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
| 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| column_names
| Names of the columns in each split of the dataset.
| This can contain None valies if some splits have unknown features (e.g. after a map() operation).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.column_names
| {'test': ['text', 'label'],
| 'train': ['text', 'label'],
| 'validation': ['text', 'label']}
| ```
|
| num_columns
| Number of columns in each split of the dataset.
| This can contain None valies if some splits have unknown features (e.g. after a map() operation).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes")
| >>> ds.num_columns
| {'test': 2, 'train': 2, 'validation': 2}
| ```
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (dict[typing.Union[str, datasets.splits.NamedSplit], ...
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class bind(functools.partial)
| Method resolution order:
| bind
| functools.partial
| builtins.object
|
| Methods defined here:
|
| __call__(self, *fn_args, **fn_kwargs)
| Call self as a function.
|
| ----------------------------------------------------------------------
| Methods inherited from functools.partial:
|
| __delattr__(self, name, /)
| Implement delattr(self, name).
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setattr__(self, name, value, /)
| Implement setattr(self, name, value).
|
| __setstate__(...)
|
| ----------------------------------------------------------------------
| Class methods inherited from functools.partial:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Static methods inherited from functools.partial:
|
| __new__(*args, **kwargs) class method of functools.partial
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from functools.partial:
|
| __dict__
|
| __vectorcalloffset__
|
| args
| tuple of arguments to future partial calls
|
| func
| function object to use in future partial calls
|
| keywords
| dictionary of keyword arguments to future partial calls
DATA
Callable = typing.Callable
Deprecated alias to collections.abc.Callable.
Callable[[int], str] signifies a function that takes a single
parameter of type int and returns a str.
The subscription syntax must always be used with exactly two
values: the argument list and the return type.
The argument list must be a list of types, a ParamSpec,
Concatenate or ellipsis. The return type must be a single type.
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
FeatureType = typing.Union[dict, list, tuple, datasets.feature....feat...
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
PathLike = typing.Union[str, bytes, os.PathLike]
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.dataset_dict (WARNING)>
VERSION
5.0.1
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\dataset_dict.py
datasets.disable_caching
Python Library Documentation: function disable_caching in module datasets.fingerprint
disable_caching()
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use [`~datasets.Dataset.save_to_disk`] to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use
the `download_mode` parameter in [`~datasets.load_dataset`].
datasets.disable_progress_bar
Python Library Documentation: function disable_progress_bars in module datasets.utils.tqdm
disable_progress_bars() -> None
Disable globally progress bars used in `datasets` except if `HF_DATASETS_DISABLE_PROGRESS_BARS` environment
variable has been set.
Use [`~utils.enable_progress_bars`] to re-enable them.
datasets.disable_progress_bars
Python Library Documentation: function disable_progress_bars in module datasets.utils.tqdm
disable_progress_bars() -> None
Disable globally progress bars used in `datasets` except if `HF_DATASETS_DISABLE_PROGRESS_BARS` environment
variable has been set.
Use [`~utils.enable_progress_bars`] to re-enable them.
datasets.doc_utils
Python Library Documentation: module datasets.utils.doc_utils in datasets.utils
NAME
datasets.utils.doc_utils
FUNCTIONS
is_documented_by(function_with_docstring: Callable)
Decorator to share docstrings across common functions.
Args:
function_with_docstring (`Callable`): Name of the function with the docstring.
DATA
Callable = typing.Callable
Deprecated alias to collections.abc.Callable.
Callable[[int], str] signifies a function that takes a single
parameter of type int and returns a str.
The subscription syntax must always be used with exactly two
values: the argument list and the return type.
The argument list must be a list of types, a ParamSpec,
Concatenate or ellipsis. The return type must be a single type.
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\doc_utils.py
datasets.download
Python Library Documentation: package datasets.download in datasets
NAME
datasets.download
PACKAGE CONTENTS
download_config
download_manager
streaming_download_manager
CLASSES
builtins.object
datasets.download.download_config.DownloadConfig
datasets.download.download_manager.DownloadManager
datasets.download.streaming_download_manager.StreamingDownloadManager
enum.Enum(builtins.object)
datasets.download.download_manager.DownloadMode
class DownloadConfig(builtins.object)
| DownloadConfig(cache_dir: Union[str, pathlib.Path, NoneType] = None, force_download: bool = False, resume_download: bool = False, local_files_only: bool = False, proxies: Optional[dict] = None, user_agent: Optional[str] = None, extract_compressed_file: bool = False, force_extract: bool = False, delete_extracted: bool = False, extract_on_the_fly: bool = False, use_etag: bool = True, num_proc: Optional[int] = None, max_retries: int = 1, token: Union[str, bool, NoneType] = None, storage_options: dict[str, typing.Any] = <factory>, download_desc: Optional[str] = None, disable_tqdm: bool = False) -> None
|
| Configuration for our cached path manager.
|
| Attributes:
| cache_dir (`str` or `Path`, *optional*):
| Specify a cache directory to save the file to (overwrite the
| default cache dir).
| force_download (`bool`, defaults to `False`):
| If `True`, re-download the file even if it's already cached in
| the cache dir.
| resume_download (`bool`, defaults to `False`):
| If `True`, resume the download if an incompletely received file is
| found.
| proxies (`dict`, *optional*):
| user_agent (`str`, *optional*):
| Optional string or dict that will be appended to the user-agent on remote
| requests.
| extract_compressed_file (`bool`, defaults to `False`):
| If `True` and the path point to a zip or tar file,
| extract the compressed file in a folder along the archive.
| force_extract (`bool`, defaults to `False`):
| If `True` when `extract_compressed_file` is `True` and the archive
| was already extracted, re-extract the archive and override the folder where it was extracted.
| delete_extracted (`bool`, defaults to `False`):
| Whether to delete (or keep) the extracted files.
| extract_on_the_fly (`bool`, defaults to `False`):
| If `True`, extract compressed files while they are being read.
| use_etag (`bool`, defaults to `True`):
| Whether to use the ETag HTTP response header to validate the cached files.
| num_proc (`int`, *optional*):
| The number of processes to launch to download the files in parallel.
| max_retries (`int`, default to `1`):
| The number of times to retry an HTTP request if it fails.
| token (`str` or `bool`, *optional*):
| Optional string or boolean to use as Bearer token
| for remote files on the Datasets Hub. If `True`, or not specified, will get token from `~/.huggingface`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the dataset file-system backend, if any.
| download_desc (`str`, *optional*):
| A description to be displayed alongside with the progress bar while downloading the files.
| disable_tqdm (`bool`, defaults to `False`):
| Whether to disable the individual files download progress bar
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, cache_dir: Union[str, pathlib.Path, NoneType] = None, force_download: bool = False, resume_download: bool = False, local_files_only: bool = False, proxies: Optional[dict] = None, user_agent: Optional[str] = None, extract_compressed_file: bool = False, force_extract: bool = False, delete_extracted: bool = False, extract_on_the_fly: bool = False, use_etag: bool = True, num_proc: Optional[int] = None, max_retries: int = 1, token: Union[str, bool, NoneType] = None, storage_options: dict[str, typing.Any] = <factory>, download_desc: Optional[str] = None, disable_tqdm: bool = False) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| __setattr__(self, name, value)
| Implement setattr(self, name, value).
|
| copy(self) -> 'DownloadConfig'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'cache_dir': typing.Union[str, pathlib.Path, NoneTy...
|
| __dataclass_fields__ = {'cache_dir': Field(name='cache_dir',type=typin...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('cache_dir', 'force_download', 'resume_download', 'l...
|
| cache_dir = None
|
| delete_extracted = False
|
| disable_tqdm = False
|
| download_desc = None
|
| extract_compressed_file = False
|
| extract_on_the_fly = False
|
| force_download = False
|
| force_extract = False
|
| local_files_only = False
|
| max_retries = 1
|
| num_proc = None
|
| proxies = None
|
| resume_download = False
|
| token = None
|
| use_etag = True
|
| user_agent = None
class DownloadManager(builtins.object)
| DownloadManager(dataset_name: Optional[str] = None, data_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, base_path: Optional[str] = None, record_checksums=False)
|
| Methods defined here:
|
| __init__(self, dataset_name: Optional[str] = None, data_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, base_path: Optional[str] = None, record_checksums=False)
| Download manager constructor.
|
| Args:
| data_dir:
| can be used to specify a manual directory to get the files from.
| dataset_name (`str`):
| name of dataset this instance will be used for. If
| provided, downloads will contain which datasets they were used for.
| download_config (`DownloadConfig`):
| to specify the cache directory and other
| download options
| base_path (`str`):
| base path that is used when relative paths are used to
| download files. This can be a remote url.
| record_checksums (`bool`, defaults to `False`):
| Whether to record the checksums of the downloaded files. If None, the value is inferred from the builder.
|
| delete_extracted_files(self)
|
| download(self, url_or_urls)
| Download given URL(s).
|
| By default, only one process is used for download. Pass customized `download_config.num_proc` to change this behavior.
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL or `list` or `dict` of URLs to download. Each URL is a `str`.
|
| Returns:
| `str` or `list` or `dict`:
| The downloaded paths matching the given input `url_or_urls`.
|
| Example:
|
| ```py
| >>> downloaded_files = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| ```
|
| download_and_extract(self, url_or_urls)
| Download and extract given `url_or_urls`.
|
| Is roughly equivalent to:
|
| ```
| extracted_paths = dl_manager.extract(dl_manager.download(url_or_urls))
| ```
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL or `list` or `dict` of URLs to download and extract. Each URL is a `str`.
|
| Returns:
| extracted_path(s): `str`, extracted paths of given URL(s).
|
| extract(self, path_or_paths)
| Extract given path(s).
|
| Args:
| path_or_paths (path or `list` or `dict`):
| Path of file to extract. Each path is a `str`.
|
| Returns:
| extracted_path(s): `str`, The extracted paths matching the given input
| path_or_paths.
|
| Example:
|
| ```py
| >>> downloaded_files = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| >>> extracted_files = dl_manager.extract(downloaded_files)
| ```
|
| get_recorded_sizes_checksums(self)
|
| iter_archive(self, path_or_buf: Union[str, _io.BufferedReader])
| Iterate over files within an archive.
|
| Args:
| path_or_buf (`str` or `io.BufferedReader`):
| Archive path or archive binary file object.
|
| Yields:
| `tuple[str, io.BufferedReader]`:
| 2-tuple (path_within_archive, file_object).
| File object is opened in binary mode.
|
| Example:
|
| ```py
| >>> archive = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| >>> files = dl_manager.iter_archive(archive)
| ```
|
| iter_files(self, paths: Union[str, list[str]])
| Iterate over file paths.
|
| Args:
| paths (`str` or `list` of `str`):
| Root paths.
|
| Yields:
| `str`: File path.
|
| Example:
|
| ```py
| >>> files = dl_manager.download_and_extract('https://huggingface.co/datasets/AI-Lab-Makerere/beans/resolve/main/data/train.zip')
| >>> files = dl_manager.iter_files(files)
| ```
|
| manage_extracted_files(self)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| downloaded_size
| Returns the total size of downloaded files.
|
| manual_dir
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| is_streaming = False
class DownloadMode(enum.Enum)
| DownloadMode(*values)
|
| `Enum` for how to treat pre-existing downloads and data.
|
| The default mode is `REUSE_DATASET_IF_EXISTS`, which will reuse both
| raw downloads and the prepared dataset if they exist.
|
| The generations modes:
|
| | | Downloads | Dataset |
| |-------------------------------------|-----------|---------|
| | `REUSE_DATASET_IF_EXISTS` (default) | Reuse | Reuse |
| | `REUSE_CACHE_IF_EXISTS` | Reuse | Fresh |
| | `FORCE_REDOWNLOAD` | Fresh | Fresh |
|
| Method resolution order:
| DownloadMode
| enum.Enum
| builtins.object
|
| Data and other attributes defined here:
|
| FORCE_REDOWNLOAD = <DownloadMode.FORCE_REDOWNLOAD: 'force_redownload'>
|
| REUSE_CACHE_IF_EXISTS = <DownloadMode.REUSE_CACHE_IF_EXISTS: 'reuse_ca...
|
| REUSE_DATASET_IF_EXISTS = <DownloadMode.REUSE_DATASET_IF_EXISTS: 'reus...
|
| ----------------------------------------------------------------------
| Data descriptors inherited from enum.Enum:
|
| name
| The name of the Enum member.
|
| value
| The value of the Enum member.
|
| ----------------------------------------------------------------------
| Static methods inherited from enum.EnumType:
|
| __contains__(value)
| Return True if `value` is in `cls`.
|
| `value` is in `cls` if:
| 1) `value` is a member of `cls`, or
| 2) `value` is the value of one of the `cls`'s members.
|
| __getitem__(name)
| Return the member matching `name`.
|
| __iter__()
| Return members in definition order.
|
| __len__()
| Return the number of members (no aliases)
|
| ----------------------------------------------------------------------
| Readonly properties inherited from enum.EnumType:
|
| __members__
| Returns a mapping of member name->value.
|
| This mapping lists all enum members, including aliases. Note that this
| is a read-only view of the internal mapping.
class StreamingDownloadManager(builtins.object)
| StreamingDownloadManager(dataset_name: Optional[str] = None, data_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, base_path: Optional[str] = None)
|
| Download manager that uses the "::" separator to navigate through (possibly remote) compressed archives.
| Contrary to the regular `DownloadManager`, the `download` and `extract` methods don't actually download nor extract
| data, but they rather return the path or url that could be opened using the `xopen` function which extends the
| built-in `open` function to stream data from remote files.
|
| Methods defined here:
|
| __init__(self, dataset_name: Optional[str] = None, data_dir: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, base_path: Optional[str] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| download(self, url_or_urls)
| Normalize URL(s) of files to stream data from.
| This is the lazy version of `DownloadManager.download` for streaming.
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL(s) of files to stream data from. Each url is a `str`.
|
| Returns:
| url(s): (`str` or `list` or `dict`), URL(s) to stream data from matching the given input url_or_urls.
|
| Example:
|
| ```py
| >>> downloaded_files = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| ```
|
| download_and_extract(self, url_or_urls)
| Prepare given `url_or_urls` for streaming (add extraction protocol).
|
| This is the lazy version of `DownloadManager.download_and_extract` for streaming.
|
| Is equivalent to:
|
| ```
| urls = dl_manager.extract(dl_manager.download(url_or_urls))
| ```
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL(s) to stream from data from. Each url is a `str`.
|
| Returns:
| url(s): (`str` or `list` or `dict`), URL(s) to stream data from matching the given input `url_or_urls`.
|
| extract(self, url_or_urls)
| Add extraction protocol for given url(s) for streaming.
|
| This is the lazy version of `DownloadManager.extract` for streaming.
|
| Args:
| url_or_urls (`str` or `list` or `dict`):
| URL(s) of files to stream data from. Each url is a `str`.
|
| Returns:
| url(s): (`str` or `list` or `dict`), URL(s) to stream data from matching the given input `url_or_urls`.
|
| Example:
|
| ```py
| >>> downloaded_files = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| >>> extracted_files = dl_manager.extract(downloaded_files)
| ```
|
| get_recorded_sizes_checksums(self)
|
| iter_archive(self, urlpath_or_buf: Union[str, _io.BufferedReader]) -> collections.abc.Iterable[tuple]
| Iterate over files within an archive.
|
| Args:
| urlpath_or_buf (`str` or `io.BufferedReader`):
| Archive path or archive binary file object.
|
| Yields:
| `tuple[str, io.BufferedReader]`:
| 2-tuple (path_within_archive, file_object).
| File object is opened in binary mode.
|
| Example:
|
| ```py
| >>> archive = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
| >>> files = dl_manager.iter_archive(archive)
| ```
|
| iter_files(self, urlpaths: Union[str, list[str]]) -> collections.abc.Iterable[str]
| Iterate over files.
|
| Args:
| urlpaths (`str` or `list` of `str`):
| Root paths.
|
| Yields:
| str: File URL path.
|
| Example:
|
| ```py
| >>> files = dl_manager.download_and_extract('https://huggingface.co/datasets/AI-Lab-Makerere/beans/resolve/main/data/train.zip')
| >>> files = dl_manager.iter_files(files)
| ```
|
| manage_extracted_files(self)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| manual_dir
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| is_streaming = True
DATA
__all__ = ['DownloadConfig', 'DownloadManager', 'DownloadMode', 'Strea...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\download\__init__.py
datasets.enable_caching
Python Library Documentation: function enable_caching in module datasets.fingerprint
enable_caching()
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use [`~datasets.Dataset.save_to_disk`] to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use
the `download_mode` parameter in [`~datasets.load_dataset`].
datasets.enable_progress_bar
Python Library Documentation: function enable_progress_bars in module datasets.utils.tqdm
enable_progress_bars() -> None
Enable globally progress bars used in `datasets` except if `HF_DATASETS_DISABLE_PROGRESS_BARS` environment
variable has been set.
Use [`~utils.disable_progress_bars`] to disable them.
datasets.enable_progress_bars
Python Library Documentation: function enable_progress_bars in module datasets.utils.tqdm
enable_progress_bars() -> None
Enable globally progress bars used in `datasets` except if `HF_DATASETS_DISABLE_PROGRESS_BARS` environment
variable has been set.
Use [`~utils.disable_progress_bars`] to disable them.
datasets.exceptions
Python Library Documentation: module datasets.exceptions in datasets
NAME
datasets.exceptions
DESCRIPTION
# SPDX-License-Identifier: Apache-2.0
# Copyright 2023 The HuggingFace Authors.
CLASSES
builtins.Exception(builtins.BaseException)
DatasetsError
ChecksumVerificationError
ExpectedMoreDownloadedFilesError
NonMatchingChecksumError
UnexpectedDownloadedFileError
DatasetBuildError
DatasetGenerationError
DatasetGenerationCastError
FileFormatError
ManualDownloadError
DefunctDatasetError
FileNotFoundDatasetsError(DatasetsError, builtins.FileNotFoundError)
DataFilesNotFoundError
DatasetNotFoundError
SplitsVerificationError
ExpectedMoreSplitsError
NonMatchingSplitsSizesError
UnexpectedSplitsError
class ChecksumVerificationError(DatasetsError)
| Error raised during checksums verifications of downloaded files.
|
| Method resolution order:
| ChecksumVerificationError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DataFilesNotFoundError(FileNotFoundDatasetsError)
| No (supported) data files found.
|
| Method resolution order:
| DataFilesNotFoundError
| FileNotFoundDatasetsError
| DatasetsError
| builtins.FileNotFoundError
| builtins.OSError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from FileNotFoundDatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.FileNotFoundError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.OSError:
|
| __reduce__(...)
| Helper for pickle.
|
| __str__(self, /)
| Return str(self).
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.OSError:
|
| __new__(*args, **kwargs) class method of builtins.OSError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.OSError:
|
| characters_written
|
| errno
| POSIX exception code
|
| filename
| exception filename
|
| filename2
| second exception filename
|
| strerror
| exception strerror
|
| winerror
| Win32 exception code
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DatasetBuildError(DatasetsError)
| Method resolution order:
| DatasetBuildError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DatasetGenerationCastError(DatasetGenerationError)
| Method resolution order:
| DatasetGenerationCastError
| DatasetGenerationError
| DatasetBuildError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Class methods defined here:
|
| from_cast_error(cast_error: datasets.table.CastError, builder_name: str, gen_kwargs: dict[str, typing.Any], token: Union[str, bool, NoneType]) -> 'DatasetGenerationCastError'
|
| ----------------------------------------------------------------------
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DatasetGenerationError(DatasetBuildError)
| Method resolution order:
| DatasetGenerationError
| DatasetBuildError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DatasetNotFoundError(FileNotFoundDatasetsError)
| Dataset not found.
|
| Raised when trying to access:
| - a missing dataset, or
| - a private/gated dataset and the user is not authenticated.
|
| Method resolution order:
| DatasetNotFoundError
| FileNotFoundDatasetsError
| DatasetsError
| builtins.FileNotFoundError
| builtins.OSError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from FileNotFoundDatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.FileNotFoundError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.OSError:
|
| __reduce__(...)
| Helper for pickle.
|
| __str__(self, /)
| Return str(self).
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.OSError:
|
| __new__(*args, **kwargs) class method of builtins.OSError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.OSError:
|
| characters_written
|
| errno
| POSIX exception code
|
| filename
| exception filename
|
| filename2
| second exception filename
|
| strerror
| exception strerror
|
| winerror
| Win32 exception code
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DatasetsError(builtins.Exception)
| Base class for exceptions in this library.
|
| Method resolution order:
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DefunctDatasetError(DatasetsError)
| The dataset has been defunct.
|
| Method resolution order:
| DefunctDatasetError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ExpectedMoreDownloadedFilesError(ChecksumVerificationError)
| Some files were supposed to be downloaded but were not.
|
| Method resolution order:
| ExpectedMoreDownloadedFilesError
| ChecksumVerificationError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ExpectedMoreSplitsError(SplitsVerificationError)
| Some recorded splits are missing.
|
| Method resolution order:
| ExpectedMoreSplitsError
| SplitsVerificationError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class FileFormatError(DatasetBuildError)
| Method resolution order:
| FileFormatError
| DatasetBuildError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class FileNotFoundDatasetsError(DatasetsError, builtins.FileNotFoundError)
| FileNotFoundError raised by this library.
|
| Method resolution order:
| FileNotFoundDatasetsError
| DatasetsError
| builtins.FileNotFoundError
| builtins.OSError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.FileNotFoundError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.OSError:
|
| __reduce__(...)
| Helper for pickle.
|
| __str__(self, /)
| Return str(self).
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.OSError:
|
| __new__(*args, **kwargs) class method of builtins.OSError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.OSError:
|
| characters_written
|
| errno
| POSIX exception code
|
| filename
| exception filename
|
| filename2
| second exception filename
|
| strerror
| exception strerror
|
| winerror
| Win32 exception code
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ManualDownloadError(DatasetBuildError)
| Method resolution order:
| ManualDownloadError
| DatasetBuildError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NonMatchingChecksumError(ChecksumVerificationError)
| The downloaded file checksum don't match the expected checksum.
|
| Method resolution order:
| NonMatchingChecksumError
| ChecksumVerificationError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NonMatchingSplitsSizesError(SplitsVerificationError)
| The splits sizes don't match the expected splits sizes.
|
| Method resolution order:
| NonMatchingSplitsSizesError
| SplitsVerificationError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class SplitsVerificationError(DatasetsError)
| Error raised during splits verifications.
|
| Method resolution order:
| SplitsVerificationError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class UnexpectedDownloadedFileError(ChecksumVerificationError)
| Some downloaded files were not expected.
|
| Method resolution order:
| UnexpectedDownloadedFileError
| ChecksumVerificationError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class UnexpectedSplitsError(SplitsVerificationError)
| The expected splits of the downloaded file is missing.
|
| Method resolution order:
| UnexpectedSplitsError
| SplitsVerificationError
| DatasetsError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors inherited from DatasetsError:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
DATA
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\exceptions.py
datasets.experimental
Python Library Documentation: function experimental in module datasets.utils.experimental
experimental(fn: Callable) -> Callable
Decorator to flag a feature as experimental.
An experimental feature trigger a warning when used as it might be subject to breaking changes in the future.
Args:
fn (`Callable`):
The function to flag as experimental.
Returns:
`Callable`: The decorated function.
Example:
```python
>>> from datasets.utils import experimental
>>> @experimental
... def my_function():
... print("Hello world!")
>>> my_function()
UserWarning: 'my_function' is experimental and might be subject to breaking changes in the future.
Hello world!
```
datasets.extract
Python Library Documentation: module datasets.utils.extract in datasets.utils
NAME
datasets.utils.extract
CLASSES
abc.ABC(builtins.object)
BaseExtractor
MagicNumberBaseExtractor(BaseExtractor, abc.ABC)
Bzip2Extractor
GzipExtractor
Lz4Extractor
RarExtractor
SevenZipExtractor
XzExtractor
ZipExtractor
ZstdExtractor
TarExtractor
builtins.object
ExtractManager
Extractor
class BaseExtractor(abc.ABC)
| Method resolution order:
| BaseExtractor
| abc.ABC
| builtins.object
|
| Class methods defined here:
|
| is_extractable(path: Union[pathlib.Path, str], **kwargs) -> bool
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'extract', 'is_extractable'})
class Bzip2Extractor(MagicNumberBaseExtractor)
| Method resolution order:
| Bzip2Extractor
| MagicNumberBaseExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| magic_numbers = [b'BZh']
|
| ----------------------------------------------------------------------
| Class methods inherited from MagicNumberBaseExtractor:
|
| is_extractable(path: Union[pathlib.Path, str], magic_number: bytes = b'') -> bool
|
| ----------------------------------------------------------------------
| Static methods inherited from MagicNumberBaseExtractor:
|
| read_magic_number(path: Union[pathlib.Path, str], magic_number_length: int)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class ExtractManager(builtins.object)
| ExtractManager(cache_dir: Optional[str] = None)
|
| Methods defined here:
|
| __init__(self, cache_dir: Optional[str] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| extract(self, input_path: str, force_extract: bool = False) -> str
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Extractor(builtins.object)
| Class methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str], extractor_format: str) -> None
|
| infer_extractor_format(path: Union[pathlib.Path, str]) -> Optional[str]
|
| is_extractable(path: Union[pathlib.Path, str], return_extractor: bool = False) -> bool
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'extractors': dict[str, type[datasets.utils.extract...
|
| extractors = {'7z': <class 'datasets.utils.extract.SevenZipExtractor'>...
class GzipExtractor(MagicNumberBaseExtractor)
| Method resolution order:
| GzipExtractor
| MagicNumberBaseExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| magic_numbers = [b'\x1f\x8b']
|
| ----------------------------------------------------------------------
| Class methods inherited from MagicNumberBaseExtractor:
|
| is_extractable(path: Union[pathlib.Path, str], magic_number: bytes = b'') -> bool
|
| ----------------------------------------------------------------------
| Static methods inherited from MagicNumberBaseExtractor:
|
| read_magic_number(path: Union[pathlib.Path, str], magic_number_length: int)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Lz4Extractor(MagicNumberBaseExtractor)
| Method resolution order:
| Lz4Extractor
| MagicNumberBaseExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| magic_numbers = [b'\x04"M\x18']
|
| ----------------------------------------------------------------------
| Class methods inherited from MagicNumberBaseExtractor:
|
| is_extractable(path: Union[pathlib.Path, str], magic_number: bytes = b'') -> bool
|
| ----------------------------------------------------------------------
| Static methods inherited from MagicNumberBaseExtractor:
|
| read_magic_number(path: Union[pathlib.Path, str], magic_number_length: int)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class MagicNumberBaseExtractor(BaseExtractor, abc.ABC)
| Method resolution order:
| MagicNumberBaseExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Class methods defined here:
|
| is_extractable(path: Union[pathlib.Path, str], magic_number: bytes = b'') -> bool
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| read_magic_number(path: Union[pathlib.Path, str], magic_number_length: int)
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'extract'})
|
| __annotations__ = {'magic_numbers': list[bytes]}
|
| magic_numbers = []
|
| ----------------------------------------------------------------------
| Static methods inherited from BaseExtractor:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class RarExtractor(MagicNumberBaseExtractor)
| Method resolution order:
| RarExtractor
| MagicNumberBaseExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| safemembers(members: list['rarfile.RarInfo'], output_path: Union[pathlib.Path, str])
| Fix for CVE-2007-4559
| Desc:
| Directory traversal vulnerability in the (1) extract and (2) extractall functions in the tarfile
| module in Python allows user-assisted remote attackers to overwrite arbitrary files via a .. (dot dot)
| sequence in filenames in a TAR archive, a related issue to CVE-2001-1267.
| See: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4559
| From: https://stackoverflow.com/a/10077309
|
| This additional mitigation is applied for rarfile as well.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| magic_numbers = [b'Rar!\x1a\x07\x00', b'Rar!\x1a\x07\x01\x00']
|
| ----------------------------------------------------------------------
| Class methods inherited from MagicNumberBaseExtractor:
|
| is_extractable(path: Union[pathlib.Path, str], magic_number: bytes = b'') -> bool
|
| ----------------------------------------------------------------------
| Static methods inherited from MagicNumberBaseExtractor:
|
| read_magic_number(path: Union[pathlib.Path, str], magic_number_length: int)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class SevenZipExtractor(MagicNumberBaseExtractor)
| Method resolution order:
| SevenZipExtractor
| MagicNumberBaseExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| safemembers(members: list['py7zr.FileInfo'], output_path: Union[pathlib.Path, str])
| Fix for CVE-2007-4559
| Desc:
| Directory traversal vulnerability in the (1) extract and (2) extractall functions in the tarfile
| module in Python allows user-assisted remote attackers to overwrite arbitrary files via a .. (dot dot)
| sequence in filenames in a TAR archive, a related issue to CVE-2001-1267.
| See: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4559
| From: https://stackoverflow.com/a/10077309
|
| This additional mitigation is applied for py7zr as well.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| magic_numbers = [b"7z\xbc\xaf'\x1c"]
|
| ----------------------------------------------------------------------
| Class methods inherited from MagicNumberBaseExtractor:
|
| is_extractable(path: Union[pathlib.Path, str], magic_number: bytes = b'') -> bool
|
| ----------------------------------------------------------------------
| Static methods inherited from MagicNumberBaseExtractor:
|
| read_magic_number(path: Union[pathlib.Path, str], magic_number_length: int)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class TarExtractor(BaseExtractor)
| Method resolution order:
| TarExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Class methods defined here:
|
| is_extractable(path: Union[pathlib.Path, str], **kwargs) -> bool
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| safemembers(members: tarfile.TarFile, output_path: Union[pathlib.Path, str])
| Fix for CVE-2007-4559
| Desc:
| Directory traversal vulnerability in the (1) extract and (2) extractall functions in the tarfile
| module in Python allows user-assisted remote attackers to overwrite arbitrary files via a .. (dot dot)
| sequence in filenames in a TAR archive, a related issue to CVE-2001-1267.
| See: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4559
| From: https://stackoverflow.com/a/10077309
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class XzExtractor(MagicNumberBaseExtractor)
| Method resolution order:
| XzExtractor
| MagicNumberBaseExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| magic_numbers = [b'\xfd7zXZ\x00']
|
| ----------------------------------------------------------------------
| Class methods inherited from MagicNumberBaseExtractor:
|
| is_extractable(path: Union[pathlib.Path, str], magic_number: bytes = b'') -> bool
|
| ----------------------------------------------------------------------
| Static methods inherited from MagicNumberBaseExtractor:
|
| read_magic_number(path: Union[pathlib.Path, str], magic_number_length: int)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class ZipExtractor(MagicNumberBaseExtractor)
| Method resolution order:
| ZipExtractor
| MagicNumberBaseExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Class methods defined here:
|
| is_extractable(path: Union[pathlib.Path, str], magic_number: bytes = b'') -> bool
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| safemembers(members: list[zipfile.ZipInfo], output_path: Union[pathlib.Path, str])
| Fix for CVE-2007-4559
| Desc:
| Directory traversal vulnerability in the (1) extract and (2) extractall functions in the tarfile
| module in Python allows user-assisted remote attackers to overwrite arbitrary files via a .. (dot dot)
| sequence in filenames in a TAR archive, a related issue to CVE-2001-1267.
| See: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4559
| From: https://stackoverflow.com/a/10077309
|
| This additional mitigation is applied for zipfile as well.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| magic_numbers = [b'PK\x03\x04', b'PK\x05\x06', b'PK\x07\x08']
|
| ----------------------------------------------------------------------
| Static methods inherited from MagicNumberBaseExtractor:
|
| read_magic_number(path: Union[pathlib.Path, str], magic_number_length: int)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class ZstdExtractor(MagicNumberBaseExtractor)
| Method resolution order:
| ZstdExtractor
| MagicNumberBaseExtractor
| BaseExtractor
| abc.ABC
| builtins.object
|
| Static methods defined here:
|
| extract(input_path: Union[pathlib.Path, str], output_path: Union[pathlib.Path, str]) -> None
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {}
|
| magic_numbers = [b'(\xb5/\xfd']
|
| ----------------------------------------------------------------------
| Class methods inherited from MagicNumberBaseExtractor:
|
| is_extractable(path: Union[pathlib.Path, str], magic_number: bytes = b'') -> bool
|
| ----------------------------------------------------------------------
| Static methods inherited from MagicNumberBaseExtractor:
|
| read_magic_number(path: Union[pathlib.Path, str], magic_number_length: int)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseExtractor:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
DATA
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
TYPE_CHECKING = False
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.utils.extract (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\extract.py
datasets.features
Python Library Documentation: package datasets.features in datasets
NAME
datasets.features
PACKAGE CONTENTS
_torchcodec
audio
features
image
mesh
nifti
pdf
translation
video
CLASSES
builtins.dict(builtins.object)
datasets.features.features.Features
builtins.object
datasets.features.audio.Audio
datasets.features.features.ClassLabel
datasets.features.features.Json
datasets.features.features.LargeList
datasets.features.features.Sequence
datasets.features.features.List
datasets.features.features.Value
datasets.features.image.Image
datasets.features.mesh.Mesh
datasets.features.nifti.Nifti
datasets.features.pdf.Pdf
datasets.features.translation.Translation
datasets.features.translation.TranslationVariableLanguages
datasets.features.video.Video
datasets.features.features._ArrayXD(builtins.object)
datasets.features.features.Array2D
datasets.features.features.Array3D
datasets.features.features.Array4D
datasets.features.features.Array5D
class Array2D(_ArrayXD)
| Array2D(shape: tuple, dtype: str, id: Optional[str] = None) -> None
|
| Create a two-dimensional array.
|
| Args:
| shape (`tuple`):
| Size of each dimension.
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'x': Array2D(shape=(1, 3), dtype='int32')})
| ```
|
| Method resolution order:
| Array2D
| _ArrayXD
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, shape: tuple, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('shape', 'dtype', 'id')
|
| id = None
|
| ----------------------------------------------------------------------
| Methods inherited from _ArrayXD:
|
| __call__(self)
| Call self as a function.
|
| __post_init__(self)
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _ArrayXD:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Array3D(_ArrayXD)
| Array3D(shape: tuple, dtype: str, id: Optional[str] = None) -> None
|
| Create a three-dimensional array.
|
| Args:
| shape (`tuple`):
| Size of each dimension.
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'x': Array3D(shape=(1, 2, 3), dtype='int32')})
| ```
|
| Method resolution order:
| Array3D
| _ArrayXD
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, shape: tuple, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('shape', 'dtype', 'id')
|
| id = None
|
| ----------------------------------------------------------------------
| Methods inherited from _ArrayXD:
|
| __call__(self)
| Call self as a function.
|
| __post_init__(self)
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _ArrayXD:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Array4D(_ArrayXD)
| Array4D(shape: tuple, dtype: str, id: Optional[str] = None) -> None
|
| Create a four-dimensional array.
|
| Args:
| shape (`tuple`):
| Size of each dimension.
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'x': Array4D(shape=(1, 2, 2, 3), dtype='int32')})
| ```
|
| Method resolution order:
| Array4D
| _ArrayXD
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, shape: tuple, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('shape', 'dtype', 'id')
|
| id = None
|
| ----------------------------------------------------------------------
| Methods inherited from _ArrayXD:
|
| __call__(self)
| Call self as a function.
|
| __post_init__(self)
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _ArrayXD:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Array5D(_ArrayXD)
| Array5D(shape: tuple, dtype: str, id: Optional[str] = None) -> None
|
| Create a five-dimensional array.
|
| Args:
| shape (`tuple`):
| Size of each dimension.
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'x': Array5D(shape=(1, 2, 2, 3, 3), dtype='int32')})
| ```
|
| Method resolution order:
| Array5D
| _ArrayXD
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, shape: tuple, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('shape', 'dtype', 'id')
|
| id = None
|
| ----------------------------------------------------------------------
| Methods inherited from _ArrayXD:
|
| __call__(self)
| Call self as a function.
|
| __post_init__(self)
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _ArrayXD:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Audio(builtins.object)
| Audio(sampling_rate: Optional[int] = None, decode: bool = True, num_channels: Optional[int] = None, stream_index: Optional[int] = None, id: Optional[str] = None) -> None
|
| Audio [`Feature`] to extract audio data from an audio file.
|
| Input: The Audio feature accepts as input:
| - A `str`: Absolute path to the audio file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the audio file (i.e. random access is allowed).
| - A `dict` with the keys:
|
| - `path`: String with relative path of the audio file to the archive file.
| - `bytes`: Bytes content of the audio file.
|
| This is useful for parquet or webdataset files which embed audio files.
|
| - A `dict` with the keys:
|
| - `array`: Array containing the audio sample
| - `sampling_rate`: Integer corresponding to the sampling rate of the audio sample.
|
| - A `torchcodec.decoders.AudioDecoder`: torchcodec audio decoder object.
|
| Output: The Audio features output data as `torchcodec.decoders.AudioDecoder` objects, with additional keys:
|
| - `array`: Array containing the audio sample
| - `sampling_rate`: Integer corresponding to the sampling rate of the audio sample.
|
| Args:
| sampling_rate (`int`, *optional*):
| Target sampling rate. If `None`, the native sampling rate is used.
| num_channels (`int`, *optional*):
| The desired number of channels of the samples. By default, the number of channels of the source is used.
| Audio decoding will return samples with shape (num_channels, num_samples)
| Currently `None` (number of channels of the source, default), `1` (mono) or `2` (stereo) channels are supported.
| The `num_channels` argument is passed to `torchcodec.decoders.AudioDecoder`.
|
| <Added version="4.4.0"/>
| decode (`bool`, defaults to `True`):
| Whether to decode the audio data. If `False`,
| returns the underlying dictionary in the format `{"path": audio_path, "bytes": audio_bytes}`.
| stream_index (`int`, *optional*):
| The streaming index to use from the file. If `None` defaults to the "best" index.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, Audio
| >>> ds = load_dataset("PolyAI/minds14", name="en-US", split="train")
| >>> ds = ds.cast_column("audio", Audio(sampling_rate=44100, num_channels=2))
| >>> ds[0]["audio"]
| <datasets.features._torchcodec.AudioDecoder object at 0x11642b6a0>
| >>> audio = ds[0]["audio"]
| >>> audio.get_samples_played_in_range(0, 10)
| AudioSamples:
| data (shape): torch.Size([2, 110592])
| pts_seconds: 0.0
| duration_seconds: 2.507755102040816
| sample_rate: 44100
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, sampling_rate: Optional[int] = None, decode: bool = True, num_channels: Optional[int] = None, stream_index: Optional[int] = None, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Audio arrow storage type.
| The Arrow types that can be converted to the Audio pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.binary()` - it must contain the audio bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Audio arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`
|
| decode_example(self, value: dict, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None) -> 'AudioDecoder'
| Decode example audio file into audio data.
|
| Args:
| value (`dict`):
| A dictionary with keys:
|
| - `path`: String with relative audio file path.
| - `bytes`: Bytes of the audio file.
| token_per_repo_id (`dict`, *optional*):
| To access and decode
| audio files from private repositories on the Hub, you can pass
| a dictionary repo_id (`str`) -> token (`bool` or `str`)
|
| Returns:
| `torchcodec.decoders.AudioDecoder`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed audio files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the Audio arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, ForwardRef('AudioDecoder')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `bytes`,`bytearray`,`dict`, `AudioDecoder`):
| Data passed as input to Audio feature.
|
| Returns:
| `dict`
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| If in the decodable state, raise an error, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('sampling_rate', 'decode', 'num_channels', 'stream_i...
|
| decode = True
|
| dtype = 'dict'
|
| id = None
|
| num_channels = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
|
| sampling_rate = None
|
| stream_index = None
class ClassLabel(builtins.object)
| ClassLabel(num_classes: dataclasses.InitVar[typing.Optional[int]] = None, names: list[str] = None, names_file: dataclasses.InitVar[typing.Optional[str]] = None, id: Optional[str] = None) -> None
|
| Feature type for integer class labels.
|
| There are 3 ways to define a `ClassLabel`, which correspond to the 3 arguments:
|
| * `num_classes`: Create 0 to (num_classes-1) labels.
| * `names`: List of label strings.
| * `names_file`: File containing the list of labels.
|
| Under the hood the labels are stored as integers.
| You can use -1 to represent unknown/missing labels.
|
| Args:
| num_classes (`int`, *optional*):
| Number of classes. All labels must be < `num_classes`.
| names (`list` of `str`, *optional*):
| String names for the integer classes.
| The order in which the names are provided is kept.
| names_file (`str`, *optional*):
| Path to a file with names for the integer classes, one per line.
|
| Example:
|
| ```py
| >>> from datasets import Features, ClassLabel
| >>> features = Features({'label': ClassLabel(num_classes=3, names=['bad', 'ok', 'good'])})
| >>> features
| {'label': ClassLabel(names=['bad', 'ok', 'good'])}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, num_classes: dataclasses.InitVar[typing.Optional[int]] = None, names: list[str] = None, names_file: dataclasses.InitVar[typing.Optional[str]] = None, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self, num_classes, names_file)
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.IntegerArray]) -> pyarrow.lib.Int64Array
| Cast an Arrow array to the `ClassLabel` arrow storage type.
| The Arrow types that can be converted to the `ClassLabel` pyarrow storage type are:
|
| - `pa.string()`
| - `pa.int()`
|
| Args:
| storage (`Union[pa.StringArray, pa.IntegerArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.Int64Array`: Array in the `ClassLabel` arrow storage type.
|
| encode_example(self, example_data)
|
| int2str(self, values: Union[int, collections.abc.Iterable]) -> Union[str, collections.abc.Iterable]
| Conversion `integer` => class name `string`.
|
| Regarding unknown/missing labels: passing negative integers raises `ValueError`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> ds.features["label"].int2str(0)
| 'neg'
| ```
|
| str2int(self, values: Union[str, collections.abc.Iterable]) -> Union[int, collections.abc.Iterable]
| Conversion class name `string` => `integer`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> ds.features["label"].str2int('neg')
| 0
| ```
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_int2str': typing.ClassVar[dict[int, int]], '_str2...
|
| __dataclass_fields__ = {'_int2str': Field(name='_int2str',type=typing....
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('num_classes', 'names', 'names_file', 'id')
|
| dtype = 'int64'
|
| id = None
|
| names = None
|
| names_file = None
|
| num_classes = None
|
| pa_type = DataType(int64)
class Features(builtins.dict)
| Features(*args, **kwargs)
|
| A special dictionary that defines the internal structure of a dataset.
|
| Instantiated with a dictionary of type `dict[str, FieldType]`, where keys are the desired column names,
| and values are the type of that column.
|
| `FieldType` can be one of the following:
| - [`Value`] feature specifies a single data type value, e.g. `int64` or `string`.
| - [`ClassLabel`] feature specifies a predefined set of classes which can have labels associated to them and
| will be stored as integers in the dataset.
| - Python `dict` specifies a composite feature containing a mapping of sub-fields to sub-features.
| It's possible to have nested fields of nested fields in an arbitrary manner.
| - [`List`] or [`LargeList`] specifies a composite feature containing a sequence of
| sub-features, all of the same feature type.
| - [`Array2D`], [`Array3D`], [`Array4D`] or [`Array5D`] feature for multidimensional arrays.
| - [`Audio`] feature to store the absolute path to an audio file or a dictionary with the relative path
| to an audio file ("path" key) and its bytes content ("bytes" key).
| This feature loads the audio lazily with a decoder.
| - [`Image`] feature to store the absolute path to an image file, an `np.ndarray` object, a `PIL.Image.Image` object
| or a dictionary with the relative path to an image file ("path" key) and its bytes content ("bytes" key).
| This feature extracts the image data.
| - [`Video`] feature to store the absolute path to a video file, a `torchcodec.decoders.VideoDecoder` object
| or a dictionary with the relative path to a video file ("path" key) and its bytes content ("bytes" key).
| This feature loads the video lazily with a decoder.
| - [`Pdf`] feature to store the absolute path to a PDF file, a `pdfplumber.pdf.PDF` object
| or a dictionary with the relative path to a PDF file ("path" key) and its bytes content ("bytes" key).
| This feature loads the PDF lazily with a PDF reader.
| - [`Nifti`] feature to store the absolute path to a NIfTI neuroimaging file, a `nibabel.Nifti1Image` object
| or a dictionary with the relative path to a NIfTI file ("path" key) and its bytes content ("bytes" key).
| This feature loads the NIfTI file lazily with nibabel.
| - [`Translation`] or [`TranslationVariableLanguages`] feature specific to Machine Translation.
| - [`Json`] feature to store unstructred data, e.g. containing mixed/abritrary types. Under the hood
|
| Method resolution order:
| Features
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __delitem__(self, key, /) from datasets.features.features.dict
| Delete self[key].
|
| __init__(*args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __reduce__(self)
| Helper for pickle.
|
| __setitem__(self, key, value, /) from datasets.features.features.dict
| Set self[key] to value.
|
| clear(...) from datasets.features.features.dict
| D.clear() -> None. Remove all items from D.
|
| copy(self) -> 'Features'
| Make a deep copy of [`Features`].
|
| Returns:
| [`Features`]
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> copy_of_features = ds.features.copy()
| >>> copy_of_features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| ```
|
| decode_batch(self, batch: dict, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
| Decode batch with custom feature decoding.
|
| Args:
| batch (`dict[str, list[Any]]`):
| Dataset batch data.
| token_per_repo_id (`dict`, *optional*):
| To access and decode audio or image files from private repositories on the Hub, you can pass
| a dictionary repo_id (str) -> token (bool or str)
|
| Returns:
| `dict[str, list[Any]]`
|
| decode_column(self, column: list, column_name: str, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
| Decode column with custom feature decoding.
|
| Args:
| column (`list[Any]`):
| Dataset column data.
| column_name (`str`):
| Dataset column name.
|
| Returns:
| `list[Any]`
|
| decode_example(self, example: dict, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
| Decode example with custom feature decoding.
|
| Args:
| example (`dict[str, Any]`):
| Dataset row data.
| token_per_repo_id (`dict`, *optional*):
| To access and decode audio or image files from private repositories on the Hub, you can pass
| a dictionary `repo_id (str) -> token (bool or str)`.
|
| Returns:
| `dict[str, Any]`
|
| encode_batch(self, batch)
| Encode batch into a format for Arrow.
|
| Args:
| batch (`dict[str, list[Any]]`):
| Data in a Dataset batch.
|
| Returns:
| `dict[str, list[Any]]`
|
| encode_column(self, column, column_name: str)
| Encode column into a format for Arrow.
|
| Args:
| column (`list[Any]`):
| Data in a Dataset column.
| column_name (`str`):
| Dataset column name.
|
| Returns:
| `list[Any]`
|
| encode_example(self, example)
| Encode example into a format for Arrow.
|
| Args:
| example (`dict[str, Any]`):
| Data in a Dataset row.
|
| Returns:
| `dict[str, Any]`
|
| flatten(self, max_depth=16) -> 'Features'
| Flatten the features. Every dictionary column is removed and is replaced by
| all the subfields it contains. The new fields are named by concatenating the
| name of the original column and the subfield name like this: `<original>.<subfield>`.
|
| If a column contains nested dictionaries, then all the lower-level subfields names are
| also concatenated to form new columns: `<original>.<subfield>.<subsubfield>`, etc.
|
| Returns:
| [`Features`]:
| The flattened features.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("rajpurkar/squad", split="train")
| >>> ds.features.flatten()
| {'answers.answer_start': List(Value('int32'), id=None),
| 'answers.text': List(Value('string'), id=None),
| 'context': Value('string'),
| 'id': Value('string'),
| 'question': Value('string'),
| 'title': Value('string')}
| ```
|
| pop(...) from datasets.features.features.dict
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /) from datasets.features.features.dict
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| reorder_fields_as(self, other: 'Features') -> 'Features'
| Reorder Features fields to match the field order of other [`Features`].
|
| The order of the fields is important since it matters for the underlying arrow data.
| Re-ordering the fields allows to make the underlying arrow data type match.
|
| Args:
| other ([`Features`]):
| The other [`Features`] to align with.
|
| Returns:
| [`Features`]
|
| Example::
|
| >>> from datasets import Features, List, Value
| >>> # let's say we have two features with a different order of nested fields (for a and b for example)
| >>> f1 = Features({"root": {"a": Value("string"), "b": Value("string")}})
| >>> f2 = Features({"root": {"b": Value("string"), "a": Value("string")}})
| >>> assert f1.type != f2.type
| >>> # re-ordering keeps the base structure (here List is defined at the root level), but makes the fields order match
| >>> f1.reorder_fields_as(f2)
| {'root': List({'b': Value('string'), 'a': Value('string')})}
| >>> assert f1.reorder_fields_as(f2).type == f2.type
|
| setdefault(self, key, default=None, /) from datasets.features.features.dict
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| to_dict(self)
|
| update(...) from datasets.features.features.dict
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_arrow_schema(pa_schema: pyarrow.lib.Schema) -> 'Features'
| Construct [`Features`] from Arrow Schema.
| It also checks the schema metadata for Hugging Face Datasets features.
| Non-nullable fields are not supported and set to nullable.
|
| Also, pa.dictionary is not supported and it uses its underlying type instead.
| Therefore datasets convert DictionaryArray objects to their actual values.
|
| Args:
| pa_schema (`pyarrow.Schema`):
| Arrow Schema.
|
| Returns:
| [`Features`]
|
| from_dict(dic) -> 'Features'
| Construct [`Features`] from dict.
|
| Regenerate the nested feature object from a deserialized dict.
| We use the `_type` key to infer the dataclass name of the feature `FieldType`.
|
| It allows for a convenient constructor syntax
| to define features from deserialized JSON dictionaries. This function is used in particular when deserializing
| a [`DatasetInfo`] that was dumped to a JSON object. This acts as an analogue to
| [`Features.from_arrow_schema`] and handles the recursive field-by-field instantiation, but doesn't require
| any mapping to/from pyarrow, except for the fact that it takes advantage of the mapping of pyarrow primitive
| dtypes that [`Value`] automatically performs.
|
| Args:
| dic (`dict[str, Any]`):
| Python dictionary.
|
| Returns:
| `Features`
|
| Example::
| >>> Features.from_dict({'_type': {'dtype': 'string', 'id': None, '_type': 'Value'}})
| {'_type': Value('string')}
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| arrow_schema
| Features schema.
|
| Returns:
| :obj:`pyarrow.Schema`
|
| type
| Features field types.
|
| Returns:
| :obj:`pyarrow.DataType`
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class Image(builtins.object)
| Image(mode: Optional[str] = None, decode: bool = True, id: Optional[str] = None) -> None
|
| Image [`Feature`] to read image data from an image file.
|
| Input: The Image feature accepts as input:
| - A `str`: Absolute path to the image file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the image file (i.e. random access is allowed).
| - A `dict` with the keys:
|
| - `path`: String with relative path of the image file to the archive file.
| - `bytes`: Bytes of the image file.
|
| This is useful for parquet or webdataset files which embed image files.
|
| - An `np.ndarray`: NumPy array representing an image.
| - A `PIL.Image.Image`: PIL image object.
|
| Output: The Image features output data as `PIL.Image.Image` objects.
|
| Args:
| mode (`str`, *optional*):
| The mode to convert the image to. If `None`, the native mode of the image is used.
| decode (`bool`, defaults to `True`):
| Whether to decode the image data. If `False`,
| returns the underlying dictionary in the format `{"path": image_path, "bytes": image_bytes}`.
|
| Examples:
|
| ```py
| >>> from datasets import load_dataset, Image
| >>> ds = load_dataset("AI-Lab-Makerere/beans", split="train")
| >>> ds.features["image"]
| Image(decode=True, id=None)
| >>> ds[0]["image"]
| <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x500 at 0x15E52E7F0>
| >>> ds = ds.cast_column('image', Image(decode=False))
| {'bytes': None,
| 'path': '/root/.cache/huggingface/datasets/downloads/extracted/b0a21163f78769a2cf11f58dfc767fb458fc7cea5c05dccc0144a2c0f0bc1292/train/healthy/healthy_train.85.jpg'}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, mode: Optional[str] = None, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray, pyarrow.lib.ListArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Image arrow storage type.
| The Arrow types that can be converted to the Image pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.large_string()` - it must contain the "path" data (will be cast to string if possible)
| - `pa.binary()` - it must contain the image bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
| - `pa.list(*)` - it must contain the image array data
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Image arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: dict, token_per_repo_id=None) -> 'PIL.Image.Image'
| Decode example image file into image data.
|
| Args:
| value (`str` or `dict`):
| A string with the absolute image file path, a dictionary with
| keys:
|
| - `path`: String with absolute or relative image file path.
| - `bytes`: The bytes of the image file.
| token_per_repo_id (`dict`, *optional*):
| To access and decode
| image files from private repositories on the Hub, you can pass
| a dictionary repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `PIL.Image.Image`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed image files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the Image arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, numpy.ndarray, ForwardRef('PIL.Image.Image')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `np.ndarray`, `PIL.Image.Image` or `dict`):
| Data passed as input to Image feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('mode', 'decode', 'id')
|
| decode = True
|
| dtype = 'PIL.Image.Image'
|
| id = None
|
| mode = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
class Json(builtins.object)
| Json(decode: bool = True, id: Optional[str] = None) -> None
|
| Feature type for JSON objects.
|
| Under the hood the objects are stored as JSON-encoded strings.
|
| Example:
|
| ```py
| >>> from datasets import Features, Json
| >>> features = Features({'json': Json()})
| >>> features
| {'json': Json()}
| ```
|
| ```py
| >>> from datasets import Dataset, Features, Json, List
| >>> features = Features({"a": List(Json())})
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, features=features)
| >>> # OR
| >>> ds = Dataset.from_dict({"a": [[{"b": 0}, {"c": 0}]]}, on_mixed_types="use_json")
| >>> ds.features
| {'a': List(Json())}
| >>> ds[0]
| {'a': [{'b': 0}, {'c': 0}]}
| >>> def f(x):
| ... for y in x["a"]:
| ... y["d"] = "foo"
| ... return x
| >>> ds = ds.map(f)
| >>> ds.features
| >>> ds[0]
| {'a': [{'b': 0, 'd': 'foo'}, {'c': 0, 'd': 'foo'}]}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: pyarrow.lib.Array) -> pyarrow.lib.JsonArray
| Cast an Arrow array to the `Json` arrow storage type.
|
| Args:
| storage (`Union[pa.StringArray, pa.IntegerArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.JsonArray`: Array in the `Json` arrow storage type.
|
| decode_example(self, example_data, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
|
| encode_example(self, example_data)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'id')
|
| decode = True
|
| id = None
|
| pa_type = JsonType(extension<arrow.json>)
class LargeList(builtins.object)
| LargeList(feature: Any, id: Optional[str] = None) -> None
|
| Feature type for large list data composed of child feature data type.
|
| It is backed by `pyarrow.LargeListType`, which is like `pyarrow.ListType` but with 64-bit rather than 32-bit offsets.
|
| Args:
| feature ([`FeatureType`]):
| Child feature data type of each item within the large list.
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, feature: Any, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'feature': typing.Any, 'id'...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=False,eq=True,o...
|
| __hash__ = None
|
| __match_args__ = ('feature', 'id')
|
| id = None
|
| pa_type = None
class List(Sequence)
| List(feature: Any, length: int = -1, id: Optional[str] = None) -> None
|
| Feature type for large list data composed of child feature data type.
|
| It is backed by `pyarrow.ListType`, which uses 32-bit offsets or a fixed length.
|
| Args:
| feature ([`FeatureType`]):
| Child feature data type of each item within the large list.
| length (optional `int`, default to -1):
| Length of the list if it is fixed.
| Defaults to -1 which means an arbitrary length.
|
| Method resolution order:
| List
| Sequence
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, feature: Any, length: int = -1, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'feature': typing.Any, 'id'...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=False,eq=True,o...
|
| __hash__ = None
|
| __match_args__ = ('feature', 'length', 'id')
|
| id = None
|
| length = -1
|
| pa_type = None
|
| ----------------------------------------------------------------------
| Static methods inherited from Sequence:
|
| __new__(cls, feature=None, length=-1, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from Sequence:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Mesh(builtins.object)
| Mesh(decode: bool = True, id: Optional[str] = None) -> None
|
| Mesh [`Feature`] to read 3D mesh data from a file.
|
| Input: The Mesh feature accepts as input:
| - A `str`: Absolute path to the mesh file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the mesh file (i.e. random access is allowed).
| - A `dict` with the keys:
|
| - `path`: String with relative path of the mesh file to the archive file.
| - `bytes`: Bytes of the mesh file.
|
| This is useful for parquet or webdataset files which embed mesh files.
|
| - A `trimesh.Trimesh` or `trimesh.Scene`: 3D mesh or scene object.
|
| Output: The Mesh feature outputs data as `trimesh.Trimesh` or `trimesh.Scene` objects.
|
| Args:
| decode (`bool`, defaults to `True`):
| Whether to decode the mesh data. If `False`,
| returns the underlying dictionary in the format `{"path": mesh_path, "bytes": mesh_bytes}`.
| Mesh decoding uses `trimesh` and supports `.glb`, `.ply`, and `.stl` files.
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Mesh arrow storage type.
| The Arrow types that can be converted to the Mesh pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.large_string()` - it must contain the "path" data (will be cast to string if possible)
| - `pa.binary()` - it must contain the mesh bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Mesh arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: dict, token_per_repo_id=None) -> Union[ForwardRef('trimesh.Trimesh'), ForwardRef('trimesh.Scene')]
| Decode example mesh file.
|
| Args:
| value (`dict`):
| A dictionary with keys:
|
| - `path`: String with absolute or relative mesh file path.
| - `bytes`: The bytes of the mesh file.
| token_per_repo_id (`dict`, *optional*):
| To access and decode
| mesh files from private repositories on the Hub, you can pass
| a dictionary repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `trimesh.Trimesh` or `trimesh.Scene`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed mesh files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`):
| Whether to embed local files data in the array.
| remote_files (`bool`, defaults to `True`):
| Whether to embed remote files data in the array.
|
| Returns:
| `pa.StructArray`: Array in the Mesh arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, ForwardRef('trimesh.Trimesh'), ForwardRef('trimesh.Scene')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `bytes`, `dict`, `trimesh.Trimesh`, or `trimesh.Scene`):
| Data passed as input to Mesh feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'id')
|
| decode = True
|
| dtype = 'trimesh.Trimesh'
|
| id = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
class Nifti(builtins.object)
| Nifti(decode: bool = True, id: Optional[str] = None) -> None
|
| **Experimental.**
| Nifti [`Feature`] to read NIfTI neuroimaging files.
|
| Input: The Nifti feature accepts as input:
| - A `str`: Absolute path to the NIfTI file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the NIfTI file (i.e. random access is allowed).
| - A `dict` with the keys:
| - `path`: String with relative path of the NIfTI file in a dataset repository.
| - `bytes`: Bytes of the NIfTI file.
| This is useful for archived files with sequential access.
|
| - A `nibabel` image object (e.g., `nibabel.nifti1.Nifti1Image`).
|
| Args:
| decode (`bool`, defaults to `True`):
| Whether to decode the NIfTI data. If `False` a string with the bytes is returned. `decode=False` is not supported when decoding examples.
|
| Examples:
|
| ```py
| >>> from datasets import Dataset, Nifti
| >>> ds = Dataset.from_dict({"nifti": ["path/to/file.nii.gz"]}).cast_column("nifti", Nifti())
| >>> ds.features["nifti"]
| Nifti(decode=True, id=None)
| >>> ds[0]["nifti"]
| <nibabel.nifti1.Nifti1Image object at 0x7f8a1c2d8f40>
| >>> ds = ds.cast_column("nifti", Nifti(decode=False))
| >>> ds[0]["nifti"]
| {'bytes': None,
| 'path': 'path/to/file.nii.gz'}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray, pyarrow.lib.BinaryArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Nifti arrow storage type.
| The Arrow types that can be converted to the Nifti pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.binary()` - it must contain the NIfTI bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray, pa.BinaryArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Nifti arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: dict, token_per_repo_id=None) -> 'Nifti1ImageWrapper'
| Decode example NIfTI file into nibabel image object.
|
| Args:
| value (`str` or `dict`):
| A string with the absolute NIfTI file path, a dictionary with
| keys:
|
| - `path`: String with absolute or relative NIfTI file path.
| - `bytes`: The bytes of the NIfTI file.
|
| token_per_repo_id (`dict`, *optional*):
| To access and decode NIfTI files from private repositories on
| the Hub, you can pass a dictionary
| repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `nibabel.Nifti1Image` objects
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed NifTI files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the NifTI arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, ForwardRef('nib.Nifti1Image')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `bytes`, `nibabel.Nifti1Image` or `dict`):
| Data passed as input to Nifti feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), Dict[str, ForwardRef('FeatureType')]]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'id')
|
| decode = True
|
| dtype = 'nibabel.nifti1.Nifti1Image'
|
| id = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
class Pdf(builtins.object)
| Pdf(decode: bool = True, id: Optional[str] = None) -> None
|
| **Experimental.**
| Pdf [`Feature`] to read pdf documents from a pdf file.
|
| Input: The Pdf feature accepts as input:
| - A `str`: Absolute path to the pdf file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the pdf file (i.e. random access is allowed).
| - A `dict` with the keys:
| - `path`: String with relative path of the pdf file in a dataset repository.
| - `bytes`: Bytes of the pdf file.
| This is useful for archived files with sequential access.
|
| - A `pdfplumber.pdf.PDF`: pdfplumber pdf object.
|
| Args:
| decode (`bool`, defaults to `True`):
| Whether to decode the pdf data. If `False`,
| returns the underlying dictionary in the format `{"path": pdf_path, "bytes": pdf_bytes}`.
|
| Examples:
|
| ```py
| >>> from datasets import Dataset, Pdf
| >>> ds = Dataset.from_dict({"pdf": ["path/to/pdf/file.pdf"]}).cast_column("pdf", Pdf())
| >>> ds.features["pdf"]
| Pdf(decode=True, id=None)
| >>> ds[0]["pdf"]
| <pdfplumber.pdf.PDF object at 0x7f8a1c2d8f40>
| >>> ds = ds.cast_column("pdf", Pdf(decode=False))
| >>> ds[0]["pdf"]
| {'bytes': None,
| 'path': 'path/to/pdf/file.pdf'}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray, pyarrow.lib.ListArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Pdf arrow storage type.
| The Arrow types that can be converted to the Pdf pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.binary()` - it must contain the image bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
| - `pa.list(*)` - it must contain the pdf array data
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Pdf arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: dict, token_per_repo_id=None) -> 'pdfplumber.pdf.PDF'
| Decode example pdf file into pdf data.
|
| Args:
| value (`str` or `dict`):
| A string with the absolute pdf file path, a dictionary with
| keys:
|
| - `path`: String with absolute or relative pdf file path.
| - `bytes`: The bytes of the pdf file.
|
| token_per_repo_id (`dict`, *optional*):
| To access and decode pdf files from private repositories on
| the Hub, you can pass a dictionary
| repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `pdfplumber.pdf.PDF`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed PDF files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the PDF arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, dict, ForwardRef('pdfplumber.pdf.PDF')]) -> dict
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `bytes`, `pdfplumber.pdf.PDF` or `dict`):
| Data passed as input to Pdf feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), Dict[str, ForwardRef('FeatureType')]]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'id')
|
| decode = True
|
| dtype = 'pdfplumber.pdf.PDF'
|
| id = None
|
| pa_type = StructType(struct<bytes: binary, path: string>)
class Sequence(builtins.object)
| Sequence(feature=None, length=-1, **kwargs)
|
| A `Sequence` is a utility that automatically converts internal dictionary feature into a dictionary of
| lists. This behavior is implemented to have a compatibility layer with the TensorFlow Datasets library but may be
| un-wanted in some cases. If you don't want this behavior, you can use a [`List`] or a [`LargeList`]
| instead of the [`Sequence`].
|
| Args:
| feature ([`FeatureType`]):
| Child feature data type of each item within the large list.
| length (optional `int`, default to -1):
| Length of the list if it is fixed.
| Defaults to -1 which means an arbitrary length.
|
| Returns:
| [`List`] of the specified feature, except `dict` of sub-features
| which are converted to `dict` of lists of sub-features for compatibility with TFDS.
|
| Static methods defined here:
|
| __new__(cls, feature=None, length=-1, **kwargs)
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
class Translation(builtins.object)
| Translation(languages: list[str], id: Optional[str] = None) -> None
|
| `Feature` for translations with fixed languages per example.
| Here for compatibility with tfds.
|
| Args:
| languages (`dict`):
| A dictionary for each example mapping string language codes to string translations.
|
| Example:
|
| ```python
| >>> # At construction time:
| >>> datasets.features.Translation(languages=['en', 'fr', 'de'])
| >>> # During data generation:
| >>> yield {
| ... 'en': 'the cat',
| ... 'fr': 'le chat',
| ... 'de': 'die katze'
| ... }
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, languages: list[str], id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| Flatten the Translation feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': typing.ClassVar[st...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('languages', 'id')
|
| dtype = 'dict'
|
| id = None
|
| pa_type = None
class TranslationVariableLanguages(builtins.object)
| TranslationVariableLanguages(languages: Optional[list] = None, num_languages: Optional[int] = None, id: Optional[str] = None) -> None
|
| `Feature` for translations with variable languages per example.
| Here for compatibility with tfds.
|
| Args:
| languages (`dict`):
| A dictionary for each example mapping string language codes to one or more string translations.
| The languages present may vary from example to example.
|
| Returns:
| - `language` or `translation` (variable-length 1D `tf.Tensor` of `tf.string`):
| Language codes sorted in ascending order or plain text translations, sorted to align with language codes.
|
| Example:
|
| ```python
| >>> # At construction time:
| >>> datasets.features.TranslationVariableLanguages(languages=['en', 'fr', 'de'])
| >>> # During data generation:
| >>> yield {
| ... 'en': 'the cat',
| ... 'fr': ['le chat', 'la chatte,']
| ... 'de': 'die katze'
| ... }
| >>> # Tensor returned :
| >>> {
| ... 'language': ['en', 'de', 'fr', 'fr'],
| ... 'translation': ['the cat', 'die katze', 'la chatte', 'le chat'],
| ... }
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, languages: Optional[list] = None, num_languages: Optional[int] = None, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| encode_example(self, translation_dict)
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| Flatten the TranslationVariableLanguages feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': typing.ClassVar[st...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('languages', 'num_languages', 'id')
|
| dtype = 'dict'
|
| id = None
|
| languages = None
|
| num_languages = None
|
| pa_type = None
class Value(builtins.object)
| Value(dtype: str, id: Optional[str] = None) -> None
|
| Scalar feature value of a particular data type.
|
| The possible dtypes of `Value` are as follows:
| - `null`
| - `bool`
| - `int8`
| - `int16`
| - `int32`
| - `int64`
| - `uint8`
| - `uint16`
| - `uint32`
| - `uint64`
| - `float16`
| - `float32` (alias float)
| - `float64` (alias double)
| - `time32[(s|ms)]`
| - `time64[(us|ns)]`
| - `timestamp[(s|ms|us|ns)]`
| - `timestamp[(s|ms|us|ns), tz=(tzstring)]`
| - `date32`
| - `date64`
| - `duration[(s|ms|us|ns)]`
| - `decimal128(precision, scale)`
| - `decimal256(precision, scale)`
| - `binary`
| - `large_binary`
| - `binary_view`
| - `string`
| - `large_string`
| - `string_view`
|
| Args:
| dtype (`str`):
| Name of the data type.
|
| Example:
|
| ```py
| >>> from datasets import Features
| >>> features = Features({'stars': Value('int32')})
| >>> features
| {'stars': Value('int32')}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, dtype: str, id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| encode_example(self, value)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'dtype': <class 'str'>, 'id...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=False,eq=True,o...
|
| __hash__ = None
|
| __match_args__ = ('dtype', 'id')
|
| id = None
|
| pa_type = None
class Video(builtins.object)
| Video(decode: bool = True, stream_index: Optional[int] = None, dimension_order: Literal['NCHW', 'NHWC'] = 'NCHW', num_ffmpeg_threads: int = 1, device: Union[str, ForwardRef('torch.device'), NoneType] = 'cpu', seek_mode: Literal['exact', 'approximate'] = 'exact', id: Optional[str] = None) -> None
|
| Video [`Feature`] to read video data from a video file.
|
| Input: The Video feature accepts as input:
| - A `str`: Absolute path to the video file (i.e. random access is allowed).
| - A `pathlib.Path`: path to the video file (i.e. random access is allowed).
| - A `dict` with the keys:
|
| - `path`: String with relative path of the video file in a dataset repository.
| - `bytes`: Bytes of the video file.
|
| This is useful for parquet or webdataset files which embed video files.
|
| - A `torchcodec.decoders.VideoDecoder`: torchcodec video decoder object.
|
| Output: The Video features output data as `torchcodec.decoders.VideoDecoder` objects.
|
| Args:
| decode (`bool`, defaults to `True`):
| Whether to decode the video data. If `False`,
| returns the underlying dictionary in the format `{"path": video_path, "bytes": video_bytes}`.
| stream_index (`int`, *optional*):
| The streaming index to use from the file. If `None` defaults to the "best" index.
| dimension_order (`str`, defaults to `NCHW`):
| The dimension order of the decoded frames.
| where N is the batch size, C is the number of channels,
| H is the height, and W is the width of the frames.
| num_ffmpeg_threads (`int`, defaults to `1`):
| The number of threads to use for decoding the video. (Recommended to keep this at 1)
| device (`str` or `torch.device`, defaults to `cpu`):
| The device to use for decoding the video.
| seek_mode (`str`, defaults to `exact`):
| Determines if frame access will be “exact” or “approximate”.
| Exact guarantees that requesting frame i will always return frame i, but doing so requires an initial scan of the file.
| Approximate is faster as it avoids scanning the file, but less accurate as it uses the file's metadata to calculate where i probably is.
| read more [here](https://docs.pytorch.org/torchcodec/stable/generated_examples/approximate_mode.html#sphx-glr-generated-examples-approximate-mode-py)
|
| Examples:
|
| ```py
| >>> from datasets import Dataset, Video
| >>> ds = Dataset.from_dict({"video":["path/to/Screen Recording.mov"]}).cast_column("video", Video())
| >>> ds.features["video"]
| Video(decode=True, id=None)
| >>> ds[0]["video"]
| <torchcodec.decoders._video_decoder.VideoDecoder object at 0x14a61e080>
| >>> video = ds[0]["video"]
| >>> video.get_frames_in_range(0, 10)
| FrameBatch:
| data (shape): torch.Size([10, 3, 50, 66])
| pts_seconds: tensor([0.4333, 0.4333, 0.4333, 0.4333, 0.4333, 0.4333, 0.4333, 0.4333, 0.4333,
| 0.4333], dtype=torch.float64)
| duration_seconds: tensor([0.0167, 0.0167, 0.0167, 0.0167, 0.0167, 0.0167, 0.0167, 0.0167, 0.0167,
| 0.0167], dtype=torch.float64)
| >>> ds.cast_column('video', Video(decode=False))[0]["video]
| {'bytes': None,
| 'path': 'path/to/Screen Recording.mov'}
| ```
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, decode: bool = True, stream_index: Optional[int] = None, dimension_order: Literal['NCHW', 'NHWC'] = 'NCHW', num_ffmpeg_threads: int = 1, device: Union[str, ForwardRef('torch.device'), NoneType] = 'cpu', seek_mode: Literal['exact', 'approximate'] = 'exact', id: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| cast_storage(self, storage: Union[pyarrow.lib.StringArray, pyarrow.lib.StructArray, pyarrow.lib.ListArray]) -> pyarrow.lib.StructArray
| Cast an Arrow array to the Video arrow storage type.
| The Arrow types that can be converted to the Video pyarrow storage type are:
|
| - `pa.string()` - it must contain the "path" data
| - `pa.binary()` - it must contain the video bytes
| - `pa.struct({"bytes": pa.binary()})`
| - `pa.struct({"path": pa.string()})`
| - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter
| - `pa.list(*)` - it must contain the video array data
|
| Args:
| storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`):
| PyArrow array to cast.
|
| Returns:
| `pa.StructArray`: Array in the Video arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| decode_example(self, value: Union[str, datasets.features.video.Example], token_per_repo_id: Optional[dict[str, Union[bool, str]]] = None) -> 'VideoDecoder'
| Decode example video file into video data.
|
| Args:
| value (`str` or `dict`):
| A string with the absolute video file path, a dictionary with
| keys:
|
| - `path`: String with absolute or relative video file path.
| - `bytes`: The bytes of the video file.
| token_per_repo_id (`dict`, *optional*):
| To access and decode
| video files from private repositories on the Hub, you can pass
| a dictionary repo_id (`str`) -> token (`bool` or `str`).
|
| Returns:
| `torchcodec.decoders.VideoDecoder`
|
| embed_storage(self, storage: pyarrow.lib.StructArray, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True) -> pyarrow.lib.StructArray
| Embed image files into the Arrow array.
|
| Args:
| storage (`pa.StructArray`):
| PyArrow array to embed.
| token_per_repo_id (`dict`, optional):
| Dictionary repo_id -> token to fetch the files bytes.
| local_files (`bool`, defaults to `True`)
| Whether to embed local files data in the array
|
| <Added version="4.8.5"/>
| remote_files (`bool`, defaults to `True`)
| Whether to embed remote files data in the array.
| E.g. files with paths that start with hf:// or https://
|
| <Added version="4.8.5"/>
|
| Returns:
| `pa.StructArray`: Array in the Video arrow storage type, that is
| `pa.struct({"bytes": pa.binary(), "path": pa.string()})`.
|
| encode_example(self, value: Union[str, bytes, bytearray, datasets.features.video.Example, numpy.ndarray, ForwardRef('VideoDecoder')]) -> datasets.features.video.Example
| Encode example into a format for Arrow.
|
| Args:
| value (`str`, `np.ndarray`, `bytes`, `bytearray`, `VideoDecoder` or `dict`):
| Data passed as input to Video feature.
|
| Returns:
| `dict` with "path" and "bytes" fields
|
| flatten(self) -> Union[ForwardRef('FeatureType'), dict[str, 'FeatureType']]
| If in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_type': <class 'str'>, 'decode': <class 'bool'>, '...
|
| __dataclass_fields__ = {'_type': Field(name='_type',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('decode', 'stream_index', 'dimension_order', 'num_ff...
|
| decode = True
|
| device = 'cpu'
|
| dimension_order = 'NCHW'
|
| dtype = 'torchcodec.decoders.VideoDecoder'
|
| id = None
|
| num_ffmpeg_threads = 1
|
| pa_type = StructType(struct<bytes: binary, path: string>)
|
| seek_mode = 'exact'
|
| stream_index = None
DATA
__all__ = ['Audio', 'Array2D', 'Array3D', 'Array4D', 'Array5D', 'Class...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\features\__init__.py
datasets.file_utils
Python Library Documentation: module datasets.utils.file_utils in datasets.utils
NAME
datasets.utils.file_utils
DESCRIPTION
Utilities for working with the local dataset cache.
This file is adapted from the AllenNLP library at https://github.com/allenai/allennlp
Copyright by the AllenNLP authors.
CLASSES
builtins.Exception(builtins.BaseException)
NonStreamableDatasetError
datasets.utils.track.TrackedIterableFromGenerator(collections.abc.Iterable)
ArchiveIterable
FilesIterable
fsspec.callbacks.TqdmCallback(fsspec.callbacks.Callback)
TqdmCallback
pathlib.WindowsPath(pathlib.Path, pathlib.PureWindowsPath)
xPath
class ArchiveIterable(datasets.utils.track.TrackedIterableFromGenerator)
| ArchiveIterable(generator, *args)
|
| An iterable of (path, fileobj) from a TAR archive, used by `iter_archive`
|
| Method resolution order:
| ArchiveIterable
| datasets.utils.track.TrackedIterableFromGenerator
| collections.abc.Iterable
| builtins.object
|
| Class methods defined here:
|
| from_buf(fileobj) -> 'ArchiveIterable'
|
| from_urlpath(urlpath_or_buf, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'ArchiveIterable'
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Methods inherited from datasets.utils.track.TrackedIterableFromGenerator:
|
| __init__(self, generator, *args)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self) -> str
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from datasets.utils.track.TrackedIterableFromGenerator:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
class FilesIterable(datasets.utils.track.TrackedIterableFromGenerator)
| FilesIterable(generator, *args)
|
| An iterable of paths from a list of directories or files
|
| Method resolution order:
| FilesIterable
| datasets.utils.track.TrackedIterableFromGenerator
| collections.abc.Iterable
| builtins.object
|
| Class methods defined here:
|
| from_urlpaths(urlpaths, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> 'FilesIterable'
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Methods inherited from datasets.utils.track.TrackedIterableFromGenerator:
|
| __init__(self, generator, *args)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self) -> str
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors inherited from datasets.utils.track.TrackedIterableFromGenerator:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
class NonStreamableDatasetError(builtins.Exception)
| Method resolution order:
| NonStreamableDatasetError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class TqdmCallback(fsspec.callbacks.TqdmCallback)
| TqdmCallback(tqdm_kwargs=None, *args, **kwargs)
|
| Method resolution order:
| TqdmCallback
| fsspec.callbacks.TqdmCallback
| fsspec.callbacks.Callback
| builtins.object
|
| Methods defined here:
|
| __init__(self, tqdm_kwargs=None, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from fsspec.callbacks.TqdmCallback:
|
| __del__(self)
|
| call(self, *args, **kwargs)
| Execute hook(s) with current state
|
| Each function is passed the internal size and current value
|
| Parameters
| ----------
| hook_name: str or None
| If given, execute on this hook
| kwargs: passed on to (all) hook(s)
|
| close(self)
| Close callback.
|
| ----------------------------------------------------------------------
| Methods inherited from fsspec.callbacks.Callback:
|
| __enter__(self)
|
| __exit__(self, *exc_args)
|
| __getattr__(self, item)
| If undefined methods are called on this class, nothing happens
|
| absolute_update(self, value)
| Set the internal value state
|
| Triggers ``call()``
|
| Parameters
| ----------
| value: int
|
| branch(self, path_1, path_2, kwargs)
| Set callbacks for child transfers
|
| If this callback is operating at a higher level, e.g., put, which may
| trigger transfers that can also be monitored. The passed kwargs are
| to be *mutated* to add ``callback=``, if this class supports branching
| to children.
|
| Parameters
| ----------
| path_1: str
| Child's source path
| path_2: str
| Child's destination path
| kwargs: dict
| arguments passed to child method, e.g., put_file.
|
| Returns
| -------
|
| branch_coro(self, fn)
| Wraps a coroutine, and pass a new child callback to it.
|
| branched(self, path_1, path_2, **kwargs)
| Return callback for child transfers
|
| If this callback is operating at a higher level, e.g., put, which may
| trigger transfers that can also be monitored. The function returns a callback
| that has to be passed to the child method, e.g., put_file,
| as `callback=` argument.
|
| The implementation uses `callback.branch` for compatibility.
| When implementing callbacks, it is recommended to override this function instead
| of `branch` and avoid calling `super().branched(...)`.
|
| Prefer using this function over `branch`.
|
| Parameters
| ----------
| path_1: str
| Child's source path
| path_2: str
| Child's destination path
| **kwargs:
| Arbitrary keyword arguments
|
| Returns
| -------
| callback: Callback
| A callback instance to be passed to the child method
|
| no_op(self, *_, **__)
|
| relative_update(self, inc=1)
| Delta increment the internal counter
|
| Triggers ``call()``
|
| Parameters
| ----------
| inc: int
|
| set_size(self, size)
| Set the internal maximum size attribute
|
| Usually called if not initially set at instantiation. Note that this
| triggers a ``call()``.
|
| Parameters
| ----------
| size: int
|
| wrap(self, iterable)
| Wrap an iterable to call ``relative_update`` on each iterations
|
| Parameters
| ----------
| iterable: Iterable
| The iterable that is being wrapped
|
| ----------------------------------------------------------------------
| Class methods inherited from fsspec.callbacks.Callback:
|
| as_callback(maybe_callback=None)
| Transform callback=... into Callback instance
|
| For the special value of ``None``, return the global instance of
| ``NoOpCallback``. This is an alternative to including
| ``callback=DEFAULT_CALLBACK`` directly in a method signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from fsspec.callbacks.Callback:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class xPath(pathlib.WindowsPath)
| xPath(*args, **kwargs)
|
| Extension of `pathlib.Path` to support both local paths and remote URLs.
|
| Method resolution order:
| xPath
| pathlib.WindowsPath
| pathlib.Path
| pathlib.PureWindowsPath
| pathlib.PurePath
| builtins.object
|
| Methods defined here:
|
| __str__(self)
| Return the string representation of the path, suitable for
| passing to system calls.
|
| __truediv__(self, p: str) -> 'xPath'
|
| exists(self, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
| Extend `pathlib.Path.exists` method to support both local and remote files.
|
| Args:
| download_config : mainly use token or storage_options to support different platforms and auth types.
|
| Returns:
| `bool`
|
| glob(self, pattern, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
| Glob function for argument of type :obj:`~pathlib.Path` that supports both local paths end remote URLs.
|
| Args:
| pattern (`str`): Pattern that resulting paths must match.
| download_config : mainly use token or storage_options to support different platforms and auth types.
|
| Yields:
| [`xPath`]
|
| is_dir(self, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
| Extend `pathlib.Path.is_dir` method to support both local and remote files.
|
| Args:
| download_config : mainly use token or storage_options to support different platforms and auth types.
|
| Returns:
| `bool`
|
| is_file(self, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
| Extend `pathlib.Path.is_file` method to support both local and remote files.
|
| Args:
| download_config : mainly use token or storage_options to support different platforms and auth types.
|
| Returns:
| `bool`
|
| joinpath(self, *p: tuple[str, ...]) -> 'xPath'
| Extend :func:`xjoin` to support argument of type :obj:`~pathlib.Path`.
|
| Args:
| *p (`tuple` of `str`): Other path components.
|
| Returns:
| [`xPath`]
|
| open(self, *args, **kwargs)
| Extend :func:`xopen` to support argument of type :obj:`~pathlib.Path`.
|
| Args:
| **args: Arguments passed to :func:`fsspec.open`.
| **kwargs: Keyword arguments passed to :func:`fsspec.open`.
|
| Returns:
| `io.FileIO`: File-like object.
|
| rglob(self, pattern, **kwargs)
| Rglob function for argument of type :obj:`~pathlib.Path` that supports both local paths end remote URLs.
|
| Args:
| pattern (`str`): Pattern that resulting paths must match.
|
| Yields:
| [`xPath`]
|
| with_suffix(self, suffix)
| Return a new path with the file suffix changed. If the path
| has no suffix, add given suffix. If the given suffix is an empty
| string, remove the suffix from the path.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| name
| Name function for argument of type :obj:`~pathlib.Path` that supports both local paths end remote URLs.
|
| Returns:
| `str`
|
| parent
| Name function for argument of type :obj:`~pathlib.Path` that supports both local paths end remote URLs.
|
| Returns:
| [`xPath`]
|
| stem
| Stem function for argument of type :obj:`~pathlib.Path` that supports both local paths end remote URLs.
|
| Returns:
| `str`
|
| suffix
| Suffix function for argument of type :obj:`~pathlib.Path` that supports both local paths end remote URLs.
|
| Returns:
| `str`
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from pathlib.Path:
|
| __enter__(self)
|
| __exit__(self, t, v, tb)
|
| __init__(self, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| absolute(self)
| Return an absolute version of this path by prepending the current
| working directory. No normalization or symlink resolution is performed.
|
| Use resolve() to get the canonical path to a file.
|
| chmod(self, mode, *, follow_symlinks=True)
| Change the permissions of the path, like os.chmod().
|
| expanduser(self)
| Return a new path with expanded ~ and ~user constructs
| (as returned by os.path.expanduser)
|
| group(self)
| Return the group name of the file gid.
|
| hardlink_to(self, target)
| Make this path a hard link pointing to the same file as *target*.
|
| Note the order of arguments (self, target) is the reverse of os.link's.
|
| is_block_device(self)
| Whether this path is a block device.
|
| is_char_device(self)
| Whether this path is a character device.
|
| is_fifo(self)
| Whether this path is a FIFO.
|
| is_junction(self)
| Whether this path is a junction.
|
| is_mount(self)
| Check if this path is a mount point
|
| is_socket(self)
| Whether this path is a socket.
|
| is_symlink(self)
| Whether this path is a symbolic link.
|
| iterdir(self)
| Yield path objects of the directory contents.
|
| The children are yielded in arbitrary order, and the
| special entries '.' and '..' are not included.
|
| lchmod(self, mode)
| Like chmod(), except if the path points to a symlink, the symlink's
| permissions are changed, rather than its target's.
|
| lstat(self)
| Like stat(), except if the path points to a symlink, the symlink's
| status information is returned, rather than its target's.
|
| mkdir(self, mode=511, parents=False, exist_ok=False)
| Create a new directory at this given path.
|
| owner(self)
| Return the login name of the file owner.
|
| read_bytes(self)
| Open the file in bytes mode, read it, and close the file.
|
| read_text(self, encoding=None, errors=None)
| Open the file in text mode, read it, and close the file.
|
| readlink(self)
| Return the path to which the symbolic link points.
|
| rename(self, target)
| Rename this path to the target path.
|
| The target path may be absolute or relative. Relative paths are
| interpreted relative to the current working directory, *not* the
| directory of the Path object.
|
| Returns the new Path instance pointing to the target path.
|
| replace(self, target)
| Rename this path to the target path, overwriting if that path exists.
|
| The target path may be absolute or relative. Relative paths are
| interpreted relative to the current working directory, *not* the
| directory of the Path object.
|
| Returns the new Path instance pointing to the target path.
|
| resolve(self, strict=False)
| Make the path absolute, resolving all symlinks on the way and also
| normalizing it.
|
| rmdir(self)
| Remove this directory. The directory must be empty.
|
| samefile(self, other_path)
| Return whether other_path is the same or not as this file
| (as returned by os.path.samefile()).
|
| stat(self, *, follow_symlinks=True)
| Return the result of the stat() system call on this path, like
| os.stat() does.
|
| symlink_to(self, target, target_is_directory=False)
| Make this path a symlink pointing to the target path.
| Note the order of arguments (link, target) is the reverse of os.symlink.
|
| touch(self, mode=438, exist_ok=True)
| Create this file with the given access mode, if it doesn't exist.
|
| unlink(self, missing_ok=False)
| Remove this file or link.
| If the path is a directory, use rmdir() instead.
|
| walk(self, top_down=True, on_error=None, follow_symlinks=False)
| Walk the directory tree from this directory, similar to os.walk().
|
| write_bytes(self, data)
| Open the file in bytes mode, write to it, and close the file.
|
| write_text(self, data, encoding=None, errors=None, newline=None)
| Open the file in text mode, write to it, and close the file.
|
| ----------------------------------------------------------------------
| Class methods inherited from pathlib.Path:
|
| cwd()
| Return a new path pointing to the current working directory.
|
| home()
| Return a new path pointing to the user's home directory (as
| returned by os.path.expanduser('~')).
|
| ----------------------------------------------------------------------
| Static methods inherited from pathlib.Path:
|
| __new__(cls, *args, **kwargs)
| Construct a PurePath from one or several strings and or existing
| PurePath objects. The strings and path objects are combined so as
| to yield a canonicalized path, which is incorporated into the
| new PurePath object.
|
| ----------------------------------------------------------------------
| Methods inherited from pathlib.PurePath:
|
| __bytes__(self)
| Return the bytes representation of the path. This is only
| recommended to use under Unix.
|
| __eq__(self, other)
| Return self==value.
|
| __fspath__(self)
|
| __ge__(self, other)
| Return self>=value.
|
| __gt__(self, other)
| Return self>value.
|
| __hash__(self)
| Return hash(self).
|
| __le__(self, other)
| Return self<=value.
|
| __lt__(self, other)
| Return self<value.
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self)
| Return repr(self).
|
| __rtruediv__(self, key)
|
| as_posix(self)
| Return the string representation of the path with forward (/)
| slashes.
|
| as_uri(self)
| Return the path as a 'file' URI.
|
| is_absolute(self)
| True if the path is absolute (has both a root and, if applicable,
| a drive).
|
| is_relative_to(self, other, /, *_deprecated)
| Return True if the path is relative to another path or False.
|
| is_reserved(self)
| Return True if the path contains one of the special names reserved
| by the system, if any.
|
| match(self, path_pattern, *, case_sensitive=None)
| Return True if this path matches the given pattern.
|
| relative_to(self, other, /, *_deprecated, walk_up=False)
| Return the relative path to another path identified by the passed
| arguments. If the operation is not possible (because this is not
| related to the other path), raise ValueError.
|
| The *walk_up* parameter controls whether ``..`` may be used to resolve
| the path.
|
| with_name(self, name)
| Return a new path with the file name changed.
|
| with_segments(self, *pathsegments)
| Construct a new path object from any number of path-like objects.
| Subclasses may override this method to customize how new path objects
| are created from methods like `iterdir()`.
|
| with_stem(self, stem)
| Return a new path with the stem changed.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from pathlib.PurePath:
|
| anchor
| The concatenation of the drive and root, or ''.
|
| drive
| The drive prefix (letter or UNC path), if any.
|
| parents
| A sequence of this path's logical parents.
|
| parts
| An object providing sequence-like access to the
| components in the filesystem path.
|
| root
| The root of the path, if any.
|
| suffixes
| A list of the final component's suffixes, if any.
|
| These include the leading periods. For example: ['.tar', '.gz']
FUNCTIONS
add_end_docstrings(*docstr)
add_start_docstrings(*docstr)
cached_path(url_or_filename, download_config=None, **download_kwargs) -> str
Given something that might be a URL (or might be a local path),
determine which. If it's a URL, download the file and cache it, and
return the path to the cached file. If it's already a local path,
make sure the file exists and then return the path.
Return:
Local path (string)
Raises:
FileNotFoundError: in case of non-recoverable file
(non-existent or no cache on disk)
ConnectionError: in case of unreachable url
and no cache on disk
ValueError: if it couldn't parse the url or filename correctly
httpx.NetworkError or requests.exceptions.ConnectionError: in case of internet connection issue
estimate_dataset_size(paths)
fsspec_get(url, temp_file, storage_options=None, desc=None, disable_tqdm=False)
fsspec_head(url, storage_options=None)
get_authentication_headers_for_url(url: str, token: Union[str, bool, NoneType] = None) -> dict
Handle the HF authentication
get_datasets_user_agent(user_agent: Union[str, dict, NoneType] = None) -> str
get_from_cache(url, cache_dir=None, force_download=False, user_agent=None, use_etag=True, token=None, storage_options=None, download_desc=None, disable_tqdm=False) -> str
Given a URL, look for the corresponding file in the local cache.
If it's not there, download it. Then return the path to the cached file.
Return:
Local path (string)
Raises:
FileNotFoundError: in case of non-recoverable file
(non-existent or no cache on disk)
ConnectionError: in case of unreachable url
and no cache on disk
hash_url_to_filename(url, etag=None)
Convert `url` into a hashed filename in a repeatable way.
If `etag` is specified, append its hash to the url's, delimited
by a period.
If the url ends with .h5 (Keras HDF5 weights) adds '.h5' to the name
so that TF 2.0 can identify it as a HDF5 file
(see https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1380)
is_local_path(url_or_filename: str) -> bool
is_relative_path(url_or_filename: str) -> bool
is_remote_url(url_or_filename: str) -> bool
readline(f: io.RawIOBase)
relative_to_absolute_path(path: ~T) -> ~T
Convert relative path to absolute path.
stack_multiprocessing_download_progress_bars()
url_or_path_join(base_name: str, *pathnames: str) -> str
url_or_path_parent(url_or_path: str) -> str
xbasename(a)
This function extends os.path.basename to support the "::" hop separator. It supports both paths and urls.
A shorthand, particularly useful where you have multiple hops, is to “chain” the URLs with the special separator "::".
This is used to access files inside a zip file over http for example.
Let's say you have a zip file at https://host.com/archive.zip, and you want to access the file inside the zip file at /folder1/file.txt.
Then you can just chain the url this way:
zip://folder1/file.txt::https://host.com/archive.zip
The xbasename function allows you to apply the basename on the first path of the chain.
Example::
>>> xbasename("zip://folder1/file.txt::https://host.com/archive.zip")
file.txt
xdirname(a)
This function extends os.path.dirname to support the "::" hop separator. It supports both paths and urls.
A shorthand, particularly useful where you have multiple hops, is to “chain” the URLs with the special separator "::".
This is used to access files inside a zip file over http for example.
Let's say you have a zip file at https://host.com/archive.zip, and you want to access the file inside the zip file at /folder1/file.txt.
Then you can just chain the url this way:
zip://folder1/file.txt::https://host.com/archive.zip
The xdirname function allows you to apply the dirname on the first path of the chain.
Example::
>>> xdirname("zip://folder1/file.txt::https://host.com/archive.zip")
zip://folder1::https://host.com/archive.zip
xet_parse(source, parser=None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
Extend `xml.etree.ElementTree.parse` function to support remote files.
Args:
source: File path or file object.
parser (`XMLParser`, *optional*, default `XMLParser`): Parser instance.
download_config : mainly use token or storage_options to support different platforms and auth types.
Returns:
`xml.etree.ElementTree.Element`: Root element of the given source document.
xexists(urlpath: str, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
Extend `os.path.exists` function to support both local and remote files.
Args:
urlpath (`str`): URL path.
download_config : mainly use token or storage_options to support different platforms and auth types.
Returns:
`bool`
xgetsize(path, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> int
Extend `os.path.getsize` function to support remote files.
Args:
path (`str`): URL path.
download_config : mainly use token or storage_options to support different platforms and auth types.
Returns:
`int`: optional
xglob(urlpath, *, recursive=False, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
Extend `glob.glob` function to support remote files.
Args:
urlpath (`str`): URL path with shell-style wildcard patterns.
recursive (`bool`, default `False`): Whether to match the "**" pattern recursively to zero or more
directories or subdirectories.
download_config : mainly use token or storage_options to support different platforms and auth types.
Returns:
`list` of `str`
xgzip_open(filepath_or_buffer, *args, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, **kwargs)
xisdir(path, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> bool
Extend `os.path.isdir` function to support remote files.
Args:
path (`str`): URL path.
download_config : mainly use token or storage_options to support different platforms and auth types.
Returns:
`bool`
xisfile(path, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> bool
Extend `os.path.isfile` function to support remote files.
Args:
path (`str`): URL path.
download_config : mainly use token or storage_options to support different platforms and auth types.
Returns:
`bool`
xjoin(a, *p)
This function extends os.path.join to support the "::" hop separator. It supports both paths and urls.
A shorthand, particularly useful where you have multiple hops, is to “chain” the URLs with the special separator "::".
This is used to access files inside a zip file over http for example.
Let's say you have a zip file at https://host.com/archive.zip, and you want to access the file inside the zip file at /folder1/file.txt.
Then you can just chain the url this way:
zip://folder1/file.txt::https://host.com/archive.zip
The xjoin function allows you to apply the join on the first path of the chain.
Example::
>>> xjoin("zip://folder1::https://host.com/archive.zip", "file.txt")
zip://folder1/file.txt::https://host.com/archive.zip
xlistdir(path: str, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> list[str]
Extend `os.listdir` function to support remote files.
Args:
path (`str`): URL path.
download_config : mainly use token or storage_options to support different platforms and auth types.
Returns:
`list` of `str`
xnumpy_load(filepath_or_buffer, *args, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, **kwargs)
xopen(file: str, mode='r', *args, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, **kwargs)
Extend `open` function to support remote files using `fsspec`.
It also has a retry mechanism in case connection fails.
The `args` and `kwargs` are passed to `fsspec.open`, except `token` which is used for queries to private repos on huggingface.co
Args:
file (`str`): Path name of the file to be opened.
mode (`str`, *optional*, default "r"): Mode in which the file is opened.
*args: Arguments to be passed to `fsspec.open`.
download_config : mainly use token or storage_options to support different platforms and auth types.
**kwargs: Keyword arguments to be passed to `fsspec.open`.
Returns:
file object
xpandas_read_csv(filepath_or_buffer, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, **kwargs)
xpandas_read_excel(filepath_or_buffer, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, **kwargs)
xpyarrow_parquet_read_table(filepath_or_buffer, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, **kwargs)
xrelpath(path, start=None)
Extend `os.path.relpath` function to support remote files.
Args:
path (`str`): URL path.
start (`str`): Start URL directory path.
Returns:
`str`
xsio_loadmat(filepath_or_buffer, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, **kwargs)
xsplit(a)
This function extends os.path.split to support the "::" hop separator. It supports both paths and urls.
A shorthand, particularly useful where you have multiple hops, is to “chain” the URLs with the special separator "::".
This is used to access files inside a zip file over http for example.
Let's say you have a zip file at https://host.com/archive.zip, and you want to access the file inside the zip file at /folder1/file.txt.
Then you can just chain the url this way:
zip://folder1/file.txt::https://host.com/archive.zip
The xsplit function allows you to apply the xsplit on the first path of the chain.
Example::
>>> xsplit("zip://folder1/file.txt::https://host.com/archive.zip")
('zip://folder1::https://host.com/archive.zip', 'file.txt')
xsplitext(a)
This function extends os.path.splitext to support the "::" hop separator. It supports both paths and urls.
A shorthand, particularly useful where you have multiple hops, is to “chain” the URLs with the special separator "::".
This is used to access files inside a zip file over http for example.
Let's say you have a zip file at https://host.com/archive.zip, and you want to access the file inside the zip file at /folder1/file.txt.
Then you can just chain the url this way:
zip://folder1/file.txt::https://host.com/archive.zip
The xsplitext function allows you to apply the splitext on the first path of the chain.
Example::
>>> xsplitext("zip://folder1/file.txt::https://host.com/archive.zip")
('zip://folder1/file::https://host.com/archive.zip', '.txt')
xwalk(urlpath, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, **kwargs)
Extend `os.walk` function to support remote files.
Args:
urlpath (`str`): URL root path.
download_config : mainly use token or storage_options to support different platforms and auth types.
**kwargs: Additional keyword arguments forwarded to the underlying filesystem.
Yields:
`tuple`: 3-tuple (dirpath, dirnames, filenames).
xxml_dom_minidom_parse(filename_or_file, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, **kwargs)
Extend `xml.dom.minidom.parse` function to support remote files.
Args:
filename_or_file (`str` or file): File path or file object.
download_config : mainly use token or storage_options to support different platforms and auth types.
**kwargs (optional): Additional keyword arguments passed to `xml.dom.minidom.parse`.
Returns:
:obj:`xml.dom.minidom.Document`: Parsed document.
DATA
BASE_KNOWN_EXTENSIONS = ['txt', 'csv', 'json', 'jsonl', 'tsv', 'conll'...
COMPRESSION_EXTENSION_TO_PROTOCOL = {'bz2': 'bz2', 'eval': 'zip', 'gz'...
COMPRESSION_FILESYSTEMS = [<class 'datasets.filesystems.compression.Bz...
CONNECTION_ERRORS_TO_RETRY = (<class 'aiohttp.client_exceptions.Client...
INCOMPLETE_SUFFIX = '.incomplete'
MAGIC_NUMBER_MAX_LENGTH = 6
MAGIC_NUMBER_TO_COMPRESSION_PROTOCOL = {b'\x04"M\x18': 'lz4', b'\x1f\x...
MAGIC_NUMBER_TO_UNSUPPORTED_COMPRESSION_PROTOCOL = {b'Rar!': 'rar'}
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
RATE_LIMIT_CODE = 429
SERVER_UNAVAILABLE_CODE = 504
SINGLE_FILE_COMPRESSION_EXTENSION_TO_PROTOCOL = {'bz2': 'bz2', 'gz': '...
SINGLE_FILE_COMPRESSION_PROTOCOLS = {'bz2', 'gzip', 'lz4', 'xz', 'zstd...
SINGLE_SLASH_AFTER_PROTOCOL_PATTERN = re.compile('(?<!:):/')
T = ~T
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.utils.file_utils (WARNING)>
VERSION
5.0.1
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\file_utils.py
datasets.filesystems
Python Library Documentation: package datasets.filesystems in datasets
NAME
datasets.filesystems
PACKAGE CONTENTS
compression
FUNCTIONS
is_remote_filesystem(fs: fsspec.spec.AbstractFileSystem) -> bool
Checks if `fs` is a remote filesystem.
Args:
fs (`fsspec.spec.AbstractFileSystem`):
An abstract super-class for pythonic file-systems, e.g. `fsspec.filesystem('file')` or `s3fs.S3FileSystem`.
rename(fs: fsspec.spec.AbstractFileSystem, src: str, dst: str)
Renames the file `src` in `fs` to `dst`.
DATA
COMPRESSION_FILESYSTEMS = [<class 'datasets.filesystems.compression.Bz...
EXTENSION_TO_COMPRESSION_FS_FILE_CLS = {'.bz2': <class 'bz2.BZ2File'>,...
List = typing.List
A generic version of list.
__annotations__ = {'COMPRESSION_FILESYSTEMS': list[datasets.filesystem...
extension = '.zstd'
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\filesystems\__init__.py
datasets.fingerprint
Python Library Documentation: module datasets.fingerprint in datasets
NAME
datasets.fingerprint
CLASSES
builtins.object
Hasher
class Hasher(builtins.object)
| Hasher that accepts python objects as inputs.
|
| Methods defined here:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| hexdigest(self) -> str
|
| update(self, value: Any) -> None
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| hash(value: Any) -> str
|
| hash_bytes(value: Union[bytes, list[bytes]]) -> str
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'dispatch': <class 'dict'>}
|
| dispatch = {}
FUNCTIONS
disable_caching()
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use [`~datasets.Dataset.save_to_disk`] to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use
the `download_mode` parameter in [`~datasets.load_dataset`].
enable_caching()
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use [`~datasets.Dataset.save_to_disk`] to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use
the `download_mode` parameter in [`~datasets.load_dataset`].
fingerprint_transform(inplace: bool, use_kwargs: Optional[list[str]] = None, ignore_kwargs: Optional[list[str]] = None, fingerprint_names: Optional[list[str]] = None, randomized_function: bool = False, version: Optional[str] = None)
Wrapper for dataset transforms to update the dataset fingerprint using ``update_fingerprint``
Args:
inplace (:obj:`bool`): If inplace is True, the fingerprint of the dataset is updated inplace.
Otherwise, a parameter "new_fingerprint" is passed to the wrapped method that should take care of
setting the fingerprint of the returned Dataset.
use_kwargs (:obj:`List[str]`, optional): optional white list of argument names to take into account
to update the fingerprint to the wrapped method that should take care of
setting the fingerprint of the returned Dataset. By default all the arguments are used.
ignore_kwargs (:obj:`List[str]`, optional): optional black list of argument names to take into account
to update the fingerprint. Note that ignore_kwargs prevails on use_kwargs.
fingerprint_names (:obj:`List[str]`, optional, defaults to ["new_fingerprint"]):
If the dataset transforms is not inplace and returns a DatasetDict, then it can require
several fingerprints (one per dataset in the DatasetDict). By specifying fingerprint_names,
one fingerprint named after each element of fingerprint_names is going to be passed.
randomized_function (:obj:`bool`, defaults to False): If the dataset transform is random and has
optional parameters "seed" and "generator", then you can set randomized_function to True.
This way, even if users set "seed" and "generator" to None, then the fingerprint is
going to be randomly generated depending on numpy's current state. In this case, the
generator is set to np.random.default_rng(np.random.get_state()[1][0]).
version (:obj:`str`, optional): version of the transform. The version is taken into account when
computing the fingerprint. If a datase transform changes (or at least if the output data
that are cached changes), then one should increase the version. If the version stays the
same, then old cached data could be reused that are not compatible with the new transform.
It should be in the format "MAJOR.MINOR.PATCH".
format_kwargs_for_fingerprint(func: Callable, args: tuple, kwargs: dict[str, typing.Any], use_kwargs: Optional[list[str]] = None, ignore_kwargs: Optional[list[str]] = None, randomized_function: bool = False) -> dict[str, typing.Any]
Format the kwargs of a transform to the format that will be used to update the fingerprint.
format_transform_for_fingerprint(func: Callable, version: Optional[str] = None) -> str
Format a transform to the format that will be used to update the fingerprint.
generate_fingerprint(dataset: 'Dataset') -> str
generate_random_fingerprint(nbits: int = 64) -> str
get_datasets_with_cache_file_in_temp_dir()
get_temporary_cache_files_directory() -> str
Return a directory that is deleted when session closes.
is_caching_enabled() -> bool
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use [`~datasets.Dataset.save_to_disk`]] to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use
the `download_mode` parameter in [`~datasets.load_dataset`].
maybe_register_dataset_for_temp_dir_deletion(dataset)
This function registers the datasets that have cache files in _TEMP_DIR_FOR_TEMP_CACHE_FILES in order
to properly delete them before deleting the temporary directory.
The temporary directory _TEMP_DIR_FOR_TEMP_CACHE_FILES is used when caching is disabled.
update_fingerprint(fingerprint, transform, transform_args)
validate_fingerprint(fingerprint: str, max_length=64)
Make sure the fingerprint is a non-empty string that is not longer that max_length=64 by default,
so that the fingerprint can be used to name cache files without issues.
DATA
Callable = typing.Callable
Deprecated alias to collections.abc.Callable.
Callable[[int], str] signifies a function that takes a single
parameter of type int and returns a str.
The subscription syntax must always be used with exactly two
values: the argument list and the return type.
The argument list must be a list of types, a ParamSpec,
Concatenate or ellipsis. The return type must be a single type.
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
INVALID_WINDOWS_CHARACTERS_IN_PATH = r'<>:/\|?*'
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
TYPE_CHECKING = False
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
__annotations__ = {'_DATASETS_WITH_TABLE_IN_TEMP_DIR': typing.Optional...
fingerprint_rng = <random.Random object>
fingerprint_warnings = {}
logger = <Logger datasets.fingerprint (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\fingerprint.py
datasets.formatting
Python Library Documentation: package datasets.formatting in datasets
NAME
datasets.formatting
DESCRIPTION
# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
PACKAGE CONTENTS
formatting
jax_formatter
np_formatter
polars_formatter
tf_formatter
torch_formatter
FUNCTIONS
get_format_type_from_alias(format_type: Optional[str]) -> Optional[str]
If the given format type is a known alias, then return its main type name. Otherwise return the type with no change.
get_formatter(format_type: Optional[str], **format_kwargs) -> datasets.formatting.formatting.Formatter
Factory function to get a Formatter given its type name and keyword arguments.
A formatter is an object that extracts and formats data from pyarrow table.
It defines the formatting for rows, columns and batches.
If the formatter for a given type name doesn't exist or is not available, an error is raised.
DATA
Dict = typing.Dict
A generic version of dict.
List = typing.List
A generic version of list.
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
Type = typing.Type
Deprecated alias to builtins.type.
builtins.type or typing.Type can be used to annotate class objects.
For example, suppose we have the following classes::
class User: ... # Abstract base for User classes
class BasicUser(User): ...
class ProUser(User): ...
class TeamUser(User): ...
And a function that takes a class argument that's a subclass of
User and returns an instance of the corresponding class::
def new_user[U](user_class: Type[U]) -> U:
user = user_class()
# (Here we could write the user object to a database)
return user
joe = new_user(BasicUser)
At this point the type checker knows that joe has type BasicUser.
__annotations__ = {'_FORMAT_TYPES': dict[typing.Optional[str], type[da...
logger = <Logger datasets.formatting (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\formatting\__init__.py
datasets.get_dataset_config_info
Python Library Documentation: function get_dataset_config_info in module datasets.inspect
get_dataset_config_info(path: str, config_name: Optional[str] = None, data_files: Union[str, collections.abc.Sequence[str], collections.abc.Mapping[str, Union[str, collections.abc.Sequence[str]]], NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, **config_kwargs) -> datasets.info.DatasetInfo
Get the meta information (DatasetInfo) about a dataset for a particular config
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
config_name (:obj:`str`, optional): Defining the name of the dataset configuration.
data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s).
download_config (:class:`~download.DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode` or :obj:`str`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset to load.
As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch.
You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository.
token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If True, or not specified, will get token from `"~/.huggingface"`.
**config_kwargs (additional keyword arguments): optional attributes for builder class which will override the attributes if supplied.
datasets.get_dataset_config_names
Python Library Documentation: function get_dataset_config_names in module datasets.inspect
get_dataset_config_names(path: str, revision: Union[datasets.utils.version.Version, str, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, data_files: Union[str, list, dict, NoneType] = None, **download_kwargs)
Get the list of available config names for a particular dataset.
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
revision (`Union[str, datasets.Version]`, *optional*):
If specified, the dataset module will be loaded from the datasets repository at this version.
By default:
- it is set to the local version of the lib.
- it will also try to load it from the main branch if it's not available at the local version of the lib.
Specifying a version that is different from your local version of the lib might cause compatibility issues.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
data_files (`Union[Dict, List, str]`, *optional*):
Defining the data_files of the dataset configuration.
**download_kwargs (additional keyword arguments):
Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied,
for example `token`.
Example:
```py
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names("nyu-mll/glue")
['cola',
'sst2',
'mrpc',
'qqp',
'stsb',
'mnli',
'mnli_mismatched',
'mnli_matched',
'qnli',
'rte',
'wnli',
'ax']
```
datasets.get_dataset_default_config_name
Python Library Documentation: function get_dataset_default_config_name in module datasets.inspect
get_dataset_default_config_name(path: str, revision: Union[datasets.utils.version.Version, str, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, data_files: Union[str, list, dict, NoneType] = None, **download_kwargs) -> Optional[str]
Get the default config name for a particular dataset.
Can return None only if the dataset has multiple configurations and no default configuration.
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
revision (`Union[str, datasets.Version]`, *optional*):
If specified, the dataset module will be loaded from the datasets repository at this version.
By default:
- it is set to the local version of the lib.
- it will also try to load it from the main branch if it's not available at the local version of the lib.
Specifying a version that is different from your local version of the lib might cause compatibility issues.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
data_files (`Union[Dict, List, str]`, *optional*):
Defining the data_files of the dataset configuration.
**download_kwargs (additional keyword arguments):
Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied,
for example `token`.
Returns:
Optional[str]: the default config name if there is one
Example:
```py
>>> from datasets import get_dataset_default_config_name
>>> get_dataset_default_config_name("openbookqa")
'main'
```
datasets.get_dataset_infos
Python Library Documentation: function get_dataset_infos in module datasets.inspect
get_dataset_infos(path: str, data_files: Union[str, list, dict, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, **config_kwargs)
Get the meta information about a dataset, returned as a dict mapping config name to DatasetInfoDict.
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
revision (`Union[str, datasets.Version]`, *optional*):
If specified, the dataset module will be loaded from the datasets repository at this version.
By default:
- it is set to the local version of the lib.
- it will also try to load it from the main branch if it's not available at the local version of the lib.
Specifying a version that is different from your local version of the lib might cause compatibility issues.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
data_files (`Union[Dict, List, str]`, *optional*):
Defining the data_files of the dataset configuration.
token (`str` or `bool`, *optional*):
Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If `True`, or not specified, will get token from `"~/.huggingface"`.
**config_kwargs (additional keyword arguments):
Optional attributes for builder class which will override the attributes if supplied.
Example:
```py
>>> from datasets import get_dataset_infos
>>> get_dataset_infos('cornell-movie-review-data/rotten_tomatoes')
{'default': DatasetInfo(description="Movie Review Dataset.
This is a dataset of containing 5,331 positive and 5,331 negative processed
sentences from Rotten Tomatoes movie reviews...), ...}
```
datasets.get_dataset_split_names
Python Library Documentation: function get_dataset_split_names in module datasets.inspect
get_dataset_split_names(path: str, config_name: Optional[str] = None, data_files: Union[str, collections.abc.Sequence[str], collections.abc.Mapping[str, Union[str, collections.abc.Sequence[str]]], NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, **config_kwargs)
Get the list of available splits for a particular config and dataset.
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
config_name (`str`, *optional*):
Defining the name of the dataset configuration.
data_files (`str` or `Sequence` or `Mapping`, *optional*):
Path(s) to source data file(s).
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
revision ([`Version`] or `str`, *optional*):
Version of the dataset to load.
As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch.
You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository.
token (`str` or `bool`, *optional*):
Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If `True`, or not specified, will get token from `"~/.huggingface"`.
**config_kwargs (additional keyword arguments):
Optional attributes for builder class which will override the attributes if supplied.
Example:
```py
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names('cornell-movie-review-data/rotten_tomatoes')
['train', 'validation', 'test']
```
datasets.hub
Python Library Documentation: module datasets.utils.hub in datasets.utils
NAME
datasets.utils.hub
DATA
hf_dataset_url = functools.partial(<function hf_hub_url at 0x000001A65...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\hub.py
datasets.info
Python Library Documentation: module datasets.info in datasets
NAME
datasets.info - DatasetInfo record information we know about a dataset.
DESCRIPTION
This includes things that we know about the dataset statically, i.e.:
- description
- canonical location
- does it have validation and tests splits
- size
- etc.
This also includes the things that can and should be computed once we've
processed the dataset as well:
- number of examples (in each split)
- etc.
CLASSES
builtins.Exception(builtins.BaseException)
MissingCachedSizesConfigError
NonMatchingCachedSizesError
builtins.dict(builtins.object)
DatasetInfosDict
builtins.object
DatasetInfo
DownloadChecksumsEntryData
PostProcessedInfo
SupervisedKeysData
class DatasetInfo(builtins.object)
| DatasetInfo(description: str = <factory>, citation: str = <factory>, homepage: str = <factory>, license: str = <factory>, features: Optional[datasets.features.features.Features] = None, post_processed: Optional[datasets.info.PostProcessedInfo] = None, supervised_keys: Optional[datasets.info.SupervisedKeysData] = None, builder_name: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, version: Union[str, datasets.utils.version.Version, NoneType] = None, splits: Optional[datasets.splits.SplitDict] = None, download_checksums: Optional[dict] = None, download_size: Optional[int] = None, post_processing_size: Optional[int] = None, dataset_size: Optional[int] = None, size_in_bytes: Optional[int] = None) -> None
|
| Information about a dataset.
|
| `DatasetInfo` documents datasets, including its name, version, and features.
| See the constructor arguments and properties for a full list.
|
| Not all fields are known on construction and may be updated later.
|
| Attributes:
| description (`str`):
| A description of the dataset.
| citation (`str`):
| A BibTeX citation of the dataset.
| homepage (`str`):
| A URL to the official homepage for the dataset.
| license (`str`):
| The dataset's license. It can be the name of the license or a paragraph containing the terms of the license.
| features ([`Features`], *optional*):
| The features used to specify the dataset's column types.
| post_processed (`PostProcessedInfo`, *optional*):
| Deprecated. Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index.
| supervised_keys (`SupervisedKeysData`, *optional*):
| Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS).
| builder_name (`str`, *optional*):
| The name of the `GeneratorBasedBuilder` subclass used to create the dataset. It is also the snake_case version of the dataset builder class name.
| config_name (`str`, *optional*):
| The name of the configuration derived from [`BuilderConfig`].
| version (`str` or [`Version`], *optional*):
| The version of the dataset.
| splits (`dict`, *optional*):
| The mapping between split name and metadata.
| download_checksums (`dict`, *optional*):
| The mapping between the URL to download the dataset's checksums and corresponding metadata.
| download_size (`int`, *optional*):
| The size of the files to download to generate the dataset, in bytes.
| post_processing_size (`int`, *optional*):
| Deprecated. Size of the dataset in bytes after post-processing, if any.
| dataset_size (`int`, *optional*):
| The combined size in bytes of the Arrow tables for all splits.
| size_in_bytes (`int`, *optional*):
| The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files).
| **config_kwargs (additional keyword arguments):
| Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`].
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, description: str = <factory>, citation: str = <factory>, homepage: str = <factory>, license: str = <factory>, features: Optional[datasets.features.features.Features] = None, post_processed: Optional[datasets.info.PostProcessedInfo] = None, supervised_keys: Optional[datasets.info.SupervisedKeysData] = None, builder_name: Optional[str] = None, dataset_name: Optional[str] = None, config_name: Optional[str] = None, version: Union[str, datasets.utils.version.Version, NoneType] = None, splits: Optional[datasets.splits.SplitDict] = None, download_checksums: Optional[dict] = None, download_size: Optional[int] = None, post_processing_size: Optional[int] = None, dataset_size: Optional[int] = None, size_in_bytes: Optional[int] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| copy(self) -> 'DatasetInfo'
|
| update(self, other_dataset_info: 'DatasetInfo', ignore_none=True)
|
| write_to_directory(self, dataset_info_dir, pretty_print=False, storage_options: Optional[dict] = None)
| Write `DatasetInfo` and license (if present) as JSON files to `dataset_info_dir`.
|
| Args:
| dataset_info_dir (`str`):
| Destination directory.
| pretty_print (`bool`, defaults to `False`):
| If `True`, the JSON will be pretty-printed with the indent level of 4.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.9.0"/>
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.info.write_to_directory("/path/to/directory/")
| ```
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_dict(dataset_info_dict: dict) -> 'DatasetInfo'
|
| from_directory(dataset_info_dir: str, storage_options: Optional[dict] = None) -> 'DatasetInfo'
| Create [`DatasetInfo`] from the JSON file in `dataset_info_dir`.
|
| This function updates all the dynamically generated fields (num_examples,
| hash, time of creation,...) of the [`DatasetInfo`].
|
| This will overwrite all previous metadata.
|
| Args:
| dataset_info_dir (`str`):
| The directory containing the metadata file. This
| should be the root directory of a specific dataset version.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.9.0"/>
|
| Example:
|
| ```py
| >>> from datasets import DatasetInfo
| >>> ds_info = DatasetInfo.from_directory("/path/to/directory/")
| ```
|
| from_merge(dataset_infos: list['DatasetInfo'])
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_INCLUDED_INFO_IN_YAML': typing.ClassVar[list[str]...
|
| __dataclass_fields__ = {'_INCLUDED_INFO_IN_YAML': Field(name='_INCLUDE...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('description', 'citation', 'homepage', 'license', 'f...
|
| builder_name = None
|
| config_name = None
|
| dataset_name = None
|
| dataset_size = None
|
| download_checksums = None
|
| download_size = None
|
| features = None
|
| post_processed = None
|
| post_processing_size = None
|
| size_in_bytes = None
|
| splits = None
|
| supervised_keys = None
|
| version = None
class DatasetInfosDict(builtins.dict)
| Method resolution order:
| DatasetInfosDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| to_dataset_card_data(self, dataset_card_data: huggingface_hub.repocard_data.DatasetCardData) -> None
|
| write_to_directory(self, dataset_infos_dir, overwrite=False, pretty_print=False) -> None
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_dataset_card_data(dataset_card_data: huggingface_hub.repocard_data.DatasetCardData) -> 'DatasetInfosDict'
|
| from_directory(dataset_infos_dir) -> 'DatasetInfosDict'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (dict[str, datasets.info.DatasetInfo],)
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class DownloadChecksumsEntryData(builtins.object)
| DownloadChecksumsEntryData(key: str = '', value: str = '') -> None
|
| DownloadChecksumsEntryData(key: str = '', value: str = '')
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, key: str = '', value: str = '') -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'key': <class 'str'>, 'value': <class 'str'>}
|
| __dataclass_fields__ = {'key': Field(name='key',type=<class 'str'>,def...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('key', 'value')
|
| key = ''
|
| value = ''
class MissingCachedSizesConfigError(builtins.Exception)
| The expected cached sizes of the download file are missing.
|
| Method resolution order:
| MissingCachedSizesConfigError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NonMatchingCachedSizesError(builtins.Exception)
| The prepared split doesn't have expected sizes.
|
| Method resolution order:
| NonMatchingCachedSizesError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class PostProcessedInfo(builtins.object)
| PostProcessedInfo(features: Optional[datasets.features.features.Features] = None, resources_checksums: Optional[dict] = None) -> None
|
| PostProcessedInfo(features: Optional[datasets.features.features.Features] = None, resources_checksums: Optional[dict] = None)
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, features: Optional[datasets.features.features.Features] = None, resources_checksums: Optional[dict] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_dict(post_processed_info_dict: dict) -> 'PostProcessedInfo'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'features': typing.Optional[datasets.features.featu...
|
| __dataclass_fields__ = {'features': Field(name='features',type=typing....
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('features', 'resources_checksums')
|
| features = None
|
| resources_checksums = None
class SupervisedKeysData(builtins.object)
| SupervisedKeysData(input: str = '', output: str = '') -> None
|
| SupervisedKeysData(input: str = '', output: str = '')
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, input: str = '', output: str = '') -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'input': <class 'str'>, 'output': <class 'str'>}
|
| __dataclass_fields__ = {'input': Field(name='input',type=<class 'str'>...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('input', 'output')
|
| input = ''
|
| output = ''
DATA
ClassVar = typing.ClassVar
Special type construct to mark class variables.
An annotation wrapped in ClassVar indicates that a given
attribute is intended to be used as a class variable and
should not be set on instances of that class.
Usage::
class Starship:
stats: ClassVar[dict[str, int]] = {} # class variable
damage: int = 10 # instance variable
ClassVar accepts only types and cannot be further subscribed.
Note that ClassVar is not a class itself, and should not
be used with isinstance() or issubclass().
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.info (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\info.py
datasets.info_utils
Python Library Documentation: module datasets.utils.info_utils in datasets.utils
NAME
datasets.utils.info_utils
CLASSES
enum.Enum(builtins.object)
VerificationMode
class VerificationMode(enum.Enum)
| VerificationMode(*values)
|
| `Enum` that specifies which verification checks to run.
|
| The default mode is `BASIC_CHECKS`, which will perform only rudimentary checks to avoid slowdowns
| when generating/downloading a dataset for the first time.
|
| The verification modes:
|
| | | Verification checks |
| |---------------------------|------------------------------------------------------------------------------ |
| | `ALL_CHECKS` | Split checks and validity (number of files, checksums) of downloaded files |
| | `BASIC_CHECKS` (default) | Same as `ALL_CHECKS` but without checking downloaded files |
| | `NO_CHECKS` | None |
|
| Method resolution order:
| VerificationMode
| enum.Enum
| builtins.object
|
| Data and other attributes defined here:
|
| ALL_CHECKS = <VerificationMode.ALL_CHECKS: 'all_checks'>
|
| BASIC_CHECKS = <VerificationMode.BASIC_CHECKS: 'basic_checks'>
|
| NO_CHECKS = <VerificationMode.NO_CHECKS: 'no_checks'>
|
| ----------------------------------------------------------------------
| Data descriptors inherited from enum.Enum:
|
| name
| The name of the Enum member.
|
| value
| The value of the Enum member.
|
| ----------------------------------------------------------------------
| Static methods inherited from enum.EnumType:
|
| __contains__(value)
| Return True if `value` is in `cls`.
|
| `value` is in `cls` if:
| 1) `value` is a member of `cls`, or
| 2) `value` is the value of one of the `cls`'s members.
|
| __getitem__(name)
| Return the member matching `name`.
|
| __iter__()
| Return members in definition order.
|
| __len__()
| Return the number of members (no aliases)
|
| ----------------------------------------------------------------------
| Readonly properties inherited from enum.EnumType:
|
| __members__
| Returns a mapping of member name->value.
|
| This mapping lists all enum members, including aliases. Note that this
| is a read-only view of the internal mapping.
FUNCTIONS
get_size_checksum_dict(path: str, record_checksum: bool = False) -> dict
Compute the file size and the sha256 checksum of a file
is_small_dataset(dataset_size)
Check if `dataset_size` is smaller than `config.IN_MEMORY_MAX_SIZE`.
Args:
dataset_size (int): Dataset size in bytes.
Returns:
bool: Whether `dataset_size` is smaller than `config.IN_MEMORY_MAX_SIZE`.
verify_checksums(expected_checksums: Optional[dict], recorded_checksums: dict, verification_name=None)
verify_splits(expected_splits: Optional[dict], recorded_splits: dict)
DATA
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
logger = <Logger datasets.utils.info_utils (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\info_utils.py
datasets.inspect
Python Library Documentation: module datasets.inspect in datasets
NAME
datasets.inspect - List and inspect datasets.
CLASSES
builtins.ValueError(builtins.Exception)
SplitsNotFoundError
class SplitsNotFoundError(builtins.ValueError)
| Method resolution order:
| SplitsNotFoundError
| builtins.ValueError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.ValueError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.ValueError:
|
| __new__(*args, **kwargs) class method of builtins.ValueError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
FUNCTIONS
get_dataset_config_info(path: str, config_name: Optional[str] = None, data_files: Union[str, collections.abc.Sequence[str], collections.abc.Mapping[str, Union[str, collections.abc.Sequence[str]]], NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, **config_kwargs) -> datasets.info.DatasetInfo
Get the meta information (DatasetInfo) about a dataset for a particular config
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
config_name (:obj:`str`, optional): Defining the name of the dataset configuration.
data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s).
download_config (:class:`~download.DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode` or :obj:`str`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset to load.
As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch.
You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository.
token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If True, or not specified, will get token from `"~/.huggingface"`.
**config_kwargs (additional keyword arguments): optional attributes for builder class which will override the attributes if supplied.
get_dataset_config_names(path: str, revision: Union[datasets.utils.version.Version, str, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, data_files: Union[str, list, dict, NoneType] = None, **download_kwargs)
Get the list of available config names for a particular dataset.
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
revision (`Union[str, datasets.Version]`, *optional*):
If specified, the dataset module will be loaded from the datasets repository at this version.
By default:
- it is set to the local version of the lib.
- it will also try to load it from the main branch if it's not available at the local version of the lib.
Specifying a version that is different from your local version of the lib might cause compatibility issues.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
data_files (`Union[Dict, List, str]`, *optional*):
Defining the data_files of the dataset configuration.
**download_kwargs (additional keyword arguments):
Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied,
for example `token`.
Example:
```py
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names("nyu-mll/glue")
['cola',
'sst2',
'mrpc',
'qqp',
'stsb',
'mnli',
'mnli_mismatched',
'mnli_matched',
'qnli',
'rte',
'wnli',
'ax']
```
get_dataset_default_config_name(path: str, revision: Union[datasets.utils.version.Version, str, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, data_files: Union[str, list, dict, NoneType] = None, **download_kwargs) -> Optional[str]
Get the default config name for a particular dataset.
Can return None only if the dataset has multiple configurations and no default configuration.
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
revision (`Union[str, datasets.Version]`, *optional*):
If specified, the dataset module will be loaded from the datasets repository at this version.
By default:
- it is set to the local version of the lib.
- it will also try to load it from the main branch if it's not available at the local version of the lib.
Specifying a version that is different from your local version of the lib might cause compatibility issues.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
data_files (`Union[Dict, List, str]`, *optional*):
Defining the data_files of the dataset configuration.
**download_kwargs (additional keyword arguments):
Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied,
for example `token`.
Returns:
Optional[str]: the default config name if there is one
Example:
```py
>>> from datasets import get_dataset_default_config_name
>>> get_dataset_default_config_name("openbookqa")
'main'
```
get_dataset_infos(path: str, data_files: Union[str, list, dict, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, **config_kwargs)
Get the meta information about a dataset, returned as a dict mapping config name to DatasetInfoDict.
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
revision (`Union[str, datasets.Version]`, *optional*):
If specified, the dataset module will be loaded from the datasets repository at this version.
By default:
- it is set to the local version of the lib.
- it will also try to load it from the main branch if it's not available at the local version of the lib.
Specifying a version that is different from your local version of the lib might cause compatibility issues.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
data_files (`Union[Dict, List, str]`, *optional*):
Defining the data_files of the dataset configuration.
token (`str` or `bool`, *optional*):
Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If `True`, or not specified, will get token from `"~/.huggingface"`.
**config_kwargs (additional keyword arguments):
Optional attributes for builder class which will override the attributes if supplied.
Example:
```py
>>> from datasets import get_dataset_infos
>>> get_dataset_infos('cornell-movie-review-data/rotten_tomatoes')
{'default': DatasetInfo(description="Movie Review Dataset.
This is a dataset of containing 5,331 positive and 5,331 negative processed
sentences from Rotten Tomatoes movie reviews...), ...}
```
get_dataset_split_names(path: str, config_name: Optional[str] = None, data_files: Union[str, collections.abc.Sequence[str], collections.abc.Mapping[str, Union[str, collections.abc.Sequence[str]]], NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, **config_kwargs)
Get the list of available splits for a particular config and dataset.
Args:
path (`str`): path to the dataset repository. Can be either:
- a local path to the dataset directory containing the data files,
e.g. `'./dataset/squad'`
- a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]),
e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'`
config_name (`str`, *optional*):
Defining the name of the dataset configuration.
data_files (`str` or `Sequence` or `Mapping`, *optional*):
Path(s) to source data file(s).
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
revision ([`Version`] or `str`, *optional*):
Version of the dataset to load.
As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch.
You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository.
token (`str` or `bool`, *optional*):
Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If `True`, or not specified, will get token from `"~/.huggingface"`.
**config_kwargs (additional keyword arguments):
Optional attributes for builder class which will override the attributes if supplied.
Example:
```py
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names('cornell-movie-review-data/rotten_tomatoes')
['train', 'validation', 'test']
```
DATA
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.inspect (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\inspect.py
datasets.interleave_datasets
Python Library Documentation: function interleave_datasets in module datasets.combine
interleave_datasets(datasets: list[~DatasetType], probabilities: Optional[list[float]] = None, seed: Optional[int] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, stopping_strategy: Literal['first_exhausted', 'all_exhausted', 'all_exhausted_without_replacement'] = 'first_exhausted') -> ~DatasetType
Interleave several datasets (sources) into a single dataset.
The new dataset is constructed by alternating between the sources to get the examples.
You can use this function on a list of [`Dataset`] objects, or on a list of [`IterableDataset`] objects.
- If `probabilities` is `None` (default) the new dataset is constructed by cycling between each source to get the examples.
- If `probabilities` is not `None`, the new dataset is constructed by getting examples from a random source at a time according to the provided probabilities.
The resulting dataset ends when one of the source datasets runs out of examples except when `oversampling` is `True`,
in which case, the resulting dataset ends when all datasets have ran out of examples at least one time.
Note for iterable datasets:
* The resulting dataset's `num_shards` is the minimum of each dataset's `num_shards` to ensure good parallelism.
If some of your datasets have a very low number of shards, you may use [`IterableDataset.reshard`].
* In a distributed setup or in PyTorch DataLoader workers, the stopping strategy is applied per process.
Therefore the "first_exhausted" strategy on an sharded iterable dataset can generate less samples in total (up to 1 missing sample per subdataset per worker).
Args:
datasets (`List[Dataset]` or `List[IterableDataset]`):
List of datasets to interleave.
probabilities (`List[float]`, *optional*, defaults to `None`):
If specified, the new dataset is constructed by sampling
examples from one source at a time according to these probabilities.
seed (`int`, *optional*, defaults to `None`):
The random seed used to choose a source for each example.
info ([`DatasetInfo`], *optional*):
Dataset information, like description, citation, etc.
<Added version="2.4.0"/>
split ([`NamedSplit`], *optional*):
Name of the dataset split.
<Added version="2.4.0"/>
stopping_strategy (`str`, defaults to `first_exhausted`):
Three strategies are proposed right now, `first_exhausted`, `all_exhausted` and `all_exhausted_without_replacement`.
By default, `first_exhausted` is an undersampling strategy, i.e the dataset construction is stopped as soon as one dataset has ran out of samples.
If the strategy is `all_exhausted`, we use an oversampling strategy, i.e the dataset construction is stopped as soon as every samples of every dataset has been added at least once.
When strategy is `all_exhausted_without_replacement` we make sure that each sample in each dataset is sampled only once.
Note that if the strategy is `all_exhausted`, the interleaved dataset size can get enormous:
- with no probabilities, the resulting dataset will have `max_length_datasets*nb_dataset` samples.
- with given probabilities, the resulting dataset will have more samples if some datasets have really low probability of visiting.
Returns:
[`Dataset`] or [`IterableDataset`]: Return type depends on the input `datasets`
parameter. `Dataset` if the input is a list of `Dataset`, `IterableDataset` if the input is a list of
`IterableDataset`.
Example:
For regular datasets (map-style):
```python
>>> from datasets import Dataset, interleave_datasets
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42, stopping_strategy="all_exhausted")
>>> dataset["a"]
[10, 0, 11, 1, 2, 20, 12, 10, 0, 1, 2, 21, 0, 11, 1, 2, 0, 1, 12, 2, 10, 0, 22]
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42)
>>> dataset["a"]
[10, 0, 11, 1, 2]
>>> dataset = interleave_datasets([d1, d2, d3])
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22, 23, 24]})
>>> dataset = interleave_datasets([d1, d2, d3])
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22, 0, 13, 23, 1, 10, 24]
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42)
>>> dataset["a"]
[10, 0, 11, 1, 2]
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42, stopping_strategy="all_exhausted")
>>> dataset["a"]
[10, 0, 11, 1, 2, 20, 12, 13, ..., 0, 1, 2, 0, 24]
For datasets in streaming mode (iterable):
>>> from datasets import interleave_datasets
>>> d1 = load_dataset('allenai/c4', 'es', split='train', streaming=True)
>>> d2 = load_dataset('allenai/c4', 'fr', split='train', streaming=True)
>>> dataset = interleave_datasets([d1, d2])
>>> iterator = iter(dataset)
>>> next(iterator)
{'text': 'Comprar Zapatillas para niña en chancla con goma por...'}
>>> next(iterator)
{'text': 'Le sacre de philippe ier, 23 mai 1059 - Compte Rendu...'
```
datasets.is_caching_enabled
Python Library Documentation: function is_caching_enabled in module datasets.fingerprint
is_caching_enabled() -> bool
When applying transforms on a dataset, the data are stored in cache files.
The caching mechanism allows to reload an existing cache file if it's already been computed.
Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated
after each transform.
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
More precisely, if the caching is disabled:
- cache files are always recreated
- cache files are written to a temporary directory that is deleted when session closes
- cache files are named using a random hash instead of the dataset fingerprint
- use [`~datasets.Dataset.save_to_disk`]] to save a transformed dataset or it will be deleted when session closes
- caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use
the `download_mode` parameter in [`~datasets.load_dataset`].
datasets.is_progress_bar_enabled
Python Library Documentation: function is_progress_bar_enabled in module datasets.utils.tqdm is_progress_bar_enabled()
datasets.iterable_dataset
Python Library Documentation: module datasets.iterable_dataset in datasets
NAME
datasets.iterable_dataset
CLASSES
builtins.Exception(builtins.BaseException)
DataSourcesShufflingDisallowed
builtins.object
DistributedConfig
FormattingConfig
IterableColumn
datasets.arrow_dataset.DatasetInfoMixin(builtins.object)
IterableDataset
_BaseExamplesIterable(builtins.object)
ArrowExamplesIterable
BufferShuffledExamplesIterable
CyclingMultiSourcesExamplesIterable
RandomlyCyclingMultiSourcesExamplesIterable
ExamplesIterable
FormattedExamplesIterable
HorizontallyConcatenatedMultiSourcesExamplesIterable
MappedExamplesIterable
FilteredExamplesIterable
RebatchedArrowExamplesIterable
RepeatExamplesIterable
SelectColumnsIterable
SkipExamplesIterable
StepExamplesIterable
TakeExamplesIterable
VerticallyConcatenatedMultiSourcesExamplesIterable
class ArrowExamplesIterable(_BaseExamplesIterable)
| ArrowExamplesIterable(generate_tables_fn: Callable[..., collections.abc.Iterator[tuple[Union[int, str, tuple[int, int], ForwardRef('BuilderKey')], pyarrow.lib.Table]]], kwargs: dict, generate_more_kwargs_fn: Optional[Callable[..., collections.abc.Iterator[dict]]] = None, sleep_on_threads_shutdown: bool = False)
|
| Method resolution order:
| ArrowExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, generate_tables_fn: Callable[..., collections.abc.Iterator[tuple[Union[int, str, tuple[int, int], ForwardRef('BuilderKey')], pyarrow.lib.Table]]], kwargs: dict, generate_more_kwargs_fn: Optional[Callable[..., collections.abc.Iterator[dict]]] = None, sleep_on_threads_shutdown: bool = False)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'ArrowExamplesIterable'
| Split shars into more shards if possible.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'ArrowExamplesIterable'
| Keep only the requested shard.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'ArrowExamplesIterable'
| Either shuffle the shards/sources of the dataset, or propagate the shuffling to the underlying iterable.
| If the order of the shards must stay fixed (when using .skip or .take for example), then this method returns self.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| features
|
| is_typed
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class BufferShuffledExamplesIterable(_BaseExamplesIterable)
| BufferShuffledExamplesIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, buffer_size: int, generator: numpy.random._generator.Generator)
|
| Method resolution order:
| BufferShuffledExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, buffer_size: int, generator: numpy.random._generator.Generator)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| load_state_dict(self, state_dict: dict) -> dict
|
| reshard_data_sources(self) -> 'BufferShuffledExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'BufferShuffledExamplesIterable'
| Keep only the requested shard.
|
| shift_rngs(self, value: int) -> '_BaseExamplesIterable'
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'BufferShuffledExamplesIterable'
| Shuffle the wrapped examples iterable as well as the shuffling buffer.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class CyclingMultiSourcesExamplesIterable(_BaseExamplesIterable)
| CyclingMultiSourcesExamplesIterable(ex_iterables: list[datasets.iterable_dataset._BaseExamplesIterable], stopping_strategy: Literal['first_exhausted', 'all_exhausted', 'all_exhausted_without_replacement'] = 'first_exhausted')
|
| Method resolution order:
| CyclingMultiSourcesExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterables: list[datasets.iterable_dataset._BaseExamplesIterable], stopping_strategy: Literal['first_exhausted', 'all_exhausted', 'all_exhausted_without_replacement'] = 'first_exhausted')
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'CyclingMultiSourcesExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'CyclingMultiSourcesExamplesIterable'
| Either keep only the requested shard, or propagate the request to the underlying iterable.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'CyclingMultiSourcesExamplesIterable'
| Shuffle each underlying examples iterable.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class DataSourcesShufflingDisallowed(builtins.Exception)
| skip() or take() freeze the order of data sources shards
|
| Method resolution order:
| DataSourcesShufflingDisallowed
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DistributedConfig(builtins.object)
| DistributedConfig(rank: int, world_size: int) -> None
|
| DistributedConfig(rank: int, world_size: int)
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, rank: int, world_size: int) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'rank': <class 'int'>, 'world_size': <class 'int'>}
|
| __dataclass_fields__ = {'rank': Field(name='rank',type=<class 'int'>,d...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('rank', 'world_size')
class ExamplesIterable(_BaseExamplesIterable)
| ExamplesIterable(generate_examples_fn: Callable[..., collections.abc.Iterator[tuple[Union[int, str, tuple[int, int], ForwardRef('BuilderKey')], dict]]], kwargs: dict, generate_more_kwargs_fn: Optional[Callable[..., collections.abc.Iterator[dict]]] = None, sleep_on_threads_shutdown: bool = False)
|
| Method resolution order:
| ExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, generate_examples_fn: Callable[..., collections.abc.Iterator[tuple[Union[int, str, tuple[int, int], ForwardRef('BuilderKey')], dict]]], kwargs: dict, generate_more_kwargs_fn: Optional[Callable[..., collections.abc.Iterator[dict]]] = None, sleep_on_threads_shutdown: bool = False)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'ExamplesIterable'
| Split shars into more shards if possible.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'ExamplesIterable'
| Keep only the requested shard.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'ExamplesIterable'
| Either shuffle the shards/sources of the dataset, or propagate the shuffling to the underlying iterable.
| If the order of the shards must stay fixed (when using .skip or .take for example), then this method returns self.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| features
|
| is_typed
|
| iter_arrow
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class FilteredExamplesIterable(MappedExamplesIterable)
| FilteredExamplesIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, function: Callable, with_indices: bool = False, input_columns: Optional[list[str]] = None, batched: bool = False, batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None, formatting: Optional[ForwardRef('FormattingConfig')] = None)
|
| Method resolution order:
| FilteredExamplesIterable
| MappedExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, function: Callable, with_indices: bool = False, input_columns: Optional[list[str]] = None, batched: bool = False, batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None, formatting: Optional[ForwardRef('FormattingConfig')] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| reshard_data_sources(self) -> 'FilteredExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'FilteredExamplesIterable'
| Keep only the requested shard.
|
| shuffle_data_sources(self, seed: Optional[int]) -> 'FilteredExamplesIterable'
| Shuffle the wrapped examples iterable.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| num_shards
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| mask_column_name = '===MASK==='
|
| ----------------------------------------------------------------------
| Methods inherited from MappedExamplesIterable:
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| ----------------------------------------------------------------------
| Readonly properties inherited from MappedExamplesIterable:
|
| features
|
| is_typed
|
| iter_arrow
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class FormattedExamplesIterable(_BaseExamplesIterable)
| FormattedExamplesIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, formatting: Optional[datasets.iterable_dataset.FormattingConfig], features: Optional[datasets.features.features.Features], token_per_repo_id: dict[str, typing.Union[str, bool, NoneType]], force_convert_to_python: bool = False)
|
| Method resolution order:
| FormattedExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, formatting: Optional[datasets.iterable_dataset.FormattingConfig], features: Optional[datasets.features.features.Features], token_per_repo_id: dict[str, typing.Union[str, bool, NoneType]], force_convert_to_python: bool = False)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'FormattedExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'FormattedExamplesIterable'
| Keep only the requested shard.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'FormattedExamplesIterable'
| Shuffle the wrapped examples iterable.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class FormattingConfig(builtins.object)
| FormattingConfig(format_type: Optional[str]) -> None
|
| FormattingConfig(format_type: Optional[str])
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, format_type: Optional[str]) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| is_table
|
| is_tensor
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'format_type': typing.Optional[str]}
|
| __dataclass_fields__ = {'format_type': Field(name='format_type',type=t...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('format_type',)
class HorizontallyConcatenatedMultiSourcesExamplesIterable(_BaseExamplesIterable)
| HorizontallyConcatenatedMultiSourcesExamplesIterable(ex_iterables: list[datasets.iterable_dataset._BaseExamplesIterable])
|
| HorizontallyConcatenatedMultiSourcesExamplesIterable merges examples together for the input list of iterables.
| It also checks that there are no duplicate columns (otherwise we don't know which one to keep).
| This check is done once when yielding the first example.
|
| However it doesn't fill missing columns with None.
| Instead, this is handled by the `IterableDataset` class or `FormattedExamplesIterable`.
|
| For information, `IterableDataset` merges the features of all the datasets to concatenate into one.
| We use `IterableDataset._resolve_features` to obtain the features of all the datasets to concatenate.
|
| Then for each example, `IterableDataset` and `FormattedExamplesIterable` automatically fill missing columns with None.
| This is done with `_apply_feature_types_on_example`.
|
| Method resolution order:
| HorizontallyConcatenatedMultiSourcesExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterables: list[datasets.iterable_dataset._BaseExamplesIterable])
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'HorizontallyConcatenatedMultiSourcesExamplesIterable'
| Doesn't reshard the wrapped examples iterable since it would break the alignment between them.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'HorizontallyConcatenatedMultiSourcesExamplesIterable'
| Doesn't shard the wrapped examples iterable since it would break the alignment between them.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'HorizontallyConcatenatedMultiSourcesExamplesIterable'
| Doesn't shuffle the wrapped examples iterable since it would break the alignment between them.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class IterableColumn(builtins.object)
| IterableColumn(source: Union[ForwardRef('IterableDataset'), ForwardRef('IterableColumn')], column_name: str)
|
| An iterable for a specific column of an [`IterableDataset`].
|
| Example:
|
| Iterate on the texts of the "text" column of a dataset:
|
| ```python
| for text in dataset["text"]:
| ...
| ```
|
| It also works with nested columns:
|
| ```python
| for source in dataset["metadata"]["source"]:
| ...
| ```
|
| Methods defined here:
|
| __getitem__(self, column_name: str) -> 'IterableColumn'
|
| __init__(self, source: Union[ForwardRef('IterableDataset'), ForwardRef('IterableColumn')], column_name: str)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> collections.abc.Iterator[typing.Any]
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class IterableDataset(datasets.arrow_dataset.DatasetInfoMixin)
| IterableDataset(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, formatting: Optional[datasets.iterable_dataset.FormattingConfig] = None, distributed: Optional[datasets.iterable_dataset.DistributedConfig] = None, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
|
| A Dataset backed by an iterable.
|
| Method resolution order:
| IterableDataset
| datasets.arrow_dataset.DatasetInfoMixin
| builtins.object
|
| Methods defined here:
|
| __getitem__(self, column_name: str) -> datasets.iterable_dataset.IterableColumn
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, formatting: Optional[datasets.iterable_dataset.FormattingConfig] = None, distributed: Optional[datasets.iterable_dataset.DistributedConfig] = None, token_per_repo_id: Optional[dict[str, Union[str, bool, NoneType]]] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
|
| __repr__(self)
| Return repr(self).
|
| __setstate__(self, d)
|
| add_column(self, name: str, column: Union[list, <built-in function array>]) -> 'IterableDataset'
| Add column to Dataset.
|
| Args:
| name (str): Column name.
| column (list or np.array): Column data to be added.
|
| Returns:
| `IterableDataset`
|
| batch(self, batch_size: Optional[int] = None, by_column: Union[str, list[str], NoneType] = None, drop_last_batch: bool = False) -> 'IterableDataset'
| Group samples from the dataset into batches.
|
| Args:
| batch_size (`int`, optional):
| The number of samples in each batch.
| by_column (`Union[str, list[str]`, optional):
| The column used to batch examples together.
| Successive examples with the same value for that column are in grouped the same batch.
| This can also be a list of columns if you want to batch by multiple columns.
| If batching by column, the batch_size is only used to control the size of the batches
| to group together or slice during acculumation.
|
| <Added version="4.9.0"/>
| drop_last_batch (`bool`, defaults to `False`):
| Whether to drop the last incomplete batch.
|
| Example:
| ```py
| >>> ds = load_dataset("some_dataset", streaming=True)
| >>> batched_ds = ds.batch(batch_size=32)
| ```
|
| cast(self, features: datasets.features.features.Features) -> 'IterableDataset'
| Cast the dataset to a new set of features.
|
| Args:
| features ([`Features`]):
| New features to cast the dataset to.
| The name of the fields in the features must match the current column names.
| The type of the data must also be convertible from one type to the other.
| For non-trivial conversion, e.g. `string` <-> `ClassLabel` you should use [`~Dataset.map`] to update the Dataset.
|
| Returns:
| `IterableDataset`: A copy of the dataset with casted features.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, ClassLabel, Value
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> ds.features
| {'label': ClassLabel(names=['neg', 'pos']),
| 'text': Value('string')}
| >>> new_features = ds.features.copy()
| >>> new_features["label"] = ClassLabel(names=["bad", "good"])
| >>> new_features["text"] = Value("large_string")
| >>> ds = ds.cast(new_features)
| >>> ds.features
| {'label': ClassLabel(names=['bad', 'good']),
| 'text': Value('large_string')}
| ```
|
| cast_column(self, column: str, feature: Union[dict, list, tuple, datasets.features.features.Value, datasets.features.features.ClassLabel, datasets.features.translation.Translation, datasets.features.translation.TranslationVariableLanguages, datasets.features.features.LargeList, datasets.features.features.List, datasets.features.features.Array2D, datasets.features.features.Array3D, datasets.features.features.Array4D, datasets.features.features.Array5D, datasets.features.audio.Audio, datasets.features.image.Image, datasets.features.mesh.Mesh, datasets.features.video.Video, datasets.features.pdf.Pdf, datasets.features.nifti.Nifti]) -> 'IterableDataset'
| Cast column to feature for decoding.
|
| Args:
| column (`str`):
| Column name.
| feature (`Feature`):
| Target feature.
|
| Returns:
| `IterableDataset`
|
| Example:
|
| ```py
| >>> from datasets import load_dataset, Audio
| >>> ds = load_dataset("PolyAI/minds14", name="en-US", split="train", streaming=True)
| >>> ds.features
| {'audio': Audio(sampling_rate=8000, mono=True, decode=True, id=None),
| 'english_transcription': Value('string'),
| 'intent_class': ClassLabel(num_classes=14, names=['abroad', 'address', 'app_error', 'atm_limit', 'balance', 'business_loan', 'card_issues', 'cash_deposit', 'direct_debit', 'freeze', 'high_value_payment', 'joint_account', 'latest_transactions', 'pay_bill']),
| 'lang_id': ClassLabel(num_classes=14, names=['cs-CZ', 'de-DE', 'en-AU', 'en-GB', 'en-US', 'es-ES', 'fr-FR', 'it-IT', 'ko-KR', 'nl-NL', 'pl-PL', 'pt-PT', 'ru-RU', 'zh-CN']),
| 'path': Value('string'),
| 'transcription': Value('string')}
| >>> ds = ds.cast_column("audio", Audio(sampling_rate=16000))
| >>> ds.features
| {'audio': Audio(sampling_rate=16000, mono=True, decode=True, id=None),
| 'english_transcription': Value('string'),
| 'intent_class': ClassLabel(num_classes=14, names=['abroad', 'address', 'app_error', 'atm_limit', 'balance', 'business_loan', 'card_issues', 'cash_deposit', 'direct_debit', 'freeze', 'high_value_payment', 'joint_account', 'latest_transactions', 'pay_bill']),
| 'lang_id': ClassLabel(num_classes=14, names=['cs-CZ', 'de-DE', 'en-AU', 'en-GB', 'en-US', 'es-ES', 'fr-FR', 'it-IT', 'ko-KR', 'nl-NL', 'pl-PL', 'pt-PT', 'ru-RU', 'zh-CN']),
| 'path': Value('string'),
| 'transcription': Value('string')}
| ```
|
| decode(self, enable: bool = True, num_threads: int = 0) -> 'IterableDataset'
| Enable or disable the dataset features decoding for audio, image, video.
|
| When enabled (default), media types are decoded:
|
| * audio -> dict of "array" and "sampling_rate" and "path"
| * image -> PIL.Image
| * video -> torchcodec.decoders.VideoDecoder
|
| You can enable multithreading using `num_threads`. This is especially useful to speed up remote
| data streaming. However it can be slower than `num_threads=0` for local data on fast disks.
|
| Disabling decoding is useful if you want to iterate on the paths or bytes of the media files
| without actually decoding their content. To disable decoding you can use `.decode(False)`, which
| is equivalent to calling `.cast()` or `.cast_column()` with all the Audio, Image and Video types
| set to `decode=False`.
|
| Args:
| enable (`bool`, defaults to `True`):
| Enable or disable features decoding.
| num_threads (`int`, defaults to `0`):
| Enable multithreading for features decoding.
|
| Returns:
| `IterableDataset`: A copy of the dataset with casted features.
|
| Examples:
|
| Disable decoding:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True)
| >>> next(iter(ds))
| {'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=2048x1024>,
| 'text': 'A distant celestial object with an icy crust, displaying a light blue shade, covered with round pits and rugged terrains.'}
| >>> ds = ds.decode(False)
| >>> ds.features
| {'image': Image(mode=None, decode=False, id=None),
| 'text': Value('string')}
| >>> next(iter(ds))
| {
| 'image': {
| 'path': 'hf://datasets/sshh12/planet-textures@69dc4cef7a5c4b2cfe387727ec8ea73d4bff7302/train/textures/0000.png',
| 'bytes': None
| },
| 'text': 'A distant celestial object with an icy crust, displaying a light blue shade, covered with round pits and rugged terrains.'
| }
| ```
|
| Speed up streaming with multithreading:
|
| ```py
| >>> import os
| >>> from datasets import load_dataset
| >>> from tqdm import tqdm
| >>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True)
| >>> num_threads = min(32, (os.cpu_count() or 1) + 4)
| >>> ds = ds.decode(num_threads=num_threads)
| >>> for _ in tqdm(ds): # 20 times faster !
| ... ...
| ```
|
| filter(self, function: Optional[Callable] = None, with_indices=False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, fn_kwargs: Optional[dict] = None) -> 'IterableDataset'
| Apply a filter function to all the elements so that the dataset only includes examples according to the filter function.
| The filtering is done on-the-fly when iterating over the dataset.
|
| If the function is asynchronous, then `filter` will run your function in parallel, with up to one thousand simulatenous calls (configurable).
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`):
| Callable with one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> bool` if `with_indices=False, batched=False`
| - `function(example: Dict[str, Any], indices: int) -> bool` if `with_indices=True, batched=False`
| - `function(example: Dict[str, List]) -> List[bool]` if `with_indices=False, batched=True`
| - `function(example: Dict[str, List], indices: List[int]) -> List[bool]` if `with_indices=True, batched=True`
|
| If the function is asynchronous, then `filter` will run your function in parallel.
| If no function is provided, defaults to an always True function: `lambda x: True`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx): ...`.
| input_columns (`str` or `List[str]`, *optional*):
| The columns to be passed into `function` as
| positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, default `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| fn_kwargs (`Dict`, *optional*, default `None`):
| Keyword arguments to be passed to `function`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> ds = ds.filter(lambda x: x["label"] == 0)
| >>> list(ds.take(3))
| [{'label': 0, 'movie_review': 'simplistic , silly and tedious .'},
| {'label': 0,
| 'movie_review': "it's so laddish and juvenile , only teenage boys could possibly find it funny ."},
| {'label': 0,
| 'movie_review': 'exploitative and largely devoid of the depth or sophistication that would make watching such a graphic treatment of the crimes bearable .'}]
| ```
|
| iter(self, batch_size: int, drop_last_batch: bool = False)
| Iterate through the batches of size `batch_size`.
|
| Args:
| batch_size (:obj:`int`): size of each batch to yield.
| drop_last_batch (:obj:`bool`, default `False`): Whether a last batch smaller than the batch_size should be
| dropped
|
| load_state_dict(self, state_dict: dict) -> None
| Load the state_dict of the dataset.
| The iteration will restart at the next example from when the state was saved.
|
| Resuming returns exactly where the checkpoint was saved except in two cases:
|
| 1. examples from shuffle buffers are lost when resuming and the buffers are refilled with new data
| 2. combinations of `.with_format(arrow)` and batched `.map()` may skip one batch.
|
| Example:
|
| ```py
| >>> from datasets import Dataset, concatenate_datasets
| >>> ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3)
| >>> for idx, example in enumerate(ds):
| ... print(example)
| ... if idx == 2:
| ... state_dict = ds.state_dict()
| ... print("checkpoint")
| ... break
| >>> ds.load_state_dict(state_dict)
| >>> print(f"restart from checkpoint")
| >>> for example in ds:
| ... print(example)
| ```
|
| which returns:
| ```
| {'a': 0}
| {'a': 1}
| {'a': 2}
| checkpoint
| restart from checkpoint
| {'a': 3}
| {'a': 4}
| {'a': 5}
| ```
|
| ```py
| >>> from torchdata.stateful_dataloader import StatefulDataLoader
| >>> ds = load_dataset("deepmind/code_contests", streaming=True, split="train")
| >>> dataloader = StatefulDataLoader(ds, batch_size=32, num_workers=4)
| >>> # checkpoint
| >>> state_dict = dataloader.state_dict() # uses ds.state_dict() under the hood
| >>> # resume from checkpoint
| >>> dataloader.load_state_dict(state_dict) # uses ds.load_state_dict() under the hood
| ```
|
| map(self, function: Optional[Callable] = None, with_indices: bool = False, input_columns: Union[str, list[str], NoneType] = None, batched: bool = False, batch_size: Optional[int] = 1000, drop_last_batch: bool = False, remove_columns: Union[str, list[str], NoneType] = None, features: Optional[datasets.features.features.Features] = None, fn_kwargs: Optional[dict] = None) -> 'IterableDataset'
| Apply a function to all the examples in the iterable dataset (individually or in batches) and update them.
| If your function returns a column that already exists, then it overwrites it.
| The function is applied on-the-fly on the examples when iterating over the dataset.
|
| You can specify whether the function should be batched or not with the `batched` parameter:
|
| - If batched is `False`, then the function takes 1 example in and should return 1 example.
| An example is a dictionary, e.g. `{"text": "Hello there !"}`.
| - If batched is `True` and `batch_size` is 1, then the function takes a batch of 1 example as input and can return a batch with 1 or more examples.
| A batch is a dictionary, e.g. a batch of 1 example is {"text": ["Hello there !"]}.
| - If batched is `True` and `batch_size` is `n` > 1, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples.
| Note that the last batch may have less than `n` examples.
| A batch is a dictionary, e.g. a batch of `n` examples is `{"text": ["Hello there !"] * n}`.
|
| If the function is asynchronous, then `map` will run your function in parallel, with up to one thousand simulatenous calls.
| It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time.
|
| Args:
| function (`Callable`, *optional*, defaults to `None`):
| Function applied on-the-fly on the examples when you iterate on the dataset.
| It must have one of the following signatures:
|
| - `function(example: Dict[str, Any]) -> Dict[str, Any]` if `batched=False` and `with_indices=False`
| - `function(example: Dict[str, Any], idx: int) -> Dict[str, Any]` if `batched=False` and `with_indices=True`
| - `function(batch: Dict[str, List]) -> Dict[str, List]` if `batched=True` and `with_indices=False`
| - `function(batch: Dict[str, List], indices: List[int]) -> Dict[str, List]` if `batched=True` and `with_indices=True`
|
| For advanced usage, the function can also return a `pyarrow.Table`.
| If the function is asynchronous, then `map` will run your function in parallel.
| Moreover if your function returns nothing (`None`), then `map` will run your function and return the dataset unchanged.
| If no function is provided, default to identity function: `lambda x: x`.
| with_indices (`bool`, defaults to `False`):
| Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx[, rank]): ...`.
| input_columns (`Optional[Union[str, List[str]]]`, defaults to `None`):
| The columns to be passed into `function`
| as positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument.
| batched (`bool`, defaults to `False`):
| Provide batch of examples to `function`.
| batch_size (`int`, *optional*, defaults to `1000`):
| Number of examples per batch provided to `function` if `batched=True`.
| `batch_size <= 0` or `batch_size == None` then provide the full dataset as a single batch to `function`.
| drop_last_batch (`bool`, defaults to `False`):
| Whether a last batch smaller than the batch_size should be
| dropped instead of being processed by the function.
| remove_columns (`[List[str]]`, *optional*, defaults to `None`):
| Remove a selection of columns while doing the mapping.
| Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding
| columns with names in `remove_columns`, these columns will be kept.
| features (`[Features]`, *optional*, defaults to `None`):
| Feature types of the resulting dataset.
| fn_kwargs (`Dict`, *optional*, default `None`):
| Keyword arguments to be passed to `function`.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> def add_prefix(example):
| ... example["text"] = "Review: " + example["text"]
| ... return example
| >>> ds = ds.map(add_prefix)
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'Review: the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'Review: the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'Review: effective but too-tepid biopic'}]
| ```
|
| push_to_hub(self, repo_id: str, config_name: str = 'default', set_default: Optional[bool] = None, split: Optional[str] = None, data_dir: Optional[str] = None, commit_message: Optional[str] = None, commit_description: Optional[str] = None, private: Optional[bool] = None, token: Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = False, max_shard_size: Union[str, int, NoneType] = None, num_shards: Optional[int] = None, embed_external_files: bool = True, num_proc: Optional[int] = None) -> huggingface_hub.hf_api.CommitInfo
| Pushes the dataset to the hub as a Parquet dataset.
| The dataset is pushed using HTTP requests and does not need to have neither git or git-lfs installed.
|
| The resulting Parquet files are self-contained by default. If your dataset contains [`Image`], [`Audio`] or [`Video`]
| data, the Parquet files will store the bytes of your images or audio files.
| You can disable this by setting `embed_external_files` to `False`.
|
| Args:
| repo_id (`str`):
| The ID of the repository to push to in the following format: `<user>/<dataset_name>` or
| `<org>/<dataset_name>`. Also accepts `<dataset_name>`, which will default to the namespace
| of the logged-in user.
|
| It could also be a location inside a bucket, e.g. `buckets/<user_or_org>/<bucket_name>/...`
| config_name (`str`, defaults to "default"):
| The configuration name (or subset) of a dataset. Defaults to "default".
| set_default (`bool`, *optional*):
| Whether to set this configuration as the default one. Otherwise, the default configuration is the one
| named "default".
| split (`str`, *optional*):
| The name of the split that will be given to that dataset. Defaults to `self.split`.
| data_dir (`str`, *optional*):
| Directory name that will contain the uploaded data files. Defaults to the `config_name` if different
| from "default", else "data".
| commit_message (`str`, *optional*):
| Message to commit while pushing. Will default to `"Upload dataset"`.
| commit_description (`str`, *optional*):
| Description of the commit that will be created.
| Additionally, description of the PR if a PR is created (`create_pr` is True).
| private (`bool`, *optional*):
| Whether to make the repo private. If `None` (default), the repo will be public unless the
| organization's default is private. This value is ignored if the repo already exists.
| token (`str`, *optional*):
| An optional authentication token for the Hugging Face Hub. If no token is passed, will default
| to the token saved locally when logging in with `huggingface-cli login`. Will raise an error
| if no token is passed and the user is not logged-in.
| revision (`str`, *optional*):
| Branch to push the uploaded files to. Defaults to the `"main"` branch.
| create_pr (`bool`, *optional*, defaults to `False`):
| Whether to create a PR with the uploaded files or directly commit.
| max_shard_size (`int` or `str`, *optional*):
| Optional maximum size of the dataset shards to be uploaded to the hub. If expressed as a string, needs to be digits followed
| by a unit (like `"5MB"`). If not provided, shard count defaults to this dataset's `.num_shards`.
| num_shards (`int`, *optional*):
| Number of shards to write. If `max_shard_size` is provided and `num_shards` is not, then the number of shards is estimated
| from `max_shard_size`.
| embed_external_files (`bool`, defaults to `True`):
| Whether to embed file bytes in the shards.
| In particular, this will do the following before the push for the fields of type:
|
| - [`Audio`] and [`Image`]: remove local path information and embed file content in the Parquet files.
| num_proc (`int`, *optional*, defaults to `None`):
| Number of processes when preparing and uploading the dataset.
| This is helpful if the dataset is made of many samples and transformations.
| I uses "spawn" context to work with hf_xet, the rust client for fast uploads to HF.
| Multiprocessing is disabled by default.
|
| Return:
| huggingface_hub.CommitInfo
|
| Example:
|
| ```python
| >>> dataset.push_to_hub("<organization>/<dataset_id>")
| >>> dataset_dict.push_to_hub("<organization>/<dataset_id>", private=True)
| >>> dataset.push_to_hub("<organization>/<dataset_id>", max_shard_size="1GB")
| >>> dataset.push_to_hub("<organization>/<dataset_id>", num_shards=1024)
| ```
|
| If your dataset has multiple splits (e.g. train/validation/test):
|
| ```python
| >>> train_dataset.push_to_hub("<organization>/<dataset_id>", split="train")
| >>> val_dataset.push_to_hub("<organization>/<dataset_id>", split="validation")
| >>> # later
| >>> dataset = load_dataset("<organization>/<dataset_id>")
| >>> train_dataset = dataset["train"]
| >>> val_dataset = dataset["validation"]
| ```
|
| If you want to add a new configuration (or subset) to a dataset (e.g. if the dataset has multiple tasks/versions/languages):
|
| ```python
| >>> english_dataset.push_to_hub("<organization>/<dataset_id>", "en")
| >>> french_dataset.push_to_hub("<organization>/<dataset_id>", "fr")
| >>> # later
| >>> english_dataset = load_dataset("<organization>/<dataset_id>", "en")
| >>> french_dataset = load_dataset("<organization>/<dataset_id>", "fr")
| ```
|
| remove_columns(self, column_names: Union[str, list[str]]) -> 'IterableDataset'
| Remove one or several column(s) in the dataset and the features associated to them.
| The removal is done on-the-fly on the examples when iterating over the dataset.
|
|
| Args:
| column_names (`Union[str, List[str]]`):
| Name of the column(s) to remove.
|
| Returns:
| `IterableDataset`: A copy of the dataset object without the columns to remove.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> next(iter(ds))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .', 'label': 1}
| >>> ds = ds.remove_columns("label")
| >>> next(iter(ds))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| rename_column(self, original_column_name: str, new_column_name: str) -> 'IterableDataset'
| Rename a column in the dataset, and move the features associated to the original column under the new column
| name.
|
| Args:
| original_column_name (`str`):
| Name of the column to rename.
| new_column_name (`str`):
| New name for the column.
|
| Returns:
| `IterableDataset`: A copy of the dataset with a renamed column.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> next(iter(ds))
| {'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| >>> ds = ds.rename_column("text", "movie_review")
| >>> next(iter(ds))
| {'label': 1,
| 'movie_review': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| rename_columns(self, column_mapping: dict[str, str]) -> 'IterableDataset'
| Rename several columns in the dataset, and move the features associated to the original columns under
| the new column names.
|
| Args:
| column_mapping (`Dict[str, str]`): A mapping of columns to rename to their new names
|
| Returns:
| `IterableDataset`: A copy of the dataset with renamed columns
|
| repeat(self, num_times: Optional[int]) -> 'IterableDataset'
| Create a new [`IterableDataset`] that repeats the underlying dataset `num_times` times.
|
| N.B. The effect of calling shuffle after repeat depends significantly on buffer size.
| With buffer_size 1, duplicate data is never seen in the same iteration, even after shuffling:
| ds.repeat(n).shuffle(seed=42, buffer_size=1) is equivalent to ds.shuffle(seed=42, buffer_size=1).repeat(n),
| and only shuffles shard orders within each iteration.
| With buffer size >= (num samples in the dataset * num_times), we get full shuffling of the repeated data, i.e. we can observe duplicates in
| the same iteration.
|
| Args:
| num_times (`int`) or (`None`):
| Number of times to repeat the dataset. If `None`, the dataset will be repeated indefinitely.
|
| Example:
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
| >>> ds = ds.take(2).repeat(2)
| >>> list(ds)
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'},
| {'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| ```
|
| reshard(self) -> 'IterableDataset'
| Reshard the dataset if possible, i.e. split the current shards further into more shards.
| This increases the number of shards and the resulting dataset has num_shards >= previous_num_shards.
| Equality may happen if no shard can be split further.
|
| The resharding mechanism depends on the dataset file format:
|
| * Parquet: shard per row group instead of per file
| * Other: not implemented yet (contributions are welcome !)
|
| Be sure to reshard/shard before using any randomizing operator (such as `shuffle`).
| It is best if the shard operator is used early in the dataset pipeline.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("fancyzhx/amazon_polarity", split="train", streaming=True)
| >>> ds
| IterableDataset({
| features: ['label', 'title', 'content'],
| num_shards: 4
| })
| >>> ds.reshard()
| IterableDataset({
| features: ['label', 'title', 'content'],
| num_shards: 3600
| })
| ```
|
| select_columns(self, column_names: Union[str, list[str]]) -> 'IterableDataset'
| Select one or several column(s) in the dataset and the features
| associated to them. The selection is done on-the-fly on the examples
| when iterating over the dataset.
|
|
| Args:
| column_names (`Union[str, List[str]]`):
| Name of the column(s) to select.
|
| Returns:
| `IterableDataset`: A copy of the dataset object with selected columns.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> next(iter(ds))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .', 'label': 1}
| >>> ds = ds.select_columns("text")
| >>> next(iter(ds))
| {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}
| ```
|
| set_epoch(self, epoch: int)
|
| shard(self, num_shards: int, index: int, contiguous: bool = True) -> 'IterableDataset'
| Return the `index`-nth shard from dataset split into `num_shards` pieces.
|
| This shards deterministically. `dataset.shard(n, i)` splits the dataset into contiguous chunks,
| so it can be easily concatenated back together after processing. If `dataset.num_shards % n == l`, then the
| first `l` datasets each have `(dataset.num_shards // n) + 1` shards, and the remaining datasets have `(dataset.num_shards // n)` shards.
| `datasets.concatenate_datasets([dset.shard(n, i) for i in range(n)])` returns a dataset with the same order as the original.
| In particular, `dataset.shard(dataset.num_shards, i)` returns a dataset with 1 shard.
|
| Note: n should be less or equal to the number of shards in the dataset `dataset.num_shards`.
|
| On the other hand, `dataset.shard(n, i, contiguous=False)` contains all the shards of the dataset whose index mod `n = i`.
|
| Be sure to shard before using any randomizing operator (such as `shuffle`).
| It is best if the shard operator is used early in the dataset pipeline.
|
| Args:
| num_shards (`int`):
| How many shards to split the dataset into.
| index (`int`):
| Which shard to select and return.
| contiguous: (`bool`, defaults to `True`):
| Whether to select contiguous blocks of indices for shards.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("fancyzhx/amazon_polarity", split="train", streaming=True)
| >>> ds
| IterableDataset({
| features: ['label', 'title', 'content'],
| num_shards: 4
| })
| >>> ds.shard(num_shards=2, index=0)
| IterableDataset({
| features: ['label', 'title', 'content'],
| num_shards: 2
| })
| ```
|
| shuffle(self, seed=None, generator: Optional[numpy.random._generator.Generator] = None, buffer_size: int = 1000, max_buffer_input_shards: int = 10) -> 'IterableDataset'
| Randomly shuffles the elements of this dataset.
|
| This dataset fills a buffer with `buffer_size` elements, then randomly samples elements from this buffer,
| replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or
| equal to the full size of the dataset is required.
|
| For instance, if your dataset contains 10,000 elements but `buffer_size` is set to 1000, then `shuffle` will
| initially select a random element from only the first 1000 elements in the buffer. Once an element is
| selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element,
| maintaining the 1000 element buffer.
|
| If the dataset is made of several shards, it fills the buffer using up to `max_buffer_input_shards` shards
| at a time and also does shuffle the order of the shards. This greatly improves the quality of the shuffling.
|
| However if the order has been fixed by using [`~datasets.IterableDataset.skip`]
| or [`~datasets.IterableDataset.take`] then the order of the shards is kept unchanged and only one shard at
| a time is used to fill the buffer.
|
| Args:
| seed (`int`, *optional*, defaults to `None`):
| Random seed that will be used to shuffle the dataset.
| It is used to sample from the shuffle buffer and also to shuffle the data shards.
| generator (`numpy.random.Generator`, *optional*):
| Numpy random Generator to use to compute the permutation of the dataset rows.
| If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy).
| buffer_size (`int`, defaults to `1000`):
| Size of the buffer.
| max_buffer_input_shards (`int`, defaults to `101000`):
| Maximum number of shards to use to feed the buffer at a time.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> list(ds.take(3))
| [{'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',
| 'label': 1},
| {'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .',
| 'label': 1},
| {'text': 'effective but too-tepid biopic', 'label': 1}]
| >>> shuffled_ds = ds.shuffle(seed=42)
| >>> list(shuffled_ds.take(3))
| [{'text': "a sports movie with action that's exciting on the field and a story you care about off it .",
| 'label': 1},
| {'text': 'at its best , the good girl is a refreshingly adult take on adultery . . .',
| 'label': 1},
| {'text': "sam jones became a very lucky filmmaker the day wilco got dropped from their record label , proving that one man's ruin may be another's fortune .",
| 'label': 1}]
| >>> resharded_ds = ds.reshard() # useful to shard Parquet datasets by row group instead of by file, improving shuffling quality on dataset with one or few files.
| >>> shuffled_resharded_ds = resharded_ds.shuffle(seed=42)
| >>> list(shuffled_resharded_ds.take(3))
| [{'text': 'this mistaken-identity picture is so film-culture referential that the final product is a ghost .',
| 'label': 0},
| {'text': 'woody allen used to ridicule movies like hollywood ending . now he makes them .',
| 'label': 0},
| {'text': "not only is undercover brother as funny , if not more so , than both austin powers films , but it's also one of the smarter , savvier spoofs to come along in some time .",
| 'label': 1}]
| ```
|
| skip(self, n: int) -> 'IterableDataset'
| Create a new [`IterableDataset`] that skips the first `n` elements.
|
| Args:
| n (`int`):
| Number of elements to skip.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'}]
| >>> ds = ds.skip(1)
| >>> list(ds.take(3))
| [{'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'},
| {'label': 1, 'text': 'effective but too-tepid biopic'},
| {'label': 1,
| 'text': 'if you sometimes like to go to the movies to have fun , wasabi is a good place to start .'}]
| ```
|
| state_dict(self) -> dict
| Get the current state_dict of the dataset.
| It corresponds to the state at the latest example it yielded.
|
| Resuming returns exactly where the checkpoint was saved except in two cases:
|
| 1. examples from shuffle buffers are lost when resuming and the buffers are refilled with new data
| 2. combinations of `.with_format(arrow)` and batched `.map()` may skip one batch.
|
| Returns:
| `dict`
|
| Example:
|
| ```py
| >>> from datasets import Dataset, concatenate_datasets
| >>> ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3)
| >>> for idx, example in enumerate(ds):
| ... print(example)
| ... if idx == 2:
| ... state_dict = ds.state_dict()
| ... print("checkpoint")
| ... break
| >>> ds.load_state_dict(state_dict)
| >>> print(f"restart from checkpoint")
| >>> for example in ds:
| ... print(example)
| ```
|
| which returns:
| ```
| {'a': 0}
| {'a': 1}
| {'a': 2}
| checkpoint
| restart from checkpoint
| {'a': 3}
| {'a': 4}
| {'a': 5}
| ```
|
| ```py
| >>> from torchdata.stateful_dataloader import StatefulDataLoader
| >>> ds = load_dataset("deepmind/code_contests", streaming=True, split="train")
| >>> dataloader = StatefulDataLoader(ds, batch_size=32, num_workers=4)
| >>> # checkpoint
| >>> state_dict = dataloader.state_dict() # uses ds.state_dict() under the hood
| >>> # resume from checkpoint
| >>> dataloader.load_state_dict(state_dict) # uses ds.load_state_dict() under the hood
| ```
|
| take(self, n: int) -> 'IterableDataset'
| Create a new [`IterableDataset`] with only the first `n` elements.
|
| Args:
| n (`int`):
| Number of elements to take.
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)
| >>> small_ds = ds.take(2)
| >>> list(small_ds)
| [{'label': 1,
| 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'},
| {'label': 1,
| 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}]
| ```
|
| to_csv(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, storage_options: Optional[dict] = None, **to_csv_kwargs) -> int
| Exports the dataset to csv.
|
| This iterates on the dataset and loads it completely in memory before writing it.
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.csv`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.csv`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
| **to_csv_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_csv`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html).
| The parameter `index` defaults to `False` if not specified.
| If you would like to write the index, pass `index=True` and also set a name for the index column by
| passing `index_label`.
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_csv("path/to/dataset/directory")
| ```
|
| to_dict(self, batch_size: Optional[int] = None, batched: bool = False) -> Union[dict, collections.abc.Iterator[dict]]
| Returns the dataset as a Python dict. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*): The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
|
| Returns:
| `dict` or `Iterator[dict]`
|
| Example:
|
| ```py
| >>> ds.to_dict()
| ```
|
| to_json(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, storage_options: Optional[dict] = None, **to_json_kwargs) -> int
| Export the dataset to JSON Lines or JSON.
|
| This iterates on the dataset and loads it completely in memory before writing it.
|
| The default output format is [JSON Lines](https://jsonlines.org/).
| To export to [JSON](https://www.json.org), pass `lines=False` argument and the desired `orient`.
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.json`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.json`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
| **to_json_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_json`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html).
| Default arguments are `lines=True` and `orient="records".
| The parameter `index` defaults to `False` if `orient` is `"split"` or `"table"`.
| If you would like to write the index, pass `index=True`.
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_json("path/to/dataset/directory/filename.jsonl")
| ```
|
| ```py
| >>> num_shards = dataset.num_shards
| >>> for index in range(num_shards):
| ... shard = dataset.shard(index, num_shards)
| ... shard.to_json(f"path/of/my/dataset/data-{index:05d}.jsonl")
| ```
|
| to_list(self) -> list
| Returns the dataset as a Python list.
|
| Returns:
| `list`
|
| Example:
|
| ```py
| >>> ds.to_list()
| ```
|
| to_pandas(self, batch_size: Optional[int] = None, batched: bool = False) -> Union[pandas.core.frame.DataFrame, collections.abc.Iterator[pandas.core.frame.DataFrame]]
| Returns the dataset as a `pandas.DataFrame`. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*):
| The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
|
| Returns:
| `pandas.DataFrame` or `Iterator[pandas.DataFrame]`
|
| Example:
|
| ```py
| >>> ds.to_pandas()
| ```
|
| to_parquet(self, path_or_buf: Union[str, bytes, os.PathLike, BinaryIO], batch_size: Optional[int] = None, storage_options: Optional[dict] = None, **parquet_writer_kwargs) -> int
| Exports the dataset to parquet
|
| Args:
| path_or_buf (`PathLike` or `FileOrBuffer`):
| Either a path to a file (e.g. `file.parquet`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.parquet`),
| or a BinaryIO, where the dataset will be saved to in the specified format.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.19.0"/>
| **parquet_writer_kwargs (additional keyword arguments):
| Parameters to pass to PyArrow's `pyarrow.parquet.ParquetWriter`.
|
| Returns:
| `int`: The number of characters or bytes written.
|
| Example:
|
| ```py
| >>> ds.to_parquet("path/to/dataset/directory")
| ```
|
| ```py
| >>> num_shards = dataset.num_shards
| >>> for index in range(num_shards):
| ... shard = dataset.shard(index, num_shards)
| ... shard.to_parquet(f"path/of/my/dataset/data-{index:05d}.parquet")
| ```
|
| to_polars(self, batch_size: Optional[int] = None, batched: bool = False, schema_overrides: Optional[dict] = None, rechunk: bool = True) -> Union[ForwardRef('pl.DataFrame'), collections.abc.Iterator['pl.DataFrame']]
| Returns the dataset as a `polars.DataFrame`. Can also return a generator for large datasets.
|
| Args:
| batch_size (`int`, *optional*):
| The size (number of rows) of the batches if `batched` is `True`.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| batched (`bool`):
| Set to `True` to return a generator that yields the dataset as batches
| of `batch_size` rows. Defaults to `False` (returns the whole datasets once).
| schema_overrides (`dict`, *optional*):
| Support type specification or override of one or more columns; note that
| any dtypes inferred from the schema param will be overridden.
| rechunk (`bool`):
| Make sure that all data is in contiguous memory. Defaults to `True`.
| Returns:
| `polars.DataFrame` or `Iterator[polars.DataFrame]`
|
| Example:
|
| ```py
| >>> ds.to_polars()
| ```
|
| to_sql(self, name: str, con: Union[str, ForwardRef('sqlalchemy.engine.Connection'), ForwardRef('sqlalchemy.engine.Engine'), ForwardRef('sqlite3.Connection')], batch_size: Optional[int] = None, **sql_writer_kwargs) -> int
| Exports the dataset to a SQL database.
|
| Args:
| name (`str`):
| Name of SQL table.
| con (`str` or `sqlite3.Connection` or `sqlalchemy.engine.Connection` or `sqlalchemy.engine.Connection`):
| A [URI string](https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls) or a SQLite3/SQLAlchemy connection object used to write to a database.
| batch_size (`int`, *optional*):
| Size of the batch to load in memory and write at once.
| Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`.
| **sql_writer_kwargs (additional keyword arguments):
| Parameters to pass to pandas's [`pandas.DataFrame.to_sql`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_sql.html).
| The parameter `index` defaults to `False` if not specified.
| If you would like to write the index, pass `index=True` and also set a name for the index column by
| passing `index_label`.
|
|
| Returns:
| `int`: The number of records written.
|
| Example:
|
| ```py
| >>> # con provided as a connection URI string
| >>> ds.to_sql("data", "sqlite:///my_own_db.sql")
| >>> # con provided as a sqlite3 connection object
| >>> import sqlite3
| >>> con = sqlite3.connect("my_own_db.sql")
| >>> with con:
| ... ds.to_sql("data", con)
| ```
|
| with_format(self, type: Optional[str] = None) -> 'IterableDataset'
| Return a dataset with the specified format.
|
| Args:
|
| type (`str`, *optional*):
| Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`.
| `None` means it returns python objects (default).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> from transformers import AutoTokenizer
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation", streaming=True)
| >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
| >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)
| >>> ds = ds.with_format("torch")
| >>> next(iter(ds))
| {'text': 'compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
| 'label': tensor(1),
| 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617,
| 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105,
| 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0]),
| 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
| 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
| 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
| 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])}
| ```
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_dict(mapping: dict, features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, num_shards: Optional[int] = 1) -> 'IterableDataset'
| Convert `dict` to a `pyarrow.Table` to create an [`IterableDataset`].
|
| Important: a dataset created with from_dict() lives in memory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. to_parquet / from_parquet.
|
| Args:
| mapping (`Mapping`):
| Mapping of strings to Arrays or Python lists.
| features ([`Features`], *optional*):
| Dataset features.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| num_shards (`int`, default to `1`):
| Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly,
| and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example.
|
| Returns:
| [`IterableDataset`]
|
| from_list(mapping: list[dict], features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, num_shards: Optional[int] = 1) -> 'IterableDataset'
| Convert a list of dicts to a `pyarrow.Table` to create an [`IterableDataset`]`.
|
| Note that the keys of the first entry will be used to determine the dataset columns,
| regardless of what is passed to features.
|
| Important: a dataset created with from_list() lives in memory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it back on disk
| and reload using e.g. from_parquet / to_parquet.
|
| Args:
| mapping (`List[dict]`): A list of mappings of strings to row values.
| features (`Features`, optional): Dataset features.
| info (`DatasetInfo`, optional): Dataset information, like description, citation, etc.
| split (`NamedSplit`, optional): Name of the dataset split.
| num_shards (`int`, default to `1`):
| Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly,
| and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example.
|
| Returns:
| [`IterableDataset`]
|
| from_pandas(df: pandas.core.frame.DataFrame, features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None, preserve_index: Optional[bool] = None, num_shards: Optional[int] = 1) -> 'IterableDataset'
| Convert `pandas.DataFrame` to a `pyarrow.Table` to create an [`IterableDataset`].
|
| The column types in the resulting Arrow Table are inferred from the dtypes of the `pandas.Series` in the
| DataFrame. In the case of non-object Series, the NumPy dtype is translated to its Arrow equivalent. In the
| case of `object`, we need to guess the datatype by looking at the Python objects in this Series.
|
| Be aware that Series of the `object` dtype don't carry enough information to always lead to a meaningful Arrow
| type. In the case that we cannot infer a type, e.g. because the DataFrame is of length 0 or the Series only
| contains `None/nan` objects, the type is set to `null`. This behavior can be avoided by constructing explicit
| features and passing it to this function.
|
| Important: a dataset created with from_pandas() lives in memory.
| This may change in the future, but in the meantime if you
| want to reduce memory usage you should write it on disk
| and reload using e.g. to_parquet / from_parquet.
|
| Args:
| df (`pandas.DataFrame`):
| Dataframe that contains the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| info (`DatasetInfo`, *optional*):
| Dataset information, like description, citation, etc.
| split (`NamedSplit`, *optional*):
| Name of the dataset split.
| preserve_index (`bool`, *optional*):
| Whether to store the index as an additional column in the resulting Dataset.
| The default of `None` will store the index as a column, except for `RangeIndex` which is stored as metadata only.
| Use `preserve_index=True` to force it to be stored as a column.
| num_shards (`int`, default to `1`):
| Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly,
| and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example.
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_pandas(df)
| ```
|
| from_polars(df: Union[ForwardRef('pl.DataFrame'), ForwardRef('pl.LazyFrame')], features: Optional[datasets.features.features.Features] = None, info: Optional[datasets.info.DatasetInfo] = None, split: Optional[datasets.splits.NamedSplit] = None) -> 'IterableDataset'
| Create an IterableDataset from a polars DataFrame or LazyFrame.
|
| Iterating over the dataset is mostly zero copy.
| Under the hood, the dataset iterates over the polars DataFrame batches/slices.
|
| Data types that do copy:
| * CategoricalType
|
| Args:
| df (`polars.DataFrame`): DataFrame to convert to Arrow Table
| features (`Features`, optional): Dataset features.
| info (`DatasetInfo`, optional): Dataset information, like description, citation, etc.
| split (`NamedSplit`, optional): Name of the dataset split.
|
| Returns:
| [`IterableDataset`]
|
| Examples:
| ```py
| >>> ds = IterableDataset.from_polars(df)
| ```
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| from_csv(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, **kwargs) -> 'IterableDataset'
| Create an IterableDataset from CSV file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the CSV file(s).
| split ([`NamedSplit`], *optional*):
| Split name to be assigned to the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`pandas.read_csv`].
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_csv('path/to/dataset.csv')
| ```
|
| from_file(filename: str) -> 'IterableDataset'
| Instantiate a IterableDataset from Arrow table at filename.
|
| Args:
| filename (`str`):
| File name of the dataset.
|
| Returns:
| [`IterableDataset`]
|
| from_generator(generator: Callable, features: Optional[datasets.features.features.Features] = None, gen_kwargs: Optional[dict] = None, split: datasets.splits.NamedSplit = NamedSplit('train')) -> 'IterableDataset'
| Create an Iterable Dataset from a generator.
|
| Args:
| generator (`Callable`):
| A generator function that `yields` examples.
| features (`Features`, *optional*):
| Dataset features.
| gen_kwargs(`dict`, *optional*):
| Keyword arguments to be passed to the `generator` callable.
| You can define a sharded iterable dataset by passing the list of shards in `gen_kwargs`.
| This can be used to improve shuffling and when iterating over the dataset with multiple workers.
| split ([`NamedSplit`], defaults to `Split.TRAIN`):
| Split name to be assigned to the dataset.
|
| <Added version="2.21.0"/>
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> def gen():
| ... yield {"text": "Good", "label": 0}
| ... yield {"text": "Bad", "label": 1}
| ...
| >>> ds = IterableDataset.from_generator(gen)
| ```
|
| ```py
| >>> def gen(shards):
| ... for shard in shards:
| ... with open(shard) as f:
| ... for line in f:
| ... yield {"line": line}
| ...
| >>> shards = [f"data{i}.txt" for i in range(32)]
| >>> ds = IterableDataset.from_generator(gen, gen_kwargs={"shards": shards})
| >>> ds = ds.shuffle(seed=42, buffer_size=10_000) # shuffles the shards order + uses a shuffle buffer
| >>> from torch.utils.data import DataLoader
| >>> dataloader = DataLoader(ds.with_format("torch"), num_workers=4) # give each worker a subset of 32/4=8 shards
| ```
|
| from_json(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, field: Optional[str] = None, **kwargs) -> 'IterableDataset'
| Create an IterableDataset from JSON or JSON Lines file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the JSON or JSON Lines file(s).
| split ([`NamedSplit`], *optional*):
| Split name to be assigned to the dataset.
| features ([`Features`], *optional*):
| Dataset features.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| field (`str`, *optional*):
| Field name of the JSON file where the dataset is contained in.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`JsonConfig`].
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_json('path/to/dataset.json')
| ```
|
| from_parquet(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, columns: Optional[list[str]] = None, filters: Union[pyarrow._compute.Expression, list[tuple], list[list[tuple]], NoneType] = None, fragment_scan_options: Optional[pyarrow._dataset_parquet.ParquetFragmentScanOptions] = None, on_bad_files: Literal['error', 'warn', 'skip'] = 'error', **kwargs) -> 'IterableDataset'
| Create an IterableDataset from Parquet file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the Parquet file(s).
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| columns (`List[str]`, *optional*):
| If not `None`, only these columns will be read from the file.
| A column name may be a prefix of a nested field, e.g. 'a' will select
| 'a.b', 'a.c', and 'a.d.e'.
| filters (`Union[pyarrow.dataset.Expression, list[tuple], list[list[tuple]]]`, *optional*):
| Return only the rows matching the filter.
| If possible the predicate will be pushed down to exploit the partition information
| or internal metadata found in the data source, e.g. Parquet statistics.
| Otherwise filters the loaded RecordBatches before yielding them.
| fragment_scan_options (`pyarrow.dataset.ParquetFragmentScanOptions`, *optional*)
| Scan-specific options for Parquet fragments.
| This is especially useful to configure buffering and caching.
|
| <Added version="4.2.0"/>
| on_bad_files (`Literal["error", "warn", "skip"]`, *optional*, defaults to "error")
| Specify what to do upon encountering a bad file (a file that can't be read). Allowed values are :
| * 'error', raise an Exception when a bad file is encountered.
| * 'warn', raise a warning when a bad file is encountered and skip that file.
| * 'skip', skip bad files without raising or warning when they are encountered.
|
| <Added version="4.2.0"/>
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`ParquetConfig`].
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_parquet('path/to/dataset.parquet')
| ```
|
| Load a subset of columns:
|
| ```python
| >>> ds = IterableDataset.from_parquet('path/to/dataset.parquet', columns=["col_0", "col_1"])
| ```
|
| Efficiently filter data, possibly skipping entire files or row groups:
|
| ```python
| >>> filters = [("col_0", "==", 0)]
| >>> ds = IterableDataset.from_parquet(parquet_files_list, filters=filters)
| ```
|
| from_spark(df: 'pyspark.sql.DataFrame', split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, **kwargs) -> 'IterableDataset'
| Create an IterableDataset from Spark DataFrame. The dataset is streamed to the driver in batches.
|
| Args:
| df (`pyspark.sql.DataFrame`):
| The DataFrame containing the desired data.
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> df = spark.createDataFrame(
| >>> data=[[1, "Elia"], [2, "Teo"], [3, "Fang"]],
| >>> columns=["id", "name"],
| >>> )
| >>> ds = IterableDataset.from_spark(df)
| ```
|
| from_text(path_or_paths: Union[str, bytes, os.PathLike, list[Union[str, bytes, os.PathLike]]], split: Optional[datasets.splits.NamedSplit] = None, features: Optional[datasets.features.features.Features] = None, keep_in_memory: bool = False, keep_linebreaks: bool = False, sample_by: Literal['line', 'paragraph', 'document'] = 'line', **kwargs) -> 'IterableDataset'
| Create an IterableDataset from text file(s).
|
| Args:
| path_or_paths (`path-like` or list of `path-like`):
| Path(s) of the text file(s).
| split (`NamedSplit`, *optional*):
| Split name to be assigned to the dataset.
| features (`Features`, *optional*):
| Dataset features.
| keep_in_memory (`bool`, defaults to `False`):
| Whether to copy the data in-memory.
| keep_linebreaks: (`bool`, defaults to False):
| Whether to keep line breaks.
| sample_by (`Literal["line", "paragraph", "document"]`, defaults to "line"):
| Whether to load data per line, praragraph or document.
| By default one row in the dataset = one line.
| **kwargs (additional keyword arguments):
| Keyword arguments to be passed to [`TextConfig`].
|
| Returns:
| [`IterableDataset`]
|
| Example:
|
| ```py
| >>> ds = IterableDataset.from_text('path/to/dataset.txt')
| ```
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| column_names
| Names of the columns in the dataset.
| This can be None if the dataset has unknown features (e.g. after a map() operation).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation", streaming=True)
| >>> ds.column_names
| ['text', 'label']
| ```
|
| epoch
|
| n_shards
|
| num_columns
| Number of columns in the dataset.
| This can be None if the dataset has unknown features (e.g. after a map() operation).
|
| Example:
|
| ```py
| >>> from datasets import load_dataset
| >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
| >>> ds.num_columns
| 2
| ```
|
| num_shards
|
| ----------------------------------------------------------------------
| Readonly properties inherited from datasets.arrow_dataset.DatasetInfoMixin:
|
| builder_name
|
| citation
|
| config_name
|
| dataset_size
|
| description
|
| download_checksums
|
| download_size
|
| features
|
| homepage
|
| info
| [`~datasets.DatasetInfo`] object containing all the metadata in the dataset.
|
| license
|
| size_in_bytes
|
| split
| [`~datasets.NamedSplit`] object corresponding to a named dataset split.
|
| supervised_keys
|
| version
|
| ----------------------------------------------------------------------
| Data descriptors inherited from datasets.arrow_dataset.DatasetInfoMixin:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class MappedExamplesIterable(_BaseExamplesIterable)
| MappedExamplesIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, function: Callable, with_indices: bool = False, input_columns: Optional[list[str]] = None, batched: bool = False, batch_size: Optional[int] = 1000, drop_last_batch: bool = False, remove_columns: Optional[list[str]] = None, fn_kwargs: Optional[dict] = None, formatting: Optional[ForwardRef('FormattingConfig')] = None, features: Optional[datasets.features.features.Features] = None, max_num_running_async_map_functions_in_parallel: Optional[int] = None, is_batch_accumulate_arrow_table_function: bool = False)
|
| Method resolution order:
| MappedExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, function: Callable, with_indices: bool = False, input_columns: Optional[list[str]] = None, batched: bool = False, batch_size: Optional[int] = 1000, drop_last_batch: bool = False, remove_columns: Optional[list[str]] = None, fn_kwargs: Optional[dict] = None, formatting: Optional[ForwardRef('FormattingConfig')] = None, features: Optional[datasets.features.features.Features] = None, max_num_running_async_map_functions_in_parallel: Optional[int] = None, is_batch_accumulate_arrow_table_function: bool = False)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'MappedExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'MappedExamplesIterable'
| Keep only the requested shard.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'MappedExamplesIterable'
| Shuffle the wrapped examples iterable.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class RandomlyCyclingMultiSourcesExamplesIterable(CyclingMultiSourcesExamplesIterable)
| RandomlyCyclingMultiSourcesExamplesIterable(ex_iterables: list[datasets.iterable_dataset._BaseExamplesIterable], generator: numpy.random._generator.Generator, probabilities: Optional[list[float]] = None, stopping_strategy: Literal['first_exhausted', 'all_exhausted', 'all_exhausted_without_replacement'] = 'first_exhausted')
|
| Method resolution order:
| RandomlyCyclingMultiSourcesExamplesIterable
| CyclingMultiSourcesExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterables: list[datasets.iterable_dataset._BaseExamplesIterable], generator: numpy.random._generator.Generator, probabilities: Optional[list[float]] = None, stopping_strategy: Literal['first_exhausted', 'all_exhausted', 'all_exhausted_without_replacement'] = 'first_exhausted')
| Initialize self. See help(type(self)) for accurate signature.
|
| reshard_data_sources(self) -> 'RandomlyCyclingMultiSourcesExamplesIterable'
| Either keep only the requested shard, or propagate the request to the underlying iterable.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'RandomlyCyclingMultiSourcesExamplesIterable'
| Either keep only the requested shard, or propagate the request to the underlying iterable.
|
| shift_rngs(self, value: int) -> '_BaseExamplesIterable'
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'RandomlyCyclingMultiSourcesExamplesIterable'
| Shuffle the data sources of each wrapped examples iterable.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| ----------------------------------------------------------------------
| Methods inherited from CyclingMultiSourcesExamplesIterable:
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| ----------------------------------------------------------------------
| Readonly properties inherited from CyclingMultiSourcesExamplesIterable:
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class RebatchedArrowExamplesIterable(_BaseExamplesIterable)
| RebatchedArrowExamplesIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, batch_size: Optional[int], drop_last_batch: bool = False, force_convert_to_arrow: bool = False)
|
| Method resolution order:
| RebatchedArrowExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, batch_size: Optional[int], drop_last_batch: bool = False, force_convert_to_arrow: bool = False)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'RebatchedArrowExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'RebatchedArrowExamplesIterable'
| Either keep only the requested shard, or propagate the request to the underlying iterable.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'RebatchedArrowExamplesIterable'
| Either shuffle the shards/sources of the dataset, or propagate the shuffling to the underlying iterable.
| If the order of the shards must stay fixed (when using .skip or .take for example), then this method returns self.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class RepeatExamplesIterable(_BaseExamplesIterable)
| RepeatExamplesIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, num_times: Optional[int])
|
| Iterable that repeats the underlying iterable a given number of times.
|
| Method resolution order:
| RepeatExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, num_times: Optional[int])
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'RepeatExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'RepeatExamplesIterable'
| Shard, then repeat shards.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'RepeatExamplesIterable'
| Shuffle the underlying iterable, then repeat.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| features
|
| is_typed
|
| iter_arrow
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class SelectColumnsIterable(_BaseExamplesIterable)
| SelectColumnsIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, column_names: list[str])
|
| Method resolution order:
| SelectColumnsIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, column_names: list[str])
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'SelectColumnsIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'SelectColumnsIterable'
| Either keep only the requested shard, or propagate the request to the underlying iterable.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'SelectColumnsIterable'
| Either shuffle the shards/sources of the dataset, or propagate the shuffling to the underlying iterable.
| If the order of the shards must stay fixed (when using .skip or .take for example), then this method returns self.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class SkipExamplesIterable(_BaseExamplesIterable)
| SkipExamplesIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, n: int, block_sources_order_when_shuffling: bool = True, split_when_sharding: bool = True)
|
| Method resolution order:
| SkipExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, n: int, block_sources_order_when_shuffling: bool = True, split_when_sharding: bool = True)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'SkipExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'SkipExamplesIterable'
| Keep only the requested shard.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'SkipExamplesIterable'
| May not shuffle the wrapped examples iterable since it would skip examples from other shards instead.
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| split_number(num, n)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class StepExamplesIterable(_BaseExamplesIterable)
| StepExamplesIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, step: int, offset: int)
|
| Method resolution order:
| StepExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, step: int, offset: int)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'StepExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'StepExamplesIterable'
| Either keep only the requested shard, or propagate the request to the underlying iterable.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'StepExamplesIterable'
| Either shuffle the shards/sources of the dataset, or propagate the shuffling to the underlying iterable.
| If the order of the shards must stay fixed (when using .skip or .take for example), then this method returns self.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class TakeExamplesIterable(_BaseExamplesIterable)
| TakeExamplesIterable(ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, n: int, block_sources_order_when_shuffling: bool = True, split_when_sharding: bool = True)
|
| Method resolution order:
| TakeExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterable: datasets.iterable_dataset._BaseExamplesIterable, n: int, block_sources_order_when_shuffling: bool = True, split_when_sharding: bool = True)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'TakeExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'TakeExamplesIterable'
| Keep only the requested shard.
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'TakeExamplesIterable'
| May not shuffle the wrapped examples iterable since it would take examples from other shards instead.
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| split_number(num, n)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class VerticallyConcatenatedMultiSourcesExamplesIterable(_BaseExamplesIterable)
| VerticallyConcatenatedMultiSourcesExamplesIterable(ex_iterables: list[datasets.iterable_dataset._BaseExamplesIterable])
|
| VerticallyConcatenatedMultiSourcesExamplesIterable simply chains the input iterables.
| It doesn't require the examples iterables to always yield the same columns.
| Instead, this is handled by the `IterableDataset` class or `FormattedExamplesIterable`.
|
| For information, `IterableDataset` merges the features of all the datasets to concatenate into one.
| We use `IterableDataset._resolve_features` to obtain the features of all the datasets to concatenate.
|
| Then for each example, `IterableDataset` and `FormattedExamplesIterable` automatically fill missing columns with None.
| This is done with `_apply_feature_types_on_example`.
|
| Method resolution order:
| VerticallyConcatenatedMultiSourcesExamplesIterable
| _BaseExamplesIterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, ex_iterables: list[datasets.iterable_dataset._BaseExamplesIterable])
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
| An examples iterable should yield tuples (example_key, example) of type (int/str, dict)
|
| reshard_data_sources(self) -> 'VerticallyConcatenatedMultiSourcesExamplesIterable'
| Either reshard the shards/sources of the dataset, i.e. further split the current shards into more shards,
| or propagate the resharding to the underlying iterable.
| If the examples iterable can't be further resharded, then this method returns self.
|
| shard_data_sources(self, num_shards: int, index: int, contiguous=True) -> 'VerticallyConcatenatedMultiSourcesExamplesIterable'
| Keep only the requested shard
|
| shuffle_data_sources(self, generator: numpy.random._generator.Generator) -> 'VerticallyConcatenatedMultiSourcesExamplesIterable'
| Shuffle all shards.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| features
|
| is_typed
|
| iter_arrow
|
| num_shards
|
| ----------------------------------------------------------------------
| Methods inherited from _BaseExamplesIterable:
|
| load_state_dict(self, state_dict: dict) -> dict
|
| split_shard_indices_by_worker(self, num_shards: int, index: int, contiguous=True) -> list[int]
|
| state_dict(self) -> dict
|
| ----------------------------------------------------------------------
| Readonly properties inherited from _BaseExamplesIterable:
|
| sleep_on_threads_shutdown
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _BaseExamplesIterable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
add_column_fn(example: dict, idx: int, name: str, column: list[dict])
add_mask(mask_function: Callable, input: Union[dict, pyarrow.lib.Table], *args, mask_column_name: str, **kwargs)
async async_add_mask(mask_function: Callable, input: Union[dict, pyarrow.lib.Table], *args, mask_column_name: str, **kwargs)
identity_func(x)
shift_ex_examples_rngs(ex_iterable: '_BaseExamplesIterable', value: int) -> '_BaseExamplesIterable'
We need to go through the ex_iterables recursively, create a new seed and return a new iterable, then set it to the containing ex_iterable.
DATA
Callable = typing.Callable
Deprecated alias to collections.abc.Callable.
Callable[[int], str] signifies a function that takes a single
parameter of type int and returns a str.
The subscription syntax must always be used with exactly two
values: the argument list and the return type.
The argument list must be a list of types, a ParamSpec,
Concatenate or ellipsis. The return type must be a single type.
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
FeatureType = typing.Union[dict, list, tuple, datasets.feature....feat...
Key = typing.Union[int, str, tuple[int, int], ForwardRef('BuilderKey')...
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
PathLike = typing.Union[str, bytes, os.PathLike]
TYPE_CHECKING = False
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.iterable_dataset (WARNING)>
VERSION
5.0.1
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\iterable_dataset.py
datasets.json
Python Library Documentation: module datasets.utils.json in datasets.utils
NAME
datasets.utils.json
FUNCTIONS
find_mixed_struct_types_field_paths(examples: list, allow_root=False) -> list[list[str]]
get_json_field_path_from_pyarrow_json_error(err_str: str) -> list[str]
get_json_field_paths_from_feature(feature: 'FeatureType') -> list[list[str]]
insert_json_field_path(json_field_paths: list[list[str]], json_field_path: list[str]) -> None
json_decode_field(example: Any, json_field_path: str) -> Any
json_encode_field(example: Any, json_field_path: list[str]) -> Any
json_encode_fields_in_json_lines(original_batch: bytes, json_field_paths: list[list[str]]) -> bytes
set_json_types_in_feature(feature: 'FeatureType', json_field_paths: list[list[str]]) -> None
ujson_dumps(*args, **kwargs)
ujson_loads(*args, **kwargs)
DATA
TYPE_CHECKING = False
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\json.py
datasets.load
Python Library Documentation: module datasets.load in datasets
NAME
datasets.load - Access datasets.
CLASSES
builtins.object
BuilderConfigsParameters
DatasetModule
_DatasetModuleFactory(builtins.object)
CachedDatasetModuleFactory
HubBucketDatasetModuleFactory
HubDatasetModuleFactory
HubDatasetModuleFactoryWithParquetExport
LocalDatasetModuleFactory
PackagedDatasetModuleFactory
class BuilderConfigsParameters(builtins.object)
| BuilderConfigsParameters(metadata_configs: Optional[datasets.utils.metadata.MetadataConfigs] = None, builder_configs: Optional[list[datasets.builder.BuilderConfig]] = None, default_config_name: Optional[str] = None) -> None
|
| Dataclass containing objects related to creation of builder configurations from yaml's metadata content.
|
| Attributes:
| metadata_configs (`MetadataConfigs`, *optional*):
| Configs parsed from yaml's metadata.
| builder_configs (`list[BuilderConfig]`, *optional*):
| List of BuilderConfig objects created from metadata_configs above.
| default_config_name (`str`):
| Name of default config taken from yaml's metadata.
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, metadata_configs: Optional[datasets.utils.metadata.MetadataConfigs] = None, builder_configs: Optional[list[datasets.builder.BuilderConfig]] = None, default_config_name: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'builder_configs': typing.Optional[list[datasets.bu...
|
| __dataclass_fields__ = {'builder_configs': Field(name='builder_configs...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('metadata_configs', 'builder_configs', 'default_conf...
|
| builder_configs = None
|
| default_config_name = None
|
| metadata_configs = None
class CachedDatasetModuleFactory(_DatasetModuleFactory)
| CachedDatasetModuleFactory(name: str, cache_dir: Optional[str] = None)
|
| Get the module of a dataset that has been loaded once already and cached.
|
| Method resolution order:
| CachedDatasetModuleFactory
| _DatasetModuleFactory
| builtins.object
|
| Methods defined here:
|
| __init__(self, name: str, cache_dir: Optional[str] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| get_module(self) -> datasets.load.DatasetModule
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _DatasetModuleFactory:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class DatasetModule(builtins.object)
| DatasetModule(module_path: str, hash: str, builder_kwargs: dict, builder_configs_parameters: datasets.load.BuilderConfigsParameters = <factory>, dataset_infos: Optional[datasets.info.DatasetInfosDict] = None) -> None
|
| DatasetModule(module_path: str, hash: str, builder_kwargs: dict, builder_configs_parameters: datasets.load.BuilderConfigsParameters = <factory>, dataset_infos: Optional[datasets.info.DatasetInfosDict] = None)
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, module_path: str, hash: str, builder_kwargs: dict, builder_configs_parameters: datasets.load.BuilderConfigsParameters = <factory>, dataset_infos: Optional[datasets.info.DatasetInfosDict] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'builder_configs_parameters': <class 'datasets.load...
|
| __dataclass_fields__ = {'builder_configs_parameters': Field(name='buil...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('module_path', 'hash', 'builder_kwargs', 'builder_co...
|
| dataset_infos = None
class HubBucketDatasetModuleFactory(_DatasetModuleFactory)
| HubBucketDatasetModuleFactory(path: str, data_dir: Optional[str] = None, data_files: Union[str, list, dict, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None)
|
| Get the module of a dataset loaded from data files of a a Storage Bucket.
| The dataset builder module to use is inferred from the data files extensions.
|
| Method resolution order:
| HubBucketDatasetModuleFactory
| _DatasetModuleFactory
| builtins.object
|
| Methods defined here:
|
| __init__(self, path: str, data_dir: Optional[str] = None, data_files: Union[str, list, dict, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| get_module(self) -> datasets.load.DatasetModule
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _DatasetModuleFactory:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class HubDatasetModuleFactory(_DatasetModuleFactory)
| HubDatasetModuleFactory(name: str, commit_hash: str, data_dir: Optional[str] = None, data_files: Union[str, list, dict, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, use_exported_dataset_infos: bool = False)
|
| Get the module of a dataset loaded from data files of a dataset repository.
| The dataset builder module to use is inferred from the data files extensions.
|
| Method resolution order:
| HubDatasetModuleFactory
| _DatasetModuleFactory
| builtins.object
|
| Methods defined here:
|
| __init__(self, name: str, commit_hash: str, data_dir: Optional[str] = None, data_files: Union[str, list, dict, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, use_exported_dataset_infos: bool = False)
| Initialize self. See help(type(self)) for accurate signature.
|
| get_module(self) -> datasets.load.DatasetModule
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _DatasetModuleFactory:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class HubDatasetModuleFactoryWithParquetExport(_DatasetModuleFactory)
| HubDatasetModuleFactoryWithParquetExport(name: str, commit_hash: str, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
|
| Get the module of a dataset loaded from parquet files of a dataset repository parquet export.
|
| Method resolution order:
| HubDatasetModuleFactoryWithParquetExport
| _DatasetModuleFactory
| builtins.object
|
| Methods defined here:
|
| __init__(self, name: str, commit_hash: str, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| get_module(self) -> datasets.load.DatasetModule
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _DatasetModuleFactory:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class LocalDatasetModuleFactory(_DatasetModuleFactory)
| LocalDatasetModuleFactory(path: str, data_dir: Optional[str] = None, data_files: Union[str, list, dict, NoneType] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None)
|
| Get the module of a dataset loaded from the user's data files. The dataset builder module to use is inferred
| from the data files extensions.
|
| Method resolution order:
| LocalDatasetModuleFactory
| _DatasetModuleFactory
| builtins.object
|
| Methods defined here:
|
| __init__(self, path: str, data_dir: Optional[str] = None, data_files: Union[str, list, dict, NoneType] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| get_module(self) -> datasets.load.DatasetModule
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _DatasetModuleFactory:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class PackagedDatasetModuleFactory(_DatasetModuleFactory)
| PackagedDatasetModuleFactory(name: str, data_dir: Optional[str] = None, data_files: Union[str, list, dict, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None)
|
| Get the dataset builder module from the ones that are packaged with the library: csv, json, etc.
|
| Method resolution order:
| PackagedDatasetModuleFactory
| _DatasetModuleFactory
| builtins.object
|
| Methods defined here:
|
| __init__(self, name: str, data_dir: Optional[str] = None, data_files: Union[str, list, dict, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| get_module(self) -> datasets.load.DatasetModule
|
| ----------------------------------------------------------------------
| Data descriptors inherited from _DatasetModuleFactory:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
configure_builder_class(builder_cls: type[datasets.builder.DatasetBuilder], builder_configs: list[datasets.builder.BuilderConfig], default_config_name: Optional[str], dataset_name: str) -> type[datasets.builder.DatasetBuilder]
Dynamically create a builder class with custom builder configs parsed from README.md file,
i.e. set BUILDER_CONFIGS class variable of a builder class to custom configs list.
create_builder_configs_from_metadata_configs(module_path: str, metadata_configs: datasets.utils.metadata.MetadataConfigs, base_path: Optional[str] = None, default_builder_kwargs: dict[str, typing.Any] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> tuple[list[datasets.builder.BuilderConfig], str]
dataset_module_factory(path: str, revision: Union[datasets.utils.version.Version, str, NoneType] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, data_dir: Optional[str] = None, data_files: Union[str, list, dict, datasets.data_files.DataFilesDict, NoneType] = None, cache_dir: Optional[str] = None, **download_kwargs) -> datasets.load.DatasetModule
Download/extract/cache a dataset module.
Dataset codes are cached inside the dynamic modules cache to allow easy import (avoid ugly sys.path tweaks).
Args:
path (str): Path or name of the dataset.
Depending on ``path``, the dataset builder that is used comes from one of the generic dataset builders (JSON, CSV, Parquet, text etc.).
For local datasets:
- if ``path`` is a local directory (containing data files only)
-> load a generic dataset builder (csv, json, text etc.) based on the content of the directory
e.g. ``'./path/to/directory/with/my/csv/data'``.
For datasets on the Hugging Face Hub (list all available datasets with ``huggingface_hub.list_datasets()``)
- if ``path`` is a dataset repository on the HF hub (containing data files only)
-> load a generic dataset builder (csv, text etc.) based on the content of the repository
e.g. ``'username/dataset_name'``, a dataset repository on the HF hub containing your data files.
For datasets in Storage Buckets on the Hugging Face Hub (list all available buckets with ``huggingface_hub.list_buckets()``)
- if `path` is a directory within a Storage Bucket on the HF Hub (containing data files only)
-> load the dataset from supported files in the directory (csv, json, parquet, etc.)
e.g. `'buckets/username/bucket_name/my_dataset'`.
revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset to load.
As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch.
You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository.
download_config (:class:`DownloadConfig`, optional): Specific download configuration parameters.
download_mode (:class:`DownloadMode` or :obj:`str`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode.
data_dir (:obj:`str`, optional): Directory with the data files. Used only if `data_files` is not specified,
in which case it's equal to pass `os.path.join(data_dir, "**")` as `data_files`.
data_files (:obj:`Union[Dict, List, str]`, optional): Defining the data_files of the dataset configuration.
cache_dir (`str`, *optional*):
Directory to read/write data. Defaults to `"~/.cache/huggingface/datasets"`.
<Added version="2.16.0"/>
**download_kwargs (additional keyword arguments): optional attributes for DownloadConfig() which will override
the attributes in download_config if supplied.
Returns:
DatasetModule
get_dataset_builder_class(dataset_module: 'DatasetModule', dataset_name: Optional[str] = None) -> type[datasets.builder.DatasetBuilder]
import_main_class(module_path) -> Optional[type[datasets.builder.DatasetBuilder]]
Import a module at module_path and return its main class: a DatasetBuilder
increase_load_count(name: str)
Update the download count of a dataset.
infer_module_for_data_files(data_files: datasets.data_files.DataFilesDict, path: Optional[str] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> tuple[typing.Optional[str], dict[str, typing.Any]]
Infer module (and builder kwargs) from data files. Raise if module names for different splits don't match.
Args:
data_files ([`DataFilesDict`]): Dict of list of data files.
path (str, *optional*): Dataset name or path.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters to authenticate on the Hugging Face Hub for private remote files.
Returns:
tuple[str, dict[str, Any]]: Tuple with
- inferred module name
- builder kwargs
infer_module_for_data_files_list(data_files_list: datasets.data_files.DataFilesList, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> tuple[typing.Optional[str], dict]
Infer module (and builder kwargs) from list of data files.
It picks the module based on the most common file extension.
In case of a draw ".parquet" is the favorite, and then alphabetical order.
Args:
data_files_list (DataFilesList): List of data files.
download_config (bool or str, optional): Mainly use `token` or `storage_options` to support different platforms and auth types.
Returns:
tuple[str, dict[str, Any]]: Tuple with
- inferred module name
- dict of builder kwargs
infer_module_for_data_files_list_in_archives(data_files_list: datasets.data_files.DataFilesList, download_config: Optional[datasets.download.download_config.DownloadConfig] = None) -> tuple[typing.Optional[str], dict]
Infer module (and builder kwargs) from list of archive data files.
Args:
data_files_list (DataFilesList): List of data files.
download_config (bool or str, optional): Mainly use `token` or `storage_options` to support different platforms and auth types.
Returns:
tuple[str, dict[str, Any]]: Tuple with
- inferred module name
- dict of builder kwargs
load_dataset(path: str, name: Optional[str] = None, data_dir: Optional[str] = None, data_files: Union[str, collections.abc.Sequence[str], collections.abc.Mapping[str, Union[str, collections.abc.Sequence[str]]], NoneType] = None, split: Union[str, datasets.splits.Split, list[str], list[datasets.splits.Split], NoneType] = None, cache_dir: Optional[str] = None, features: Optional[datasets.features.features.Features] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, verification_mode: Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None, keep_in_memory: Optional[bool] = None, save_infos: bool = False, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, streaming: bool = False, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **config_kwargs) -> Union[datasets.dataset_dict.DatasetDict, datasets.arrow_dataset.Dataset, datasets.dataset_dict.IterableDatasetDict, datasets.iterable_dataset.IterableDataset]
Load a dataset from the Hugging Face Hub, or a local dataset.
You can find the list of datasets on the [Hub](https://huggingface.co/datasets) or with [`huggingface_hub.list_datasets`].
A dataset is a directory that contains some data files in generic formats (JSON, CSV, Parquet, etc.) and possibly
in a generic structure (Webdataset, ImageFolder, AudioFolder, VideoFolder, MeshFolder, etc.)
This function does the following under the hood:
1. Load a dataset builder:
* Find the most common data format in the dataset and pick its associated builder (JSON, CSV, Parquet, Webdataset, ImageFolder, AudioFolder, MeshFolder, etc.)
* Find which file goes into which split (e.g. train/test) based on file and directory names or on the YAML configuration
* It is also possible to specify `data_files` manually, and which dataset builder to use (e.g. "parquet").
2. Run the dataset builder:
In the general case:
* Download the data files from the dataset if they are not already available locally or cached.
* Process and cache the dataset in typed Arrow tables for caching.
Arrow table are arbitrarily long, typed tables which can store nested objects and be mapped to numpy/pandas/python generic types.
They can be directly accessed from disk, loaded in RAM or even streamed over the web.
In the streaming case:
* Don't download or cache anything. Instead, the dataset is lazily loaded and will be streamed on-the-fly when iterating on it.
3. Return a dataset built from the requested splits in `split` (default: all).
Args:
path (`str`):
Path or name of the dataset.
- if `path` is a dataset repository on the HF hub (list all available datasets with [`huggingface_hub.list_datasets`])
-> load the dataset from supported files in the repository (csv, json, parquet, etc.)
e.g. `'username/dataset_name'`, a dataset repository on the HF hub containing the data files.
- if `path` is a directory within a Storage Bucket on the HF Hub (list your buckets with [`huggingface_hub.list_buckets`])
-> load the dataset from supported files in the directory (csv, json, parquet, etc.)
e.g. `'buckets/username/bucket_name/my_dataset'`.
- if `path` is a local directory
-> load the dataset from supported files in the directory (csv, json, parquet, etc.)
e.g. `'./path/to/directory/with/my/csv/data'`.
- if `path` is the name of a dataset builder and `data_files` or `data_dir` is specified
(available builders are "json", "csv", "parquet", "arrow", "text", "xml", "webdataset", "imagefolder", "audiofolder", "videofolder", "meshfolder")
-> load the dataset from the files in `data_files` or `data_dir`
e.g. `'parquet'`.
Use a `hf://` path like `'hf://datasets/username/dataset_name'` to allow remote only.
Use an absolute path to allow local only.
name (`str`, *optional*):
Defining the name of the dataset configuration.
data_dir (`str`, *optional*):
Defining the `data_dir` of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and `data_files` is `None`,
the behavior is equal to passing `os.path.join(data_dir, **)` as `data_files` to reference all the files in a directory.
data_files (`str` or `Sequence` or `Mapping`, *optional*):
Path(s) to source data file(s).
split (`Split` or `str`):
Which split of the data to load.
If `None`, will return a `dict` with all splits (typically `datasets.Split.TRAIN` and `datasets.Split.TEST`).
If given, will return a single Dataset.
Splits can be combined and specified like in tensorflow-datasets.
cache_dir (`str`, *optional*):
Directory to read/write data. Defaults to `"~/.cache/huggingface/datasets"`.
features (`Features`, *optional*):
Set the features type to use for this dataset.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/...).
<Added version="2.9.1"/>
keep_in_memory (`bool`, defaults to `None`):
Whether to copy the dataset in-memory. If `None`, the dataset
will not be copied in-memory unless explicitly enabled by setting `datasets.config.IN_MEMORY_MAX_SIZE` to
nonzero. See more details in the [improve performance](../cache#improve-performance) section.
revision ([`Version`] or `str`, *optional*):
Version of the dataset to load.
As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch.
You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository.
token (`str` or `bool`, *optional*):
Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If `True`, or not specified, will get token from `"~/.huggingface"`.
streaming (`bool`, defaults to `False`):
If set to `True`, don't download the data files. Instead, it streams the data progressively while
iterating on the dataset. An [`IterableDataset`] or [`IterableDatasetDict`] is returned instead in this case.
Note that streaming works for datasets that use data formats that support being iterated over like txt, csv, jsonl for example.
Json files may be downloaded completely. Also streaming from remote zip or gzip files is supported but other compressed formats
like rar and xz are not yet supported. The tgz format doesn't allow streaming.
num_proc (`int`, *optional*, defaults to `None`):
Number of processes when downloading and generating the dataset locally.
Multiprocessing is disabled by default.
<Added version="2.7.0"/>
storage_options (`dict`, *optional*, defaults to `None`):
**Experimental**. Key/value pairs to be passed on to the dataset file-system backend, if any.
<Added version="2.11.0"/>
**config_kwargs (additional keyword arguments):
Keyword arguments to be passed to the `BuilderConfig`
and used in the [`DatasetBuilder`].
Returns:
[`Dataset`] or [`DatasetDict`]:
- if `split` is not `None`: the dataset requested,
- if `split` is `None`, a [`~datasets.DatasetDict`] with each split.
or [`IterableDataset`] or [`IterableDatasetDict`]: if `streaming=True`
- if `split` is not `None`, the dataset is requested
- if `split` is `None`, a [`~datasets.streaming.IterableDatasetDict`] with each split.
Example:
Load a dataset from the Hugging Face Hub:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('cornell-movie-review-data/rotten_tomatoes', split='train')
# Load a subset or dataset configuration (here 'sst2')
>>> from datasets import load_dataset
>>> ds = load_dataset('nyu-mll/glue', 'sst2', split='train')
# Manual mapping of data files to splits
>>> data_files = {'train': 'train.csv', 'test': 'test.csv'}
>>> ds = load_dataset('namespace/your_dataset_name', data_files=data_files)
# Manual selection of a directory to load
>>> ds = load_dataset('namespace/your_dataset_name', data_dir='folder_name')
```
Load a dataset from a Storage Bucket on the Hugging Face Hub:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('buckets/username/bucket_name/rotten_tomatoes', split='train')
```
Load a local dataset:
```py
# Load a CSV file
>>> from datasets import load_dataset
>>> ds = load_dataset('csv', data_files='path/to/local/my_dataset.csv')
# Load a JSON file
>>> from datasets import load_dataset
>>> ds = load_dataset('json', data_files='path/to/local/my_dataset.json')
```
Load an [`~datasets.IterableDataset`]:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('cornell-movie-review-data/rotten_tomatoes', split='train', streaming=True)
```
Load an image dataset with the `ImageFolder` dataset builder:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('imagefolder', data_dir='/path/to/images', split='train')
```
load_dataset_builder(path: str, name: Optional[str] = None, data_dir: Optional[str] = None, data_files: Union[str, collections.abc.Sequence[str], collections.abc.Mapping[str, Union[str, collections.abc.Sequence[str]]], NoneType] = None, cache_dir: Optional[str] = None, features: Optional[datasets.features.features.Features] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, storage_options: Optional[dict] = None, **config_kwargs) -> datasets.builder.DatasetBuilder
Load a dataset builder which can be used to:
- Inspect general information that is required to build a dataset (cache directory, config, dataset info, features, data files, etc.)
- Download and prepare the dataset as Arrow files in the cache
- Get a streaming dataset without downloading or caching anything
You can find the list of datasets on the [Hub](https://huggingface.co/datasets) or with [`huggingface_hub.list_datasets`].
A dataset is a directory that contains some data files in generic formats (JSON, CSV, Parquet, etc.) and possibly
in a generic structure (Webdataset, ImageFolder, AudioFolder, VideoFolder, MeshFolder, etc.)
Args:
path (`str`):
Path or name of the dataset.
- if `path` is a dataset repository on the HF hub (list all available datasets with [`huggingface_hub.list_datasets`])
-> load the dataset builder from supported files in the repository (csv, json, parquet, etc.)
e.g. `'username/dataset_name'`, a dataset repository on the HF hub containing the data files.
- if `path` is a directory within a Storage Bucket on the HF Hub (list your buckets with [`huggingface_hub.list_buckets`])
-> load the dataset from supported files in the directory (csv, json, parquet, etc.)
e.g. `'buckets/username/bucket_name/my_dataset'`.
- if `path` is a local directory
-> load the dataset builder from supported files in the directory (csv, json, parquet, etc.)
e.g. `'./path/to/directory/with/my/csv/data'`.
- if `path` is the name of a dataset builder and `data_files` or `data_dir` is specified
(available builders are "json", "csv", "parquet", "arrow", "text", "xml", "webdataset", "imagefolder", "audiofolder", "videofolder", "meshfolder")
-> load the dataset builder from the files in `data_files` or `data_dir`
e.g. `'parquet'`.
Use a `hf://` path like `'hf://datasets/username/dataset_name'` to allow remote only.
Use an absolute path to allow local only.
name (`str`, *optional*):
Defining the name of the dataset configuration.
data_dir (`str`, *optional*):
Defining the `data_dir` of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and `data_files` is `None`,
the behavior is equal to passing `os.path.join(data_dir, **)` as `data_files` to reference all the files in a directory.
data_files (`str` or `Sequence` or `Mapping`, *optional*):
Path(s) to source data file(s).
cache_dir (`str`, *optional*):
Directory to read/write data. Defaults to `"~/.cache/huggingface/datasets"`.
features ([`Features`], *optional*):
Set the features type to use for this dataset.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
revision ([`Version`] or `str`, *optional*):
Version of the dataset to load.
As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch.
You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository.
token (`str` or `bool`, *optional*):
Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If `True`, or not specified, will get token from `"~/.huggingface"`.
storage_options (`dict`, *optional*, defaults to `None`):
**Experimental**. Key/value pairs to be passed on to the dataset file-system backend, if any.
<Added version="2.11.0"/>
**config_kwargs (additional keyword arguments):
Keyword arguments to be passed to the [`BuilderConfig`]
and used in the [`DatasetBuilder`].
Returns:
[`DatasetBuilder`]
Example:
```py
>>> from datasets import load_dataset_builder
>>> ds_builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')
>>> ds_builder.info.features
{'label': ClassLabel(names=['neg', 'pos']),
'text': Value('string')}
```
load_from_disk(dataset_path: Union[str, bytes, os.PathLike], keep_in_memory: Optional[bool] = None, storage_options: Optional[dict] = None) -> Union[datasets.arrow_dataset.Dataset, datasets.dataset_dict.DatasetDict]
Loads a dataset that was previously saved using [`~Dataset.save_to_disk`] from a dataset directory, or
from a filesystem using any implementation of `fsspec.spec.AbstractFileSystem`.
Args:
dataset_path (`path-like`):
Path (e.g. `"dataset/train"`) or remote URI (e.g. `"s3://my-bucket/dataset/train"`)
of the [`Dataset`] or [`DatasetDict`] directory where the dataset/dataset-dict will be
loaded from.
keep_in_memory (`bool`, defaults to `None`):
Whether to copy the dataset in-memory. If `None`, the dataset
will not be copied in-memory unless explicitly enabled by setting `datasets.config.IN_MEMORY_MAX_SIZE` to
nonzero. See more details in the [improve performance](../cache#improve-performance) section.
storage_options (`dict`, *optional*):
Key/value pairs to be passed on to the file-system backend, if any.
<Added version="2.9.0"/>
Returns:
[`Dataset`] or [`DatasetDict`]:
- If `dataset_path` is a path of a dataset directory: the dataset requested.
- If `dataset_path` is a path of a dataset dict directory, a [`DatasetDict`] with each split.
Example:
```py
>>> from datasets import load_from_disk
>>> ds = load_from_disk('path/to/dataset/directory')
```
DATA
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
PathLike = typing.Union[str, bytes, os.PathLike]
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
hf_dataset_url = functools.partial(<function hf_hub_url at 0x000001A65...
logger = <Logger datasets.load (WARNING)>
VERSION
5.0.1
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\load.py
datasets.load_dataset
Python Library Documentation: function load_dataset in module datasets.load
load_dataset(path: str, name: Optional[str] = None, data_dir: Optional[str] = None, data_files: Union[str, collections.abc.Sequence[str], collections.abc.Mapping[str, Union[str, collections.abc.Sequence[str]]], NoneType] = None, split: Union[str, datasets.splits.Split, list[str], list[datasets.splits.Split], NoneType] = None, cache_dir: Optional[str] = None, features: Optional[datasets.features.features.Features] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, verification_mode: Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None, keep_in_memory: Optional[bool] = None, save_infos: bool = False, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, streaming: bool = False, num_proc: Optional[int] = None, storage_options: Optional[dict] = None, **config_kwargs) -> Union[datasets.dataset_dict.DatasetDict, datasets.arrow_dataset.Dataset, datasets.dataset_dict.IterableDatasetDict, datasets.iterable_dataset.IterableDataset]
Load a dataset from the Hugging Face Hub, or a local dataset.
You can find the list of datasets on the [Hub](https://huggingface.co/datasets) or with [`huggingface_hub.list_datasets`].
A dataset is a directory that contains some data files in generic formats (JSON, CSV, Parquet, etc.) and possibly
in a generic structure (Webdataset, ImageFolder, AudioFolder, VideoFolder, MeshFolder, etc.)
This function does the following under the hood:
1. Load a dataset builder:
* Find the most common data format in the dataset and pick its associated builder (JSON, CSV, Parquet, Webdataset, ImageFolder, AudioFolder, MeshFolder, etc.)
* Find which file goes into which split (e.g. train/test) based on file and directory names or on the YAML configuration
* It is also possible to specify `data_files` manually, and which dataset builder to use (e.g. "parquet").
2. Run the dataset builder:
In the general case:
* Download the data files from the dataset if they are not already available locally or cached.
* Process and cache the dataset in typed Arrow tables for caching.
Arrow table are arbitrarily long, typed tables which can store nested objects and be mapped to numpy/pandas/python generic types.
They can be directly accessed from disk, loaded in RAM or even streamed over the web.
In the streaming case:
* Don't download or cache anything. Instead, the dataset is lazily loaded and will be streamed on-the-fly when iterating on it.
3. Return a dataset built from the requested splits in `split` (default: all).
Args:
path (`str`):
Path or name of the dataset.
- if `path` is a dataset repository on the HF hub (list all available datasets with [`huggingface_hub.list_datasets`])
-> load the dataset from supported files in the repository (csv, json, parquet, etc.)
e.g. `'username/dataset_name'`, a dataset repository on the HF hub containing the data files.
- if `path` is a directory within a Storage Bucket on the HF Hub (list your buckets with [`huggingface_hub.list_buckets`])
-> load the dataset from supported files in the directory (csv, json, parquet, etc.)
e.g. `'buckets/username/bucket_name/my_dataset'`.
- if `path` is a local directory
-> load the dataset from supported files in the directory (csv, json, parquet, etc.)
e.g. `'./path/to/directory/with/my/csv/data'`.
- if `path` is the name of a dataset builder and `data_files` or `data_dir` is specified
(available builders are "json", "csv", "parquet", "arrow", "text", "xml", "webdataset", "imagefolder", "audiofolder", "videofolder", "meshfolder")
-> load the dataset from the files in `data_files` or `data_dir`
e.g. `'parquet'`.
Use a `hf://` path like `'hf://datasets/username/dataset_name'` to allow remote only.
Use an absolute path to allow local only.
name (`str`, *optional*):
Defining the name of the dataset configuration.
data_dir (`str`, *optional*):
Defining the `data_dir` of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and `data_files` is `None`,
the behavior is equal to passing `os.path.join(data_dir, **)` as `data_files` to reference all the files in a directory.
data_files (`str` or `Sequence` or `Mapping`, *optional*):
Path(s) to source data file(s).
split (`Split` or `str`):
Which split of the data to load.
If `None`, will return a `dict` with all splits (typically `datasets.Split.TRAIN` and `datasets.Split.TEST`).
If given, will return a single Dataset.
Splits can be combined and specified like in tensorflow-datasets.
cache_dir (`str`, *optional*):
Directory to read/write data. Defaults to `"~/.cache/huggingface/datasets"`.
features (`Features`, *optional*):
Set the features type to use for this dataset.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`):
Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/...).
<Added version="2.9.1"/>
keep_in_memory (`bool`, defaults to `None`):
Whether to copy the dataset in-memory. If `None`, the dataset
will not be copied in-memory unless explicitly enabled by setting `datasets.config.IN_MEMORY_MAX_SIZE` to
nonzero. See more details in the [improve performance](../cache#improve-performance) section.
revision ([`Version`] or `str`, *optional*):
Version of the dataset to load.
As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch.
You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository.
token (`str` or `bool`, *optional*):
Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If `True`, or not specified, will get token from `"~/.huggingface"`.
streaming (`bool`, defaults to `False`):
If set to `True`, don't download the data files. Instead, it streams the data progressively while
iterating on the dataset. An [`IterableDataset`] or [`IterableDatasetDict`] is returned instead in this case.
Note that streaming works for datasets that use data formats that support being iterated over like txt, csv, jsonl for example.
Json files may be downloaded completely. Also streaming from remote zip or gzip files is supported but other compressed formats
like rar and xz are not yet supported. The tgz format doesn't allow streaming.
num_proc (`int`, *optional*, defaults to `None`):
Number of processes when downloading and generating the dataset locally.
Multiprocessing is disabled by default.
<Added version="2.7.0"/>
storage_options (`dict`, *optional*, defaults to `None`):
**Experimental**. Key/value pairs to be passed on to the dataset file-system backend, if any.
<Added version="2.11.0"/>
**config_kwargs (additional keyword arguments):
Keyword arguments to be passed to the `BuilderConfig`
and used in the [`DatasetBuilder`].
Returns:
[`Dataset`] or [`DatasetDict`]:
- if `split` is not `None`: the dataset requested,
- if `split` is `None`, a [`~datasets.DatasetDict`] with each split.
or [`IterableDataset`] or [`IterableDatasetDict`]: if `streaming=True`
- if `split` is not `None`, the dataset is requested
- if `split` is `None`, a [`~datasets.streaming.IterableDatasetDict`] with each split.
Example:
Load a dataset from the Hugging Face Hub:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('cornell-movie-review-data/rotten_tomatoes', split='train')
# Load a subset or dataset configuration (here 'sst2')
>>> from datasets import load_dataset
>>> ds = load_dataset('nyu-mll/glue', 'sst2', split='train')
# Manual mapping of data files to splits
>>> data_files = {'train': 'train.csv', 'test': 'test.csv'}
>>> ds = load_dataset('namespace/your_dataset_name', data_files=data_files)
# Manual selection of a directory to load
>>> ds = load_dataset('namespace/your_dataset_name', data_dir='folder_name')
```
Load a dataset from a Storage Bucket on the Hugging Face Hub:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('buckets/username/bucket_name/rotten_tomatoes', split='train')
```
Load a local dataset:
```py
# Load a CSV file
>>> from datasets import load_dataset
>>> ds = load_dataset('csv', data_files='path/to/local/my_dataset.csv')
# Load a JSON file
>>> from datasets import load_dataset
>>> ds = load_dataset('json', data_files='path/to/local/my_dataset.json')
```
Load an [`~datasets.IterableDataset`]:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('cornell-movie-review-data/rotten_tomatoes', split='train', streaming=True)
```
Load an image dataset with the `ImageFolder` dataset builder:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset('imagefolder', data_dir='/path/to/images', split='train')
```
datasets.load_dataset_builder
Python Library Documentation: function load_dataset_builder in module datasets.load
load_dataset_builder(path: str, name: Optional[str] = None, data_dir: Optional[str] = None, data_files: Union[str, collections.abc.Sequence[str], collections.abc.Mapping[str, Union[str, collections.abc.Sequence[str]]], NoneType] = None, cache_dir: Optional[str] = None, features: Optional[datasets.features.features.Features] = None, download_config: Optional[datasets.download.download_config.DownloadConfig] = None, download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None, revision: Union[datasets.utils.version.Version, str, NoneType] = None, token: Union[bool, str, NoneType] = None, storage_options: Optional[dict] = None, **config_kwargs) -> datasets.builder.DatasetBuilder
Load a dataset builder which can be used to:
- Inspect general information that is required to build a dataset (cache directory, config, dataset info, features, data files, etc.)
- Download and prepare the dataset as Arrow files in the cache
- Get a streaming dataset without downloading or caching anything
You can find the list of datasets on the [Hub](https://huggingface.co/datasets) or with [`huggingface_hub.list_datasets`].
A dataset is a directory that contains some data files in generic formats (JSON, CSV, Parquet, etc.) and possibly
in a generic structure (Webdataset, ImageFolder, AudioFolder, VideoFolder, MeshFolder, etc.)
Args:
path (`str`):
Path or name of the dataset.
- if `path` is a dataset repository on the HF hub (list all available datasets with [`huggingface_hub.list_datasets`])
-> load the dataset builder from supported files in the repository (csv, json, parquet, etc.)
e.g. `'username/dataset_name'`, a dataset repository on the HF hub containing the data files.
- if `path` is a directory within a Storage Bucket on the HF Hub (list your buckets with [`huggingface_hub.list_buckets`])
-> load the dataset from supported files in the directory (csv, json, parquet, etc.)
e.g. `'buckets/username/bucket_name/my_dataset'`.
- if `path` is a local directory
-> load the dataset builder from supported files in the directory (csv, json, parquet, etc.)
e.g. `'./path/to/directory/with/my/csv/data'`.
- if `path` is the name of a dataset builder and `data_files` or `data_dir` is specified
(available builders are "json", "csv", "parquet", "arrow", "text", "xml", "webdataset", "imagefolder", "audiofolder", "videofolder", "meshfolder")
-> load the dataset builder from the files in `data_files` or `data_dir`
e.g. `'parquet'`.
Use a `hf://` path like `'hf://datasets/username/dataset_name'` to allow remote only.
Use an absolute path to allow local only.
name (`str`, *optional*):
Defining the name of the dataset configuration.
data_dir (`str`, *optional*):
Defining the `data_dir` of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and `data_files` is `None`,
the behavior is equal to passing `os.path.join(data_dir, **)` as `data_files` to reference all the files in a directory.
data_files (`str` or `Sequence` or `Mapping`, *optional*):
Path(s) to source data file(s).
cache_dir (`str`, *optional*):
Directory to read/write data. Defaults to `"~/.cache/huggingface/datasets"`.
features ([`Features`], *optional*):
Set the features type to use for this dataset.
download_config ([`DownloadConfig`], *optional*):
Specific download configuration parameters.
download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`):
Download/generate mode.
revision ([`Version`] or `str`, *optional*):
Version of the dataset to load.
As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch.
You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository.
token (`str` or `bool`, *optional*):
Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
If `True`, or not specified, will get token from `"~/.huggingface"`.
storage_options (`dict`, *optional*, defaults to `None`):
**Experimental**. Key/value pairs to be passed on to the dataset file-system backend, if any.
<Added version="2.11.0"/>
**config_kwargs (additional keyword arguments):
Keyword arguments to be passed to the [`BuilderConfig`]
and used in the [`DatasetBuilder`].
Returns:
[`DatasetBuilder`]
Example:
```py
>>> from datasets import load_dataset_builder
>>> ds_builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')
>>> ds_builder.info.features
{'label': ClassLabel(names=['neg', 'pos']),
'text': Value('string')}
```
datasets.load_from_disk
Python Library Documentation: function load_from_disk in module datasets.load
load_from_disk(dataset_path: Union[str, bytes, os.PathLike], keep_in_memory: Optional[bool] = None, storage_options: Optional[dict] = None) -> Union[datasets.arrow_dataset.Dataset, datasets.dataset_dict.DatasetDict]
Loads a dataset that was previously saved using [`~Dataset.save_to_disk`] from a dataset directory, or
from a filesystem using any implementation of `fsspec.spec.AbstractFileSystem`.
Args:
dataset_path (`path-like`):
Path (e.g. `"dataset/train"`) or remote URI (e.g. `"s3://my-bucket/dataset/train"`)
of the [`Dataset`] or [`DatasetDict`] directory where the dataset/dataset-dict will be
loaded from.
keep_in_memory (`bool`, defaults to `None`):
Whether to copy the dataset in-memory. If `None`, the dataset
will not be copied in-memory unless explicitly enabled by setting `datasets.config.IN_MEMORY_MAX_SIZE` to
nonzero. See more details in the [improve performance](../cache#improve-performance) section.
storage_options (`dict`, *optional*):
Key/value pairs to be passed on to the file-system backend, if any.
<Added version="2.9.0"/>
Returns:
[`Dataset`] or [`DatasetDict`]:
- If `dataset_path` is a path of a dataset directory: the dataset requested.
- If `dataset_path` is a path of a dataset dict directory, a [`DatasetDict`] with each split.
Example:
```py
>>> from datasets import load_from_disk
>>> ds = load_from_disk('path/to/dataset/directory')
```
datasets.logging
Python Library Documentation: module datasets.utils.logging in datasets.utils
NAME
datasets.utils.logging - Logging utilities.
FUNCTIONS
disable_propagation() -> None
Disable propagation of the library log outputs.
Note that log propagation is disabled by default.
enable_propagation() -> None
Enable propagation of the library log outputs.
Please disable the Hugging Face datasets library's default handler to prevent double logging if the root logger has
been configured.
get_logger(name: Optional[str] = None) -> logging.Logger
Return a logger with the specified name.
This function can be used in dataset builders.
get_verbosity() -> int
Return the current level for the HuggingFace datasets library's root logger.
Returns:
Logging level, e.g., `datasets.logging.DEBUG` and `datasets.logging.INFO`.
> [!TIP]
> HuggingFace datasets library has following logging levels:
> - `datasets.logging.CRITICAL`, `datasets.logging.FATAL`
> - `datasets.logging.ERROR`
> - `datasets.logging.WARNING`, `datasets.logging.WARN`
> - `datasets.logging.INFO`
> - `datasets.logging.DEBUG`
set_verbosity(verbosity: int) -> None
Set the level for the Hugging Face Datasets library's root logger.
Args:
verbosity:
Logging level, e.g., `datasets.logging.DEBUG` and `datasets.logging.INFO`.
set_verbosity_debug()
Set the level for the Hugging Face datasets library's root logger to `DEBUG`.
This will display all the logging information and tqdm bars.
Shortcut to `datasets.logging.set_verbosity(datasets.logging.DEBUG)`.
set_verbosity_error()
Set the level for the Hugging Face datasets library's root logger to `ERROR`.
This will display only the errors logging information and tqdm bars.
Shortcut to `datasets.logging.set_verbosity(datasets.logging.ERROR)`.
set_verbosity_info()
Set the level for the Hugging Face datasets library's root logger to `INFO`.
This will display most of the logging information and tqdm bars.
Shortcut to `datasets.logging.set_verbosity(datasets.logging.INFO)`.
set_verbosity_warning()
Set the level for the Hugging Face datasets library's root logger to `WARNING`.
This will display only the warning and errors logging information and tqdm bars.
Shortcut to `datasets.logging.set_verbosity(datasets.logging.WARNING)`.
DATA
CRITICAL = 50
DEBUG = 10
ERROR = 40
FATAL = 50
INFO = 20
NOTSET = 0
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
WARN = 30
WARNING = 30
log_levels = {'critical': 50, 'debug': 10, 'error': 40, 'info': 20, 'w...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\logging.py
datasets.metadata
Python Library Documentation: module datasets.utils.metadata in datasets.utils
NAME
datasets.utils.metadata
CLASSES
builtins.dict(builtins.object)
MetadataConfigs
class MetadataConfigs(builtins.dict)
| Should be in format {config_name: {**config_params}}.
|
| Method resolution order:
| MetadataConfigs
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| get_default_config_name(self) -> Optional[str]
|
| to_dataset_card_data(self, dataset_card_data: huggingface_hub.repocard_data.DatasetCardData) -> None
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_dataset_card_data(dataset_card_data: huggingface_hub.repocard_data.DatasetCardData) -> 'MetadataConfigs'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| FIELD_NAME = 'configs'
|
| __annotations__ = {'FIELD_NAME': typing.ClassVar[str]}
|
| __orig_bases__ = (dict[str, dict[str, typing.Any]],)
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
DATA
ClassVar = typing.ClassVar
Special type construct to mark class variables.
An annotation wrapped in ClassVar indicates that a given
attribute is intended to be used as a class variable and
should not be set on instances of that class.
Usage::
class Starship:
stats: ClassVar[dict[str, int]] = {} # class variable
damage: int = 10 # instance variable
ClassVar accepts only types and cannot be further subscribed.
Note that ClassVar is not a class itself, and should not
be used with isinstance() or issubclass().
METADATA_CONFIGS_FIELD = 'configs'
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
known_task_ids = {'audio-classification': [], 'audio-to-audio': [], 'a...
logger = <Logger datasets.utils.metadata (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\metadata.py
datasets.naming
Python Library Documentation: module datasets.naming in datasets
NAME
datasets.naming - Utilities for file names.
FUNCTIONS
camelcase_to_snakecase(name)
Convert camel-case string to snake-case.
filename_prefix_for_name(name)
filename_prefix_for_split(name, split)
filenames_for_dataset_split(path, dataset_name, split, filetype_suffix=None, shard_lengths=None)
filepattern_for_dataset_split(dataset_name, split, data_dir, filetype_suffix=None)
snakecase_to_camelcase(name)
Convert snake-case string to camel-case string.
DATA
INVALID_WINDOWS_CHARACTERS_IN_PATH = r'<>:/\|?*'
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\naming.py
datasets.packaged_modules
Python Library Documentation: package datasets.packaged_modules in datasets
NAME
datasets.packaged_modules
PACKAGE CONTENTS
arrow (package)
audiofolder (package)
cache (package)
conll (package)
csv (package)
eval (package)
folder_based_builder (package)
generator (package)
hdf5 (package)
iceberg (package)
imagefolder (package)
json (package)
lance (package)
meshfolder (package)
niftifolder (package)
pandas (package)
parquet (package)
pdffolder (package)
spark (package)
sql (package)
text (package)
tsfile (package)
videofolder (package)
webdataset (package)
xml (package)
DATA
Dict = typing.Dict
A generic version of dict.
List = typing.List
A generic version of list.
Tuple = typing.Tuple
Deprecated alias to builtins.tuple.
Tuple[X, Y] is the cross-product type of X and Y.
Example: Tuple[T1, T2] is a tuple of two elements corresponding
to type variables T1 and T2. Tuple[int, float, str] is a tuple
of an int, a float and a string.
To specify a variable-length tuple of homogeneous type, use Tuple[T, ...].
__annotations__ = {'_EXTENSION_TO_MODULE': dict[str, tuple[str, dict]]...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\packaged_modules\__init__.py
datasets.parallel
Python Library Documentation: package datasets.parallel in datasets
NAME
datasets.parallel
PACKAGE CONTENTS
parallel
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\parallel\__init__.py
datasets.patching
Python Library Documentation: module datasets.utils.patching in datasets.utils
NAME
datasets.utils.patching
CLASSES
builtins.object
patch_submodule
class patch_submodule(builtins.object)
| patch_submodule(obj, target: str, new, attrs=None)
|
| Patch a submodule attribute of an object, by keeping all other submodules intact at all levels.
|
| Example::
|
| >>> import importlib
| >>> from datasets.load import dataset_module_factory
| >>> from datasets.streaming import patch_submodule, xjoin
| >>>
| >>> dataset_module = dataset_module_factory("stanfordnlp/snli")
| >>> snli_module = importlib.import_module(dataset_module.module_path)
| >>> patcher = patch_submodule(snli_module, "os.path.join", xjoin)
| >>> patcher.start()
| >>> assert snli_module.os.path.join is xjoin
|
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, *exc_info)
|
| __init__(self, obj, target: str, new, attrs=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| start(self)
| Activate a patch.
|
| stop(self)
| Stop an active patch.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
DATA
logger = <Logger datasets.utils.patching (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\patching.py
datasets.percent
Python Library Documentation: class PercentSlice in module datasets.splits class PercentSlice(builtins.object) | Syntactic sugar for defining slice subsplits: `datasets.percent[75:-5]`. | | See the | [guide on splits](../loading#slice-splits) | for more information. | | Data descriptors defined here: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object
datasets.py_utils
Python Library Documentation: module datasets.utils.py_utils in datasets.utils
NAME
datasets.utils.py_utils - Some python utils function and classes.
CLASSES
builtins.dict(builtins.object)
NonMutableDict
builtins.object
NestedDataStructure
builtins.property(builtins.object)
classproperty
class NestedDataStructure(builtins.object)
| NestedDataStructure(data=None)
|
| Methods defined here:
|
| __init__(self, data=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| flatten(self, data=None)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class NonMutableDict(builtins.dict)
| NonMutableDict(*args, **kwargs)
|
| Dict where keys can only be added but not modified.
|
| Will raise an error if the user try to overwrite one key. The error message
| can be customized during construction. It will be formatted using {key} for
| the overwritten key.
|
| Method resolution order:
| NonMutableDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __init__(self, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setitem__(self, key, value)
| Set self[key] to value.
|
| update(self, other)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class classproperty(builtins.property)
| classproperty(fget=None, fset=None, fdel=None, doc=None)
|
| Descriptor to be used as decorator for @classmethods.
|
| Method resolution order:
| classproperty
| builtins.property
| builtins.object
|
| Methods defined here:
|
| __get__(self, obj, objtype=None)
| Return an attribute of instance, which is of type owner.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.property:
|
| __delete__(self, instance, /)
| Delete an attribute of instance.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __set__(self, instance, value, /)
| Set an attribute of instance to value.
|
| __set_name__(...)
| Method to set name of a property.
|
| deleter(...)
| Descriptor to obtain a copy of the property with a different deleter.
|
| getter(...)
| Descriptor to obtain a copy of the property with a different getter.
|
| setter(...)
| Descriptor to obtain a copy of the property with a different setter.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.property:
|
| __new__(*args, **kwargs) class method of builtins.property
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.property:
|
| __isabstractmethod__
|
| fdel
|
| fget
|
| fset
FUNCTIONS
asdict(obj)
Convert an object to its dictionary representation recursively.
<Added version="2.4.0"/>
convert_file_size_to_int(size: Union[int, str]) -> int
Converts a size expressed as a string with digits an unit (like `"50MB"`) to an integer (in bytes).
Args:
size (`int` or `str`): The size to convert. Will be directly returned if an `int`.
Example:
```py
>>> convert_file_size_to_int("1MiB")
1048576
```
copyfunc(func)
first_non_null_non_empty_value(iterable)
Return the index and the value of the first non-null non-empty value in the iterable. If all values are None or empty, return -1 as index.
first_non_null_value(iterable)
Return the index and the value of the first non-null value in the iterable. If all values are None, return -1 as index.
glob_pattern_to_regex(pattern)
has_sufficient_disk_space(needed_bytes, directory='.')
iflatmap_unordered(pool: Union[multiprocessing.pool.Pool, multiprocess.pool.Pool], func: Callable[..., collections.abc.Iterable[~Y]], *, kwargs_iterable: collections.abc.Iterable[dict]) -> collections.abc.Iterable[~Y]
iter_batched(iterable: collections.abc.Iterable[~T], n: int) -> collections.abc.Iterable[list[~T]]
map_nested(function: Callable[[Any], Any], data_struct: Any, dict_only: bool = False, map_list: bool = True, map_tuple: bool = False, map_numpy: bool = False, num_proc: Optional[int] = None, parallel_min_length: int = 2, batched: bool = False, batch_size: Optional[int] = 1000, types: Optional[tuple] = None, disable_tqdm: bool = True, desc: Optional[str] = None) -> Any
Apply a function recursively to each element of a nested data struct.
Use multiprocessing if num_proc > 1 and the length of data_struct is greater than or equal to
`parallel_min_length`.
<Changed version="2.5.0">
Before version 2.5.0, multiprocessing was not used if `num_proc` was greater than or equal to ``len(iterable)``.
Now, if `num_proc` is greater than or equal to ``len(iterable)``, `num_proc` is set to ``len(iterable)`` and
multiprocessing is used.
</Changed>
Args:
function (`Callable`): Function to be applied to `data_struct`.
data_struct (`Any`): Data structure to apply `function` to.
dict_only (`bool`, default `False`): Whether only apply `function` recursively to `dict` values in
`data_struct`.
map_list (`bool`, default `True`): Whether also apply `function` recursively to `list` elements (besides `dict`
values).
map_tuple (`bool`, default `False`): Whether also apply `function` recursively to `tuple` elements (besides
`dict` values).
map_numpy (`bool, default `False`): Whether also apply `function` recursively to `numpy.array` elements (besides
`dict` values).
num_proc (`int`, *optional*): Number of processes.
The level in the data struct used for multiprocessing is the first level that has smaller sub-structs,
starting from the root.
parallel_min_length (`int`, default `2`): Minimum length of `data_struct` required for parallel
processing.
<Added version="2.5.0"/>
batched (`bool`, defaults to `False`):
Provide batch of items to `function`.
<Added version="2.19.0"/>
batch_size (`int`, *optional*, defaults to `1000`):
Number of items per batch provided to `function` if `batched=True`.
If `batch_size <= 0` or `batch_size == None`, provide the full iterable as a single batch to `function`.
<Added version="2.19.0"/>
types (`tuple`, *optional*): Additional types (besides `dict` values) to apply `function` recursively to their
elements.
disable_tqdm (`bool`, default `True`): Whether to disable the tqdm progressbar.
desc (`str`, *optional*): Prefix for the tqdm progressbar.
Returns:
`Any`
no_op_if_value_is_null(func)
If the value is None, return None, else call `func`.
size_str(size_in_bytes)
Returns a human readable size string.
If size_in_bytes is None, then returns "Unknown size".
For example `size_str(1.5 * datasets.units.GiB) == "1.50 GiB"`.
Args:
size_in_bytes: `int` or `None`, the size, in bytes, that we want to
format as a human-readable size string.
string_to_dict(string: str, pattern: str) -> Optional[dict[str, str]]
Un-format a string using a python f-string pattern.
From https://stackoverflow.com/a/36838374
Example::
>>> p = 'hello, my name is {name} and I am a {age} year old {what}'
>>> s = p.format(name='cody', age=18, what='quarterback')
>>> s
'hello, my name is cody and I am a 18 year old quarterback'
>>> string_to_dict(s, p)
{'age': '18', 'name': 'cody', 'what': 'quarterback'}
Args:
string (str): input string
pattern (str): pattern formatted like a python f-string
This can be a regex - so in case of un-formatting paths you should use posix paths.
Otherwise backslashes for windows paths can cause issues.
Returns:
Optional[dict[str, str]]: dictionary of variable -> value, retrieved from the input using the pattern, or
`None` if the string does not match the pattern.
temp_seed(seed: int, set_pytorch=False, set_tensorflow=False)
Temporarily set the random seed. This works for python numpy, pytorch and tensorflow.
temporary_assignment(obj, attr, value)
Temporarily assign obj.attr to value.
unique_values(values)
Iterate over iterable and return only unique values in order.
zip_dict(*dicts)
Iterate over items of dictionaries grouped by their keys.
DATA
Callable = typing.Callable
Deprecated alias to collections.abc.Callable.
Callable[[int], str] signifies a function that takes a single
parameter of type int and returns a str.
The subscription syntax must always be used with exactly two
values: the argument list and the return type.
The argument list must be a list of types, a ParamSpec,
Concatenate or ellipsis. The return type must be a single type.
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
Final = typing.Final
Special typing construct to indicate final names to type checkers.
A final name cannot be re-assigned or overridden in a subclass.
For example::
MAX_SIZE: Final = 9000
MAX_SIZE += 1 # Error reported by type checker
class Connection:
TIMEOUT: Final[int] = 10
class FastConnector(Connection):
TIMEOUT = 1 # Error reported by type checker
There is no runtime checking of these properties.
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
T = ~T
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
Y = ~Y
logger = <Logger datasets.utils.py_utils (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\py_utils.py
datasets.search
Python Library Documentation: module datasets.search in datasets
NAME
datasets.search
CLASSES
builtins.Exception(builtins.BaseException)
MissingIndex
builtins.object
BaseIndex
ElasticSearchIndex
FaissIndex
IndexableMixin
builtins.tuple(builtins.object)
BatchedNearestExamplesResults
BatchedSearchResults
NearestExamplesResults
SearchResults
class BaseIndex(builtins.object)
| Base class for indexing
|
| Methods defined here:
|
| save(self, file: Union[str, pathlib.PurePath])
| Serialize the index on disk
|
| search(self, query, k: int = 10, **kwargs) -> datasets.search.SearchResults
| To implement.
| This method has to return the scores and the indices of the retrieved examples given a certain query.
|
| search_batch(self, queries, k: int = 10, **kwargs) -> datasets.search.BatchedSearchResults
| Find the nearest examples indices to the query.
|
| Args:
| queries (`Union[List[str], np.ndarray]`): The queries as a list of strings if `column` is a text index or as a numpy array if `column` is a vector index.
| k (`int`): The number of examples to retrieve per query.
|
| Output:
| total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query.
| total_indices (`List[List[int]]`): The indices of the retrieved examples per query.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| load(file: Union[str, pathlib.PurePath]) -> 'BaseIndex'
| Deserialize the index from disk
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class BatchedNearestExamplesResults(builtins.tuple)
| BatchedNearestExamplesResults(total_scores: list[list[float]], total_examples: list[dict])
|
| BatchedNearestExamplesResults(total_scores, total_examples)
|
| Method resolution order:
| BatchedNearestExamplesResults
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __getnewargs__(self) from collections.BatchedNearestExamplesResults
| Return self as a plain tuple. Used by copy and pickle.
|
| __repr__(self) from collections.BatchedNearestExamplesResults
| Return a nicely formatted representation string
|
| _asdict(self) from collections.BatchedNearestExamplesResults
| Return a new dict which maps field names to their values.
|
| _replace(self, /, **kwds) from collections.BatchedNearestExamplesResults
| Return a new BatchedNearestExamplesResults object replacing specified fields with new values
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| _make(iterable) from collections.BatchedNearestExamplesResults
| Make a new BatchedNearestExamplesResults object from a sequence or iterable
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(_cls, total_scores: list[list[float]], total_examples: list[dict]) from namedtuple_BatchedNearestExamplesResults.BatchedNearestExamplesResults
| Create new instance of BatchedNearestExamplesResults(total_scores, total_examples)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| total_scores
| Alias for field number 0
|
| total_examples
| Alias for field number 1
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'total_examples': list[dict], 'total_scores': list[...
|
| __match_args__ = ('total_scores', 'total_examples')
|
| __orig_bases__ = (<function NamedTuple>,)
|
| _field_defaults = {}
|
| _fields = ('total_scores', 'total_examples')
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class BatchedSearchResults(builtins.tuple)
| BatchedSearchResults(total_scores: list[list[float]], total_indices: list[list[int]])
|
| BatchedSearchResults(total_scores, total_indices)
|
| Method resolution order:
| BatchedSearchResults
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __getnewargs__(self) from collections.BatchedSearchResults
| Return self as a plain tuple. Used by copy and pickle.
|
| __repr__(self) from collections.BatchedSearchResults
| Return a nicely formatted representation string
|
| _asdict(self) from collections.BatchedSearchResults
| Return a new dict which maps field names to their values.
|
| _replace(self, /, **kwds) from collections.BatchedSearchResults
| Return a new BatchedSearchResults object replacing specified fields with new values
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| _make(iterable) from collections.BatchedSearchResults
| Make a new BatchedSearchResults object from a sequence or iterable
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(_cls, total_scores: list[list[float]], total_indices: list[list[int]]) from namedtuple_BatchedSearchResults.BatchedSearchResults
| Create new instance of BatchedSearchResults(total_scores, total_indices)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| total_scores
| Alias for field number 0
|
| total_indices
| Alias for field number 1
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'total_indices': list[list[int]], 'total_scores': l...
|
| __match_args__ = ('total_scores', 'total_indices')
|
| __orig_bases__ = (<function NamedTuple>,)
|
| _field_defaults = {}
|
| _fields = ('total_scores', 'total_indices')
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class ElasticSearchIndex(BaseIndex)
| ElasticSearchIndex(host: Optional[str] = None, port: Optional[int] = None, es_client: Optional[ForwardRef('Elasticsearch')] = None, es_index_name: Optional[str] = None, es_index_config: Optional[dict] = None)
|
| Sparse index using Elasticsearch. It is used to index text and run queries based on BM25 similarity.
| An Elasticsearch server needs to be accessible, and a python client is declared with
| ```
| es_client = Elasticsearch([{'host': 'localhost', 'port': '9200'}])
| ```
| for example.
|
| Method resolution order:
| ElasticSearchIndex
| BaseIndex
| builtins.object
|
| Methods defined here:
|
| __init__(self, host: Optional[str] = None, port: Optional[int] = None, es_client: Optional[ForwardRef('Elasticsearch')] = None, es_index_name: Optional[str] = None, es_index_config: Optional[dict] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| add_documents(self, documents: Union[list[str], ForwardRef('Dataset')], column: Optional[str] = None)
| Add documents to the index.
| If the documents are inside a certain column, you can specify it using the `column` argument.
|
| search(self, query: str, k=10, **kwargs) -> datasets.search.SearchResults
| Find the nearest examples indices to the query.
|
| Args:
| query (`str`): The query as a string.
| k (`int`): The number of examples to retrieve.
|
| Output:
| scores (`List[List[float]`): The retrieval scores of the retrieved examples.
| indices (`List[List[int]]`): The indices of the retrieved examples.
|
| search_batch(self, queries, k: int = 10, max_workers=10, **kwargs) -> datasets.search.BatchedSearchResults
| Find the nearest examples indices to the query.
|
| Args:
| queries (`Union[List[str], np.ndarray]`): The queries as a list of strings if `column` is a text index or as a numpy array if `column` is a vector index.
| k (`int`): The number of examples to retrieve per query.
|
| Output:
| total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query.
| total_indices (`List[List[int]]`): The indices of the retrieved examples per query.
|
| ----------------------------------------------------------------------
| Methods inherited from BaseIndex:
|
| save(self, file: Union[str, pathlib.PurePath])
| Serialize the index on disk
|
| ----------------------------------------------------------------------
| Class methods inherited from BaseIndex:
|
| load(file: Union[str, pathlib.PurePath]) -> 'BaseIndex'
| Deserialize the index from disk
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseIndex:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class FaissIndex(BaseIndex)
| FaissIndex(device: Union[int, list[int], NoneType] = None, string_factory: Optional[str] = None, metric_type: Optional[int] = None, custom_index: Optional[ForwardRef('faiss.Index')] = None)
|
| Dense index using Faiss. It is used to index vectors.
| Faiss is a library for efficient similarity search and clustering of dense vectors.
| It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.
| You can find more information about Faiss here:
| - For index types and the string factory: https://github.com/facebookresearch/faiss/wiki/The-index-factory
| - For GPU settings: https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU
|
| Method resolution order:
| FaissIndex
| BaseIndex
| builtins.object
|
| Methods defined here:
|
| __init__(self, device: Union[int, list[int], NoneType] = None, string_factory: Optional[str] = None, metric_type: Optional[int] = None, custom_index: Optional[ForwardRef('faiss.Index')] = None)
| Create a Dense index using Faiss. You can specify `device` if you want to run it on GPU (`device` must be the GPU index).
| You can find more information about Faiss here:
| - For `string factory`: https://github.com/facebookresearch/faiss/wiki/The-index-factory
|
| add_vectors(self, vectors: Union[<built-in function array>, ForwardRef('Dataset')], column: Optional[str] = None, batch_size: int = 1000, train_size: Optional[int] = None, faiss_verbose: Optional[bool] = None)
| Add vectors to the index.
| If the arrays are inside a certain column, you can specify it using the `column` argument.
|
| save(self, file: Union[str, pathlib.PurePath], storage_options: Optional[dict] = None)
| Serialize the FaissIndex on disk
|
| search(self, query: <built-in function array>, k=10, **kwargs) -> datasets.search.SearchResults
| Find the nearest examples indices to the query.
|
| Args:
| query (`np.array`): The query as a numpy array.
| k (`int`): The number of examples to retrieve.
|
| Output:
| scores (`List[List[float]`): The retrieval scores of the retrieved examples.
| indices (`List[List[int]]`): The indices of the retrieved examples.
|
| search_batch(self, queries: <built-in function array>, k=10, **kwargs) -> datasets.search.BatchedSearchResults
| Find the nearest examples indices to the queries.
|
| Args:
| queries (`np.array`): The queries as a numpy array.
| k (`int`): The number of examples to retrieve.
|
| Output:
| total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query.
| total_indices (`List[List[int]]`): The indices of the retrieved examples per query.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| load(file: Union[str, pathlib.PurePath], device: Union[int, list[int], NoneType] = None, storage_options: Optional[dict] = None) -> 'FaissIndex'
| Deserialize the FaissIndex from disk
|
| ----------------------------------------------------------------------
| Data descriptors inherited from BaseIndex:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class IndexableMixin(builtins.object)
| Add indexing features to `datasets.Dataset`
|
| Methods defined here:
|
| __getitem__(self, key)
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| __len__(self)
|
| add_elasticsearch_index(self, column: str, index_name: Optional[str] = None, host: Optional[str] = None, port: Optional[int] = None, es_client: Optional[ForwardRef('Elasticsearch')] = None, es_index_name: Optional[str] = None, es_index_config: Optional[dict] = None)
| Add a text index using ElasticSearch for fast retrieval.
|
| Args:
| column (`str`): The column of the documents to add to the index.
| index_name (Optional `str`): The index_name/identifier of the index. This is the index name that is used to call `.get_nearest` or `.search`.
| By default it corresponds to `column`.
| host (Optional `str`, defaults to localhost):
| host of where ElasticSearch is running
| port (Optional `str`, defaults to 9200):
| port of where ElasticSearch is running
| es_client (Optional `elasticsearch.Elasticsearch`):
| The elasticsearch client used to create the index if host and port are None.
| es_index_name (Optional `str`): The elasticsearch index name used to create the index.
| es_index_config (Optional `dict`):
| The configuration of the elasticsearch index.
| Default config is:
|
| Config::
|
| {
| "settings": {
| "number_of_shards": 1,
| "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},
| },
| "mappings": {
| "properties": {
| "text": {
| "type": "text",
| "analyzer": "standard",
| "similarity": "BM25"
| },
| }
| },
| }
|
| add_faiss_index(self, column: str, index_name: Optional[str] = None, device: Union[int, list[int], NoneType] = None, string_factory: Optional[str] = None, metric_type: Optional[int] = None, custom_index: Optional[ForwardRef('faiss.Index')] = None, batch_size: int = 1000, train_size: Optional[int] = None, faiss_verbose: bool = False)
| Add a dense index using Faiss for fast retrieval.
| The index is created using the vectors of the specified column.
| You can specify `device` if you want to run it on GPU (`device` must be the GPU index, see more below).
| You can find more information about Faiss here:
| - For `string factory`: https://github.com/facebookresearch/faiss/wiki/The-index-factory
|
| Args:
| column (`str`): The column of the vectors to add to the index.
| index_name (Optional `str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`.
| By default it corresponds to `column`.
| device (Optional `Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs.
| If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU.
| string_factory (Optional `str`): This is passed to the index factory of Faiss to create the index. Default index class is IndexFlatIP.
| metric_type (Optional `int`): Type of metric. Ex: `faiss.METRIC_INNER_PRODUCT` or `faiss.METRIC_L2`.
| custom_index (Optional `faiss.Index`): Custom Faiss index that you already have instantiated and configured for your needs.
| batch_size (Optional `int`): Size of the batch to use while adding vectors to the FaissIndex. Default value is 1000.
| <Added version="2.4.0"/>
| train_size (Optional `int`): If the index needs a training step, specifies how many vectors will be used to train the index.
| faiss_verbose (`bool`, defaults to False): Enable the verbosity of the Faiss index.
|
| add_faiss_index_from_external_arrays(self, external_arrays: <built-in function array>, index_name: str, device: Union[int, list[int], NoneType] = None, string_factory: Optional[str] = None, metric_type: Optional[int] = None, custom_index: Optional[ForwardRef('faiss.Index')] = None, batch_size: int = 1000, train_size: Optional[int] = None, faiss_verbose: bool = False)
| Add a dense index using Faiss for fast retrieval.
| The index is created using the vectors of `external_arrays`.
| You can specify `device` if you want to run it on GPU (`device` must be the GPU index).
| You can find more information about Faiss here:
| - For `string factory`: https://github.com/facebookresearch/faiss/wiki/The-index-factory
|
| Args:
| external_arrays (`np.array`): If you want to use arrays from outside the lib for the index, you can set `external_arrays`.
| It will use `external_arrays` to create the Faiss index instead of the arrays in the given `column`.
| index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`.
| device (Optional `Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs.
| If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU.
| string_factory (Optional `str`): This is passed to the index factory of Faiss to create the index. Default index class is IndexFlatIP.
| metric_type (Optional `int`): Type of metric. Ex: `faiss.METRIC_INNER_PRODUCT` or `faiss.METRIC_L2`.
| custom_index (Optional `faiss.Index`): Custom Faiss index that you already have instantiated and configured for your needs.
| batch_size (Optional `int`): Size of the batch to use while adding vectors to the FaissIndex. Default value is 1000.
| <Added version="2.4.0"/>
| train_size (Optional `int`): If the index needs a training step, specifies how many vectors will be used to train the index.
| faiss_verbose (`bool`, defaults to False): Enable the verbosity of the Faiss index.
|
| drop_index(self, index_name: str)
| Drop the index with the specified column.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index.
|
| get_index(self, index_name: str) -> datasets.search.BaseIndex
| List the `index_name`/identifiers of all the attached indexes.
|
| Args:
| index_name (`str`): Index name.
|
| Returns:
| [`BaseIndex`]
|
| get_nearest_examples(self, index_name: str, query: Union[str, <built-in function array>], k: int = 10, **kwargs) -> datasets.search.NearestExamplesResults
| Find the nearest examples in the dataset to the query.
|
| Args:
| index_name (`str`):
| The index_name/identifier of the index.
| query (`Union[str, np.ndarray]`):
| The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve.
|
| Returns:
| `(scores, examples)`:
| A tuple of `(scores, examples)` where:
| - **scores** (`List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples
| - **examples** (`dict`): the retrieved examples
|
| get_nearest_examples_batch(self, index_name: str, queries: Union[list[str], <built-in function array>], k: int = 10, **kwargs) -> datasets.search.BatchedNearestExamplesResults
| Find the nearest examples in the dataset to the query.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index.
| queries (`Union[List[str], np.ndarray]`):
| The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve per query.
|
| Returns:
| `(total_scores, total_examples)`:
| A tuple of `(total_scores, total_examples)` where:
| - **total_scores** (`List[List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples per query
| - **total_examples** (`List[dict]`): the retrieved examples per query
|
| is_index_initialized(self, index_name: str) -> bool
|
| list_indexes(self) -> list[str]
| List the `colindex_nameumns`/identifiers of all the attached indexes.
|
| load_elasticsearch_index(self, index_name: str, es_index_name: str, host: Optional[str] = None, port: Optional[int] = None, es_client: Optional[ForwardRef('Elasticsearch')] = None, es_index_config: Optional[dict] = None)
| Load an existing text index using ElasticSearch for fast retrieval.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index. This is the index name that is used to call `get_nearest` or `search`.
| es_index_name (`str`):
| The name of elasticsearch index to load.
| host (`str`, *optional*, defaults to `localhost`):
| Host of where ElasticSearch is running.
| port (`str`, *optional*, defaults to `9200`):
| Port of where ElasticSearch is running.
| es_client (`elasticsearch.Elasticsearch`, *optional*):
| The elasticsearch client used to create the index if host and port are `None`.
| es_index_config (`dict`, *optional*):
| The configuration of the elasticsearch index.
| Default config is:
| ```
| {
| "settings": {
| "number_of_shards": 1,
| "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},
| },
| "mappings": {
| "properties": {
| "text": {
| "type": "text",
| "analyzer": "standard",
| "similarity": "BM25"
| },
| }
| },
| }
| ```
|
| load_faiss_index(self, index_name: str, file: Union[str, pathlib.PurePath], device: Union[int, list[int], NoneType] = None, storage_options: Optional[dict] = None)
| Load a FaissIndex from disk.
|
| If you want to do additional configurations, you can have access to the faiss index object by doing
| `.get_index(index_name).faiss_index` to make it fit your needs.
|
| Args:
| index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to
| call `.get_nearest` or `.search`.
| file (`str`): The path to the serialized faiss index on disk or remote URI (e.g. `"s3://my-bucket/index.faiss"`).
| device (Optional `Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs.
| If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU.
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.11.0"/>
|
| save_faiss_index(self, index_name: str, file: Union[str, pathlib.PurePath], storage_options: Optional[dict] = None)
| Save a FaissIndex on disk.
|
| Args:
| index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`.
| file (`str`): The path to the serialized faiss index on disk or remote URI (e.g. `"s3://my-bucket/index.faiss"`).
| storage_options (`dict`, *optional*):
| Key/value pairs to be passed on to the file-system backend, if any.
|
| <Added version="2.11.0"/>
|
| search(self, index_name: str, query: Union[str, <built-in function array>], k: int = 10, **kwargs) -> datasets.search.SearchResults
| Find the nearest examples indices in the dataset to the query.
|
| Args:
| index_name (`str`):
| The name/identifier of the index.
| query (`Union[str, np.ndarray]`):
| The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve.
|
| Returns:
| `(scores, indices)`:
| A tuple of `(scores, indices)` where:
| - **scores** (`List[List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples
| - **indices** (`List[List[int]]`): the indices of the retrieved examples
|
| search_batch(self, index_name: str, queries: Union[list[str], <built-in function array>], k: int = 10, **kwargs) -> datasets.search.BatchedSearchResults
| Find the nearest examples indices in the dataset to the query.
|
| Args:
| index_name (`str`):
| The `index_name`/identifier of the index.
| queries (`Union[List[str], np.ndarray]`):
| The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index.
| k (`int`):
| The number of examples to retrieve per query.
|
| Returns:
| `(total_scores, total_indices)`:
| A tuple of `(total_scores, total_indices)` where:
| - **total_scores** (`List[List[float]`): the retrieval scores from either FAISS (`IndexFlatL2` by default) or ElasticSearch of the retrieved examples per query
| - **total_indices** (`List[List[int]]`): the indices of the retrieved examples per query
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class MissingIndex(builtins.Exception)
| Method resolution order:
| MissingIndex
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NearestExamplesResults(builtins.tuple)
| NearestExamplesResults(scores: list[float], examples: dict)
|
| NearestExamplesResults(scores, examples)
|
| Method resolution order:
| NearestExamplesResults
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __getnewargs__(self) from collections.NearestExamplesResults
| Return self as a plain tuple. Used by copy and pickle.
|
| __repr__(self) from collections.NearestExamplesResults
| Return a nicely formatted representation string
|
| _asdict(self) from collections.NearestExamplesResults
| Return a new dict which maps field names to their values.
|
| _replace(self, /, **kwds) from collections.NearestExamplesResults
| Return a new NearestExamplesResults object replacing specified fields with new values
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| _make(iterable) from collections.NearestExamplesResults
| Make a new NearestExamplesResults object from a sequence or iterable
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(_cls, scores: list[float], examples: dict) from namedtuple_NearestExamplesResults.NearestExamplesResults
| Create new instance of NearestExamplesResults(scores, examples)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| scores
| Alias for field number 0
|
| examples
| Alias for field number 1
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'examples': <class 'dict'>, 'scores': list[float]}
|
| __match_args__ = ('scores', 'examples')
|
| __orig_bases__ = (<function NamedTuple>,)
|
| _field_defaults = {}
|
| _fields = ('scores', 'examples')
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class SearchResults(builtins.tuple)
| SearchResults(scores: list[float], indices: list[int])
|
| SearchResults(scores, indices)
|
| Method resolution order:
| SearchResults
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __getnewargs__(self) from collections.SearchResults
| Return self as a plain tuple. Used by copy and pickle.
|
| __repr__(self) from collections.SearchResults
| Return a nicely formatted representation string
|
| _asdict(self) from collections.SearchResults
| Return a new dict which maps field names to their values.
|
| _replace(self, /, **kwds) from collections.SearchResults
| Return a new SearchResults object replacing specified fields with new values
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| _make(iterable) from collections.SearchResults
| Make a new SearchResults object from a sequence or iterable
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(_cls, scores: list[float], indices: list[int]) from namedtuple_SearchResults.SearchResults
| Create new instance of SearchResults(scores, indices)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| scores
| Alias for field number 0
|
| indices
| Alias for field number 1
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'indices': list[int], 'scores': list[float]}
|
| __match_args__ = ('scores', 'indices')
|
| __orig_bases__ = (<function NamedTuple>,)
|
| _field_defaults = {}
|
| _fields = ('scores', 'indices')
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
DATA
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
TYPE_CHECKING = False
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.search (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\search.py
datasets.sharding
Python Library Documentation: module datasets.utils.sharding in datasets.utils
NAME
datasets.utils.sharding
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\sharding.py
datasets.splits
Python Library Documentation: module datasets.splits in datasets
NAME
datasets.splits - Splits related API.
CLASSES
builtins.dict(builtins.object)
SplitDict
builtins.object
PercentSlice
Split
SplitBase
NamedSplit
NamedSplitAll
SplitGenerator
SplitInfo
SplitReadInstruction
SubSplitInfo
builtins.tuple(builtins.object)
SlicedSplitInfo
builtins.type(builtins.object)
PercentSliceMeta
class NamedSplit(SplitBase)
| NamedSplit(name: str)
|
| Descriptor corresponding to a named split (train, test, ...).
|
| Example:
| Each descriptor can be composed with other using addition or slice:
|
| ```py
| split = datasets.Split.TRAIN.subsplit(datasets.percent[0:25]) + datasets.Split.TEST
| ```
|
| The resulting split will correspond to 25% of the train split merged with
| 100% of the test split.
|
| A split cannot be added twice, so the following will fail:
|
| ```py
| split = (
| datasets.Split.TRAIN.subsplit(datasets.percent[:25]) +
| datasets.Split.TRAIN.subsplit(datasets.percent[75:])
| ) # Error
| split = datasets.Split.TEST + datasets.Split.ALL # Error
| ```
|
| The slices can be applied only one time. So the following are valid:
|
| ```py
| split = (
| datasets.Split.TRAIN.subsplit(datasets.percent[:25]) +
| datasets.Split.TEST.subsplit(datasets.percent[:50])
| )
| split = (datasets.Split.TRAIN + datasets.Split.TEST).subsplit(datasets.percent[:50])
| ```
|
| But this is not valid:
|
| ```py
| train = datasets.Split.TRAIN
| test = datasets.Split.TEST
| split = train.subsplit(datasets.percent[:25]).subsplit(datasets.percent[:25])
| split = (train.subsplit(datasets.percent[:25]) + test).subsplit(datasets.percent[:50])
| ```
|
| Method resolution order:
| NamedSplit
| SplitBase
| builtins.object
|
| Methods defined here:
|
| __eq__(self, other)
| Equality: datasets.Split.TRAIN == 'train'.
|
| __hash__(self)
| Return hash(self).
|
| __init__(self, name: str)
| Initialize self. See help(type(self)) for accurate signature.
|
| __lt__(self, other)
| Return self<value.
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| get_read_instruction(self, split_dict)
| Parse the descriptor tree and compile all read instructions together.
|
| Args:
| split_dict: `dict`, The `dict[split_name, SplitInfo]` of the dataset
|
| Returns:
| split_read_instruction: `SplitReadInstruction`
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Methods inherited from SplitBase:
|
| __add__(self, other)
| Merging: datasets.Split.TRAIN + datasets.Split.TEST.
|
| __ne__(self, other)
| InEquality: datasets.Split.TRAIN != 'test'.
|
| subsplit(self, arg=None, k=None, percent=None, weighted=None)
| Divides this split into subsplits.
|
| There are 3 ways to define subsplits, which correspond to the 3
| arguments `k` (get `k` even subsplits), `percent` (get a slice of the
| dataset with `datasets.percent`), and `weighted` (get subsplits with proportions
| specified by `weighted`).
|
| Example::
|
| ```
| # 50% train, 50% test
| train, test = split.subsplit(k=2)
| # 50% train, 25% test, 25% validation
| train, test, validation = split.subsplit(weighted=[2, 1, 1])
| # Extract last 20%
| subsplit = split.subsplit(datasets.percent[-20:])
| ```
|
| Warning: k and weighted will be converted into percent which mean that
| values below the percent will be rounded up or down. The final split may be
| bigger to deal with remainders. For instance:
|
| ```
| train, test, valid = split.subsplit(k=3) # 33%, 33%, 34%
| s1, s2, s3, s4 = split.subsplit(weighted=[2, 2, 1, 1]) # 33%, 33%, 16%, 18%
| ```
|
| Args:
| arg: If no kwargs are given, `arg` will be interpreted as one of
| `k`, `percent`, or `weighted` depending on the type.
| For example:
| ```
| split.subsplit(10) # Equivalent to split.subsplit(k=10)
| split.subsplit(datasets.percent[:-20]) # percent=datasets.percent[:-20]
| split.subsplit([1, 1, 2]) # weighted=[1, 1, 2]
| ```
| k: `int` If set, subdivide the split into `k` equal parts.
| percent: `datasets.percent slice`, return a single subsplit corresponding to
| a slice of the original split. For example:
| `split.subsplit(datasets.percent[-20:]) # Last 20% of the dataset`.
| weighted: `list[int]`, return a list of subsplits whose proportions match
| the normalized sum of the list. For example:
| `split.subsplit(weighted=[1, 1, 2]) # 25%, 25%, 50%`.
|
| Returns:
| A subsplit or list of subsplits extracted from this split object.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from SplitBase:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class NamedSplitAll(NamedSplit)
| Split corresponding to the union of all defined dataset splits.
|
| Method resolution order:
| NamedSplitAll
| NamedSplit
| SplitBase
| builtins.object
|
| Methods defined here:
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| get_read_instruction(self, split_dict)
| Parse the descriptor tree and compile all read instructions together.
|
| Args:
| split_dict: `dict`, The `dict[split_name, SplitInfo]` of the dataset
|
| Returns:
| split_read_instruction: `SplitReadInstruction`
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Methods inherited from NamedSplit:
|
| __eq__(self, other)
| Equality: datasets.Split.TRAIN == 'train'.
|
| __hash__(self)
| Return hash(self).
|
| __lt__(self, other)
| Return self<value.
|
| __str__(self)
| Return str(self).
|
| ----------------------------------------------------------------------
| Methods inherited from SplitBase:
|
| __add__(self, other)
| Merging: datasets.Split.TRAIN + datasets.Split.TEST.
|
| __ne__(self, other)
| InEquality: datasets.Split.TRAIN != 'test'.
|
| subsplit(self, arg=None, k=None, percent=None, weighted=None)
| Divides this split into subsplits.
|
| There are 3 ways to define subsplits, which correspond to the 3
| arguments `k` (get `k` even subsplits), `percent` (get a slice of the
| dataset with `datasets.percent`), and `weighted` (get subsplits with proportions
| specified by `weighted`).
|
| Example::
|
| ```
| # 50% train, 50% test
| train, test = split.subsplit(k=2)
| # 50% train, 25% test, 25% validation
| train, test, validation = split.subsplit(weighted=[2, 1, 1])
| # Extract last 20%
| subsplit = split.subsplit(datasets.percent[-20:])
| ```
|
| Warning: k and weighted will be converted into percent which mean that
| values below the percent will be rounded up or down. The final split may be
| bigger to deal with remainders. For instance:
|
| ```
| train, test, valid = split.subsplit(k=3) # 33%, 33%, 34%
| s1, s2, s3, s4 = split.subsplit(weighted=[2, 2, 1, 1]) # 33%, 33%, 16%, 18%
| ```
|
| Args:
| arg: If no kwargs are given, `arg` will be interpreted as one of
| `k`, `percent`, or `weighted` depending on the type.
| For example:
| ```
| split.subsplit(10) # Equivalent to split.subsplit(k=10)
| split.subsplit(datasets.percent[:-20]) # percent=datasets.percent[:-20]
| split.subsplit([1, 1, 2]) # weighted=[1, 1, 2]
| ```
| k: `int` If set, subdivide the split into `k` equal parts.
| percent: `datasets.percent slice`, return a single subsplit corresponding to
| a slice of the original split. For example:
| `split.subsplit(datasets.percent[-20:]) # Last 20% of the dataset`.
| weighted: `list[int]`, return a list of subsplits whose proportions match
| the normalized sum of the list. For example:
| `split.subsplit(weighted=[1, 1, 2]) # 25%, 25%, 50%`.
|
| Returns:
| A subsplit or list of subsplits extracted from this split object.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from SplitBase:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class PercentSlice(builtins.object)
| Syntactic sugar for defining slice subsplits: `datasets.percent[75:-5]`.
|
| See the
| [guide on splits](../loading#slice-splits)
| for more information.
|
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class PercentSliceMeta(builtins.type)
| # 2 requirements:
| # 1. datasets.percent be sliceable
| # 2. datasets.percent be documented
| #
| # Instances are not documented, so we want datasets.percent to be a class, but to
| # have it be sliceable, we need this metaclass.
|
| Method resolution order:
| PercentSliceMeta
| builtins.type
| builtins.object
|
| Methods defined here:
|
| __getitem__(cls, slice_value)
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.type:
|
| __call__(self, /, *args, **kwargs)
| Call self as a function.
|
| __delattr__(self, name, /)
| Implement delattr(self, name).
|
| __dir__(self, /)
| Specialized __dir__ implementation for types.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __instancecheck__(self, instance, /)
| Check if an object is an instance.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __ror__(self, value, /)
| Return value|self.
|
| __setattr__(self, name, value, /)
| Implement setattr(self, name, value).
|
| __sizeof__(self, /)
| Return memory consumption of the type object.
|
| __subclasscheck__(self, subclass, /)
| Check if a class is a subclass.
|
| __subclasses__(self, /)
| Return a list of immediate subclasses.
|
| mro(self, /)
| Return a type's method resolution order.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.type:
|
| __prepare__(...)
| __prepare__() -> dict
| used to create the namespace for the class statement
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.type:
|
| __new__(*args, **kwargs) class method of builtins.type
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.type:
|
| __abstractmethods__
|
| __dict__
|
| __text_signature__
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.type:
|
| __base__ = <class 'type'>
| type(object) -> the object's type
| type(name, bases, dict, **kwds) -> a new type
|
|
| __bases__ = (<class 'type'>,)
|
| __basicsize__ = 920
|
| __dictoffset__ = 264
|
| __flags__ = 2156420608
|
| __itemsize__ = 40
|
| __mro__ = (<class 'datasets.splits.PercentSliceMeta'>, <class 'type'>,...
|
| __type_params__ = ()
|
| __weakrefoffset__ = 368
class SlicedSplitInfo(builtins.tuple)
| SlicedSplitInfo(split_info, slice_value)
|
| SlicedSplitInfo(split_info, slice_value)
|
| Method resolution order:
| SlicedSplitInfo
| builtins.tuple
| builtins.object
|
| Methods defined here:
|
| __getnewargs__(self) from collections.SlicedSplitInfo
| Return self as a plain tuple. Used by copy and pickle.
|
| __repr__(self) from collections.SlicedSplitInfo
| Return a nicely formatted representation string
|
| _asdict(self) from collections.SlicedSplitInfo
| Return a new dict which maps field names to their values.
|
| _replace(self, /, **kwds) from collections.SlicedSplitInfo
| Return a new SlicedSplitInfo object replacing specified fields with new values
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| _make(iterable) from collections.SlicedSplitInfo
| Make a new SlicedSplitInfo object from a sequence or iterable
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| __new__(_cls, split_info, slice_value) from namedtuple_SlicedSplitInfo.SlicedSplitInfo
| Create new instance of SlicedSplitInfo(split_info, slice_value)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| split_info
| Alias for field number 0
|
| slice_value
| Alias for field number 1
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __match_args__ = ('split_info', 'slice_value')
|
| _field_defaults = {}
|
| _fields = ('split_info', 'slice_value')
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.tuple:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmul__(self, value, /)
| Return value*self.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.tuple:
|
| __class_getitem__(...)
| See PEP 585
class Split(builtins.object)
| Split(name)
|
| `Enum` for dataset splits.
|
| Datasets are typically split into different subsets to be used at various
| stages of training and evaluation.
|
| - `TRAIN`: the training data.
| - `VALIDATION`: the validation data. If present, this is typically used as
| evaluation data while iterating on a model (e.g. changing hyperparameters,
| model architecture, etc.).
| - `TEST`: the testing data. This is the data to report metrics on. Typically
| you do not want to use this during model iteration as you may overfit to it.
| - `ALL`: the union of all defined dataset splits.
|
| All splits, including compositions inherit from `datasets.SplitBase`.
|
| See the [guide](../load_hub#splits) on splits for more information.
|
| Example:
|
| ```py
| >>> datasets.SplitGenerator(
| ... name=datasets.Split.TRAIN,
| ... gen_kwargs={"split_key": "train", "files": dl_manager.download_and extract(url)},
| ... ),
| ... datasets.SplitGenerator(
| ... name=datasets.Split.VALIDATION,
| ... gen_kwargs={"split_key": "validation", "files": dl_manager.download_and extract(url)},
| ... ),
| ... datasets.SplitGenerator(
| ... name=datasets.Split.TEST,
| ... gen_kwargs={"split_key": "test", "files": dl_manager.download_and extract(url)},
| ... )
| ```
|
| Static methods defined here:
|
| __new__(cls, name)
| Create a custom split with datasets.Split('custom_name').
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| ALL = NamedSplitAll()
|
| TEST = NamedSplit('test')
|
| TRAIN = NamedSplit('train')
|
| VALIDATION = NamedSplit('validation')
class SplitBase(builtins.object)
| Abstract base class for Split compositionality.
|
| See the
| [guide on splits](../loading#slice-splits)
| for more information.
|
| There are three parts to the composition:
| 1) The splits are composed (defined, merged, split,...) together before
| calling the `.as_dataset()` function. This is done with the `__add__`,
| `__getitem__`, which return a tree of `SplitBase` (whose leaf
| are the `NamedSplit` objects)
|
| ```
| split = datasets.Split.TRAIN + datasets.Split.TEST.subsplit(datasets.percent[:50])
| ```
|
| 2) The `SplitBase` is forwarded to the `.as_dataset()` function
| to be resolved into actual read instruction. This is done by the
| `.get_read_instruction()` method which takes the real dataset splits
| (name, number of shards,...) and parse the tree to return a
| `SplitReadInstruction()` object
|
| ```
| read_instruction = split.get_read_instruction(self.info.splits)
| ```
|
| 3) The `SplitReadInstruction` is then used in the `tf.data.Dataset` pipeline
| to define which files to read and how to skip examples within file.
|
| Methods defined here:
|
| __add__(self, other)
| Merging: datasets.Split.TRAIN + datasets.Split.TEST.
|
| __eq__(self, other)
| Equality: datasets.Split.TRAIN == 'train'.
|
| __ne__(self, other)
| InEquality: datasets.Split.TRAIN != 'test'.
|
| get_read_instruction(self, split_dict)
| Parse the descriptor tree and compile all read instructions together.
|
| Args:
| split_dict: `dict`, The `dict[split_name, SplitInfo]` of the dataset
|
| Returns:
| split_read_instruction: `SplitReadInstruction`
|
| subsplit(self, arg=None, k=None, percent=None, weighted=None)
| Divides this split into subsplits.
|
| There are 3 ways to define subsplits, which correspond to the 3
| arguments `k` (get `k` even subsplits), `percent` (get a slice of the
| dataset with `datasets.percent`), and `weighted` (get subsplits with proportions
| specified by `weighted`).
|
| Example::
|
| ```
| # 50% train, 50% test
| train, test = split.subsplit(k=2)
| # 50% train, 25% test, 25% validation
| train, test, validation = split.subsplit(weighted=[2, 1, 1])
| # Extract last 20%
| subsplit = split.subsplit(datasets.percent[-20:])
| ```
|
| Warning: k and weighted will be converted into percent which mean that
| values below the percent will be rounded up or down. The final split may be
| bigger to deal with remainders. For instance:
|
| ```
| train, test, valid = split.subsplit(k=3) # 33%, 33%, 34%
| s1, s2, s3, s4 = split.subsplit(weighted=[2, 2, 1, 1]) # 33%, 33%, 16%, 18%
| ```
|
| Args:
| arg: If no kwargs are given, `arg` will be interpreted as one of
| `k`, `percent`, or `weighted` depending on the type.
| For example:
| ```
| split.subsplit(10) # Equivalent to split.subsplit(k=10)
| split.subsplit(datasets.percent[:-20]) # percent=datasets.percent[:-20]
| split.subsplit([1, 1, 2]) # weighted=[1, 1, 2]
| ```
| k: `int` If set, subdivide the split into `k` equal parts.
| percent: `datasets.percent slice`, return a single subsplit corresponding to
| a slice of the original split. For example:
| `split.subsplit(datasets.percent[-20:]) # Last 20% of the dataset`.
| weighted: `list[int]`, return a list of subsplits whose proportions match
| the normalized sum of the list. For example:
| `split.subsplit(weighted=[1, 1, 2]) # 25%, 25%, 50%`.
|
| Returns:
| A subsplit or list of subsplits extracted from this split object.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset({'get_read_instruction'})
|
| __hash__ = None
class SplitDict(builtins.dict)
| SplitDict(*args, dataset_name=None, **kwargs)
|
| Split info object.
|
| Method resolution order:
| SplitDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __getitem__(self, key: Union[datasets.splits.SplitBase, str])
| Return self[key].
|
| __init__(self, *args, dataset_name=None, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setitem__(self, key: Union[datasets.splits.SplitBase, str], value: datasets.splits.SplitInfo)
| Set self[key] to value.
|
| add(self, split_info: datasets.splits.SplitInfo)
| Add the split info.
|
| copy(self)
| D.copy() -> a shallow copy of D
|
| to_split_dict(self)
| Returns a list of SplitInfo protos that we have.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_split_dict(split_infos: Union[list, dict], dataset_name: Optional[str] = None)
| Returns a new SplitDict initialized from a Dict or List of `split_infos`.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| total_num_examples
| Return the total number of examples.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (dict[str, datasets.splits.SplitInfo],)
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __gt__(self, value, /)
| Return self>value.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class SplitGenerator(builtins.object)
| SplitGenerator(name: str, gen_kwargs: dict = <factory>) -> None
|
| Defines the split information for the generator.
|
| This should be used as returned value of
| `GeneratorBasedBuilder._split_generators`.
| See `GeneratorBasedBuilder._split_generators` for more info and example
| of usage.
|
| Args:
| name (`str`):
| Name of the `Split` for which the generator will
| create the examples.
| **gen_kwargs (additional keyword arguments):
| Keyword arguments to forward to the `DatasetBuilder._generate_examples` method
| of the builder.
|
| Example:
|
| ```py
| >>> datasets.SplitGenerator(
| ... name=datasets.Split.TRAIN,
| ... gen_kwargs={"split_key": "train", "files": dl_manager.download_and_extract(url)},
| ... )
| ```
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, name: str, gen_kwargs: dict = <factory>) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'gen_kwargs': <class 'dict'>, 'name': <class 'str'>...
|
| __dataclass_fields__ = {'gen_kwargs': Field(name='gen_kwargs',type=<cl...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('name', 'gen_kwargs')
class SplitInfo(builtins.object)
| SplitInfo(name: str = '', num_bytes: int = 0, num_examples: int = 0, shard_lengths: Optional[list[int]] = None, original_shard_lengths: Optional[list[int]] = None, dataset_name: Optional[str] = None) -> None
|
| SplitInfo(name: str = '', num_bytes: int = 0, num_examples: int = 0, shard_lengths: Optional[list[int]] = None, original_shard_lengths: Optional[list[int]] = None, dataset_name: Optional[str] = None)
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, name: str = '', num_bytes: int = 0, num_examples: int = 0, shard_lengths: Optional[list[int]] = None, original_shard_lengths: Optional[list[int]] = None, dataset_name: Optional[str] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| file_instructions
| Returns the list of dict(filename, take, skip).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'dataset_name': typing.Optional[str], 'name': <clas...
|
| __dataclass_fields__ = {'dataset_name': Field(name='dataset_name',type...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('name', 'num_bytes', 'num_examples', 'shard_lengths'...
|
| dataset_name = None
|
| name = ''
|
| num_bytes = 0
|
| num_examples = 0
|
| original_shard_lengths = None
|
| shard_lengths = None
class SplitReadInstruction(builtins.object)
| SplitReadInstruction(split_info=None)
|
| Object containing the reading instruction for the dataset.
|
| Similarly to `SplitDescriptor` nodes, this object can be composed with itself,
| but the resolution happens instantaneously, instead of keeping track of the
| tree, such as all instructions are compiled and flattened in a single
| SplitReadInstruction object containing the list of files and slice to use.
|
| Once resolved, the instructions can be accessed with:
|
| ```
| read_instructions.get_list_sliced_split_info() # List of splits to use
| ```
|
| Methods defined here:
|
| __add__(self, other)
| Merging split together.
|
| __getitem__(self, slice_value)
| Sub-splits.
|
| __init__(self, split_info=None)
| Initialize self. See help(type(self)) for accurate signature.
|
| add(self, sliced_split)
| Add a SlicedSplitInfo the read instructions.
|
| get_list_sliced_split_info(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class SubSplitInfo(builtins.object)
| SubSplitInfo(instructions: datasets.arrow_reader.FileInstructions) -> None
|
| Wrapper around a sub split info.
| This class exposes info on the subsplit:
| ```
| ds, info = datasets.load_dataset(..., split='train[75%:]', with_info=True)
| info.splits['train[75%:]'].num_examples
| ```
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, instructions: datasets.arrow_reader.FileInstructions) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| file_instructions
| Returns the list of dict(filename, take, skip).
|
| num_examples
| Returns the number of examples in the subsplit.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'instructions': <class 'datasets.arrow_reader.FileI...
|
| __dataclass_fields__ = {'instructions': Field(name='instructions',type...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('instructions',)
percent = class PercentSlice(builtins.object)
| Syntactic sugar for defining slice subsplits: `datasets.percent[75:-5]`.
|
| See the
| [guide on splits](../loading#slice-splits)
| for more information.
|
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
DATA
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\splits.py
datasets.stratify
Python Library Documentation: module datasets.utils.stratify in datasets.utils
NAME
datasets.utils.stratify
FUNCTIONS
approximate_mode(class_counts, n_draws, rng)
Computes approximate mode of multivariate hypergeometric.
This is an approximation to the mode of the multivariate
hypergeometric given by class_counts and n_draws.
It shouldn't be off by more than one.
It is the mostly likely outcome of drawing n_draws many
samples from the population given by class_counts.
Args
----------
class_counts : ndarray of int
Population per class.
n_draws : int
Number of draws (samples to draw) from the overall population.
rng : random state
Used to break ties.
Returns
-------
sampled_classes : ndarray of int
Number of samples drawn from each class.
np.sum(sampled_classes) == n_draws
stratified_shuffle_split_generate_indices(y, n_train, n_test, rng, n_splits=10)
Provides train/test indices to split data in train/test sets.
It's reference is taken from StratifiedShuffleSplit implementation
of scikit-learn library.
Args
----------
n_train : int,
represents the absolute number of train samples.
n_test : int,
represents the absolute number of test samples.
random_state : int or RandomState instance, default=None
Controls the randomness of the training and testing indices produced.
Pass an int for reproducible output across multiple function calls.
n_splits : int, default=10
Number of re-shuffling & splitting iterations.
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\stratify.py
datasets.streaming
Python Library Documentation: module datasets.streaming in datasets
NAME
datasets.streaming
FUNCTIONS
extend_dataset_builder_for_streaming(builder: 'DatasetBuilder')
Extend the dataset builder module and the modules imported by it to support streaming.
Args:
builder (:class:`DatasetBuilder`): Dataset builder instance.
extend_module_for_streaming(module_path, download_config: Optional[datasets.download.download_config.DownloadConfig] = None)
Extend the module to support streaming.
We patch some functions in the module to use `fsspec` to support data streaming:
- We use `fsspec.open` to open and read remote files. We patch the module function:
- `open`
- We use the "::" hop separator to join paths and navigate remote compressed/archive files. We patch the module
functions:
- `os.path.join`
- `pathlib.Path.joinpath` and `pathlib.Path.__truediv__` (called when using the "/" operator)
The patched functions are replaced with custom functions defined to work with the
:class:`~download.streaming_download_manager.StreamingDownloadManager`.
Args:
module_path: Path to the module to be extended.
download_config: Mainly use `token` or `storage_options` to support different platforms and auth types.
DATA
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
TYPE_CHECKING = False
logger = <Logger datasets.streaming (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\streaming.py
datasets.table
Python Library Documentation: module datasets.table in datasets
NAME
datasets.table
CLASSES
builtins.ValueError(builtins.Exception)
CastError
builtins.object
IndexedTableMixin
Table
ConcatenationTable
TableBlock
InMemoryTable
MemoryMappedTable
class CastError(builtins.ValueError)
| CastError(*args, table_column_names: list[str], requested_column_names: list[str]) -> None
|
| When it's not possible to cast an Arrow table to a specific schema or set of features
|
| Method resolution order:
| CastError
| builtins.ValueError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Methods defined here:
|
| __init__(self, *args, table_column_names: list[str], requested_column_names: list[str]) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __reduce__(self)
| Helper for pickle.
|
| details(self)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.ValueError:
|
| __new__(*args, **kwargs) class method of builtins.ValueError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ConcatenationTable(Table)
| ConcatenationTable(table: pyarrow.lib.Table, blocks: list[list[datasets.table.TableBlock]])
|
| The table comes from the concatenation of several tables called blocks.
| It enables concatenation on both axis 0 (append rows) and axis 1 (append columns).
|
| The underlying tables are called "blocks" and can be either `InMemoryTable`
| or `MemoryMappedTable` objects.
| This allows to combine tables that come from memory or that are memory mapped.
| When a `ConcatenationTable` is pickled, then each block is pickled:
| - the `InMemoryTable` objects are pickled by copying all the data in memory.
| - the MemoryMappedTable objects are pickled without copying the data into memory.
| Instead, only the path to the memory mapped arrow file is pickled, as well as the list
| of transforms to "replays" when reloading the table from the disk.
|
| Its implementation requires to store each block separately.
| The `blocks` attributes stores a list of list of blocks.
| The first axis concatenates the tables along the axis 0 (it appends rows),
| while the second axis concatenates tables along the axis 1 (it appends columns).
|
| If some columns are missing when concatenating on axis 0, they are filled with null values.
| This is done using `pyarrow.concat_tables(tables, promote=True)`.
|
| You can access the fully combined table by accessing the `ConcatenationTable.table` attribute,
| and the blocks by accessing the `ConcatenationTable.blocks` attribute.
|
| Method resolution order:
| ConcatenationTable
| Table
| IndexedTableMixin
| builtins.object
|
| Methods defined here:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, table: pyarrow.lib.Table, blocks: list[list[datasets.table.TableBlock]])
| Initialize self. See help(type(self)) for accurate signature.
|
| __setstate__(self, state)
|
| add_column(self, *args, **kwargs)
| Add column to Table at position.
|
| A new table is returned with the column added, the original table
| object is left unchanged.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`: New table with the passed column added.
|
| append_column(self, *args, **kwargs)
| Append column at end of columns.
|
| Args:
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`:
| New table with the passed column added.
|
| cast(self, target_schema, *args, **kwargs)
| Cast table values to another schema.
|
| Args:
| target_schema (`Schema`):
| Schema to cast to, the names and order of fields must match.
| safe (`bool`, defaults to `True`):
| Check for overflows or other unsafe conversions.
|
| Returns:
| `datasets.table.Table`
|
| combine_chunks(self, *args, **kwargs)
| Make a new table by combining the chunks this table has.
|
| All the underlying chunks in the `ChunkedArray` of each column are
| concatenated into zero or one chunk.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| drop(self, columns, *args, **kwargs)
| Drop one or more columns and return a new table.
|
| Args:
| columns (`List[str]`):
| List of field names referencing existing columns.
|
| Raises:
| `KeyError` : if any of the passed columns name are not existing.
|
| Returns:
| `datasets.table.Table`:
| New table without the columns.
|
| filter(self, mask, *args, **kwargs)
| Select records from a Table. See `pyarrow.compute.filter` for full usage.
|
| flatten(self, *args, **kwargs)
| Flatten this Table. Each column with a struct type is flattened
| into one column per struct field. Other columns are left unchanged.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| remove_column(self, i, *args, **kwargs)
| Create new Table with the indicated column removed.
|
| Args:
| i (`int`):
| Index of column to remove.
|
| Returns:
| `datasets.table.Table`:
| New table without the column.
|
| rename_columns(self, names, *args, **kwargs)
| Create new table with columns renamed to provided names.
|
| replace_schema_metadata(self, *args, **kwargs)
| EXPERIMENTAL: Create shallow copy of table by replacing schema
| key-value metadata with the indicated new metadata (which may be `None`,
| which deletes any existing metadata).
|
| Args:
| metadata (`dict`, defaults to `None`):
|
| Returns:
| `datasets.table.Table`: shallow_copy
|
| select(self, columns, *args, **kwargs)
| Select columns of the table.
|
| Returns a new table with the specified columns, and metadata preserved.
|
| Args:
| columns (:obj:`Union[List[str], List[int]]`):
| The column names or integer indices to select.
|
| Returns:
| :class:`datasets.table.Table`: New table with the specified columns, and metadata preserved.
|
| set_column(self, *args, **kwargs)
| Replace column in Table at position.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`:
| New table with the passed column set.
|
| slice(self, offset=0, length=None)
| Compute zero-copy slice of this Table.
|
| Args:
| offset (`int`, defaults to `0`):
| Offset from start of table to slice.
| length (`int`, defaults to `None`):
| Length of slice (default is until end of table starting from
| offset).
|
| Returns:
| `datasets.table.Table`
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_blocks(blocks: ~TableBlockContainer) -> 'ConcatenationTable'
|
| from_tables(tables: list[typing.Union[pyarrow.lib.Table, datasets.table.Table]], axis: int = 0) -> 'ConcatenationTable'
| Create `ConcatenationTable` from list of tables.
|
| Args:
| tables (list of `Table` or list of `pyarrow.Table`):
| List of tables.
| axis (`{0, 1}`, defaults to `0`, meaning over rows):
| Axis to concatenate over, where `0` means over rows (vertically) and `1` means over columns
| (horizontally).
|
| <Added version="1.6.0"/>
|
| ----------------------------------------------------------------------
| Methods inherited from Table:
|
| __deepcopy__(self, memo: dict)
|
| __eq__(self, other)
| Return self==value.
|
| __getitem__(self, i)
|
| __len__(self)
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| column(self, *args, **kwargs)
| Select a column by its column name, or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the column to retrieve.
|
| Returns:
| `pyarrow.ChunkedArray`
|
| equals(self, *args, **kwargs)
| Check if contents of two tables are equal.
|
| Args:
| other ([`~datasets.table.Table`]):
| Table to compare against.
| check_metadata `bool`, defaults to `False`):
| Whether schema metadata equality should be checked as well.
|
| Returns:
| `bool`
|
| field(self, *args, **kwargs)
| Select a schema field by its column name or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the field to retrieve.
|
| Returns:
| `pyarrow.Field`
|
| itercolumns(self, *args, **kwargs)
| Iterator over all columns in their numerical order.
|
| Yields:
| `pyarrow.ChunkedArray`
|
| to_batches(self, *args, **kwargs)
| Convert Table to list of (contiguous) `RecordBatch` objects.
|
| Args:
| max_chunksize (`int`, defaults to `None`):
| Maximum size for `RecordBatch` chunks. Individual chunks may be
| smaller depending on the chunk layout of individual columns.
|
| Returns:
| `List[pyarrow.RecordBatch]`
|
| to_pandas(self, *args, **kwargs)
| Convert to a pandas-compatible NumPy array or DataFrame, as appropriate.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| Arrow MemoryPool to use for allocations. Uses the default memory
| pool is not passed.
| strings_to_categorical (`bool`, defaults to `False`):
| Encode string (UTF8) and binary types to `pandas.Categorical`.
| categories (`list`, defaults to `empty`):
| List of fields that should be returned as `pandas.Categorical`. Only
| applies to table-like data structures.
| zero_copy_only (`bool`, defaults to `False`):
| Raise an `ArrowException` if this function call would require copying
| the underlying data.
| integer_object_nulls (`bool`, defaults to `False`):
| Cast integers with nulls to objects.
| date_as_object (`bool`, defaults to `True`):
| Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype.
| timestamp_as_object (`bool`, defaults to `False`):
| Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is
| useful if you have timestamps that don't fit in the normal date
| range of nanosecond timestamps (1678 CE-2262 CE).
| If `False`, all timestamps are converted to `datetime64[ns]` dtype.
| use_threads (`bool`, defaults to `True`):
| Whether to parallelize the conversion using multiple threads.
| deduplicate_objects (`bool`, defaults to `False`):
| Do not create multiple copies Python objects when created, to save
| on memory use. Conversion will be slower.
| ignore_metadata (`bool`, defaults to `False`):
| If `True`, do not use the 'pandas' metadata to reconstruct the
| DataFrame index, if present.
| safe (`bool`, defaults to `True`):
| For certain data types, a cast is needed in order to store the
| data in a pandas DataFrame or Series (e.g. timestamps are always
| stored as nanoseconds in pandas). This option controls whether it
| is a safe cast or not.
| split_blocks (`bool`, defaults to `False`):
| If `True`, generate one internal "block" for each column when
| creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this
| can temporarily reduce memory note that various pandas operations
| can trigger "consolidation" which may balloon memory use.
| self_destruct (`bool`, defaults to `False`):
| EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow
| memory while converting the Arrow object to pandas. If you use the
| object after calling `to_pandas` with this option it will crash your
| program.
| types_mapper (`function`, defaults to `None`):
| A function mapping a pyarrow DataType to a pandas `ExtensionDtype`.
| This can be used to override the default pandas type for conversion
| of built-in pyarrow types or in absence of `pandas_metadata` in the
| Table schema. The function receives a pyarrow DataType and is
| expected to return a pandas `ExtensionDtype` or `None` if the
| default conversion should be used for that type. If you have
| a dictionary mapping, you can pass `dict.get` as function.
|
| Returns:
| `pandas.Series` or `pandas.DataFrame`: `pandas.Series` or `pandas.DataFrame` depending on type of object
|
| to_pydict(self, *args, **kwargs)
| Convert the Table to a `dict` or `OrderedDict`.
|
| Returns:
| `dict`
|
| to_pylist(self, *args, **kwargs)
| Convert the Table to a list
|
| Returns:
| `list`
|
| to_reader(self, max_chunksize: Optional[int] = None)
| Convert the Table to a RecordBatchReader.
|
| Note that this method is zero-copy, it merely exposes the same data under a different API.
|
| Args:
| max_chunksize (`int`, defaults to `None`)
| Maximum size for RecordBatch chunks. Individual chunks may be smaller depending
| on the chunk layout of individual columns.
|
| Returns:
| `pyarrow.RecordBatchReader`
|
| to_string(self, *args, **kwargs)
|
| validate(self, *args, **kwargs)
| Perform validation checks. An exception is raised if validation fails.
|
| By default only cheap validation checks are run. Pass `full=True`
| for thorough validation checks (potentially `O(n)`).
|
| Args:
| full (`bool`, defaults to `False`):
| If `True`, run expensive checks, otherwise cheap checks only.
|
| Raises:
| `pa.lib.ArrowInvalid`: if validation fails
|
| ----------------------------------------------------------------------
| Readonly properties inherited from Table:
|
| column_names
| Names of the table's columns.
|
| columns
| List of all columns in numerical order.
|
| Returns:
| `List[pa.ChunkedArray]`
|
| nbytes
| Total number of bytes consumed by the elements of the table.
|
| num_columns
| Number of columns in this table.
|
| Returns:
| int
|
| num_rows
| Number of rows in this table.
|
| Due to the definition of a table, all columns have the same number of
| rows.
|
| Returns:
| int
|
| schema
| Schema of the table and its columns.
|
| Returns:
| `pyarrow.Schema`
|
| shape
| Dimensions of the table: (#rows, #columns).
|
| Returns:
| `(int, int)`: Number of rows and number of columns.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from Table:
|
| __hash__ = None
|
| ----------------------------------------------------------------------
| Methods inherited from IndexedTableMixin:
|
| fast_gather(self, indices: Union[list[int], numpy.ndarray]) -> pyarrow.lib.Table
| Create a pa.Table by gathering the records at the records at the specified indices. Should be faster
| than pa.concat_tables(table.fast_slice(int(i) % table.num_rows, 1) for i in indices) since NumPy can compute
| the binary searches in parallel, highly optimized C
|
| fast_slice(self, offset=0, length=None) -> pyarrow.lib.Table
| Slice the Table using interpolation search.
| The behavior is the same as `pyarrow.Table.slice` but it's significantly faster.
|
| Interpolation search is used to find the start and end indexes of the batches we want to keep.
| The batches to keep are then concatenated to form the sliced Table.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from IndexedTableMixin:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class InMemoryTable(TableBlock)
| InMemoryTable(table: pyarrow.lib.Table)
|
| The table is said in-memory when it is loaded into the user's RAM.
|
| Pickling it does copy all the data using memory.
| Its implementation is simple and uses the underlying pyarrow Table methods directly.
|
| This is different from the `MemoryMapped` table, for which pickling doesn't copy all the
| data in memory. For a `MemoryMapped`, unpickling instead reloads the table from the disk.
|
| `InMemoryTable` must be used when data fit in memory, while `MemoryMapped` are reserved for
| data bigger than memory or when you want the memory footprint of your application to
| stay low.
|
| Method resolution order:
| InMemoryTable
| TableBlock
| Table
| IndexedTableMixin
| builtins.object
|
| Methods defined here:
|
| __getstate__(self)
| Helper for pickle.
|
| __setstate__(self, state)
|
| add_column(self, *args, **kwargs)
| Add column to Table at position.
|
| A new table is returned with the column added, the original table
| object is left unchanged.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`: New table with the passed column added.
|
| append_column(self, *args, **kwargs)
| Append column at end of columns.
|
| Args:
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`:
| New table with the passed column added.
|
| cast(self, *args, **kwargs)
| Cast table values to another schema.
|
| Args:
| target_schema (`Schema`):
| Schema to cast to, the names and order of fields must match.
| safe (`bool`, defaults to `True`):
| Check for overflows or other unsafe conversions.
|
| Returns:
| `datasets.table.Table`
|
| combine_chunks(self, *args, **kwargs)
| Make a new table by combining the chunks this table has.
|
| All the underlying chunks in the `ChunkedArray` of each column are
| concatenated into zero or one chunk.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| drop(self, *args, **kwargs)
| Drop one or more columns and return a new table.
|
| Args:
| columns (`List[str]`):
| List of field names referencing existing columns.
|
| Raises:
| `KeyError` : if any of the passed columns name are not existing.
|
| Returns:
| `datasets.table.Table`:
| New table without the columns.
|
| filter(self, *args, **kwargs)
| Select records from a Table. See `pyarrow.compute.filter` for full usage.
|
| flatten(self, *args, **kwargs)
| Flatten this Table. Each column with a struct type is flattened
| into one column per struct field. Other columns are left unchanged.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| remove_column(self, *args, **kwargs)
| Create new Table with the indicated column removed.
|
| Args:
| i (`int`):
| Index of column to remove.
|
| Returns:
| `datasets.table.Table`:
| New table without the column.
|
| rename_columns(self, *args, **kwargs)
| Create new table with columns renamed to provided names.
|
| replace_schema_metadata(self, *args, **kwargs)
| EXPERIMENTAL: Create shallow copy of table by replacing schema
| key-value metadata with the indicated new metadata (which may be `None`,
| which deletes any existing metadata).
|
| Args:
| metadata (`dict`, defaults to `None`):
|
| Returns:
| `datasets.table.Table`: shallow_copy
|
| select(self, *args, **kwargs)
| Select columns of the table.
|
| Returns a new table with the specified columns, and metadata preserved.
|
| Args:
| columns (:obj:`Union[List[str], List[int]]`):
| The column names or integer indices to select.
|
| Returns:
| :class:`datasets.table.Table`: New table with the specified columns, and metadata preserved.
|
| set_column(self, *args, **kwargs)
| Replace column in Table at position.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`:
| New table with the passed column set.
|
| slice(self, offset=0, length=None)
| Compute zero-copy slice of this Table.
|
| Args:
| offset (`int`, defaults to `0`):
| Offset from start of table to slice.
| length (`int`, defaults to `None`):
| Length of slice (default is until end of table starting from
| offset).
|
| Returns:
| `datasets.table.Table`
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_arrays(*args, **kwargs)
| Construct a Table from Arrow arrays.
|
| Args:
| arrays (`List[Union[pyarrow.Array, pyarrow.ChunkedArray]]`):
| Equal-length arrays that should form the table.
| names (`List[str]`, *optional*):
| Names for the table columns. If not passed, schema must be passed.
| schema (`Schema`, defaults to `None`):
| Schema for the created table. If not passed, names must be passed.
| metadata (`Union[dict, Mapping]`, defaults to `None`):
| Optional metadata for the schema (if inferred).
|
| Returns:
| `datasets.table.Table`
|
| from_batches(*args, **kwargs)
| Construct a Table from a sequence or iterator of Arrow `RecordBatches`.
|
| Args:
| batches (`Union[Sequence[pyarrow.RecordBatch], Iterator[pyarrow.RecordBatch]]`):
| Sequence of `RecordBatch` to be converted, all schemas must be equal.
| schema (`Schema`, defaults to `None`):
| If not passed, will be inferred from the first `RecordBatch`.
|
| Returns:
| `datasets.table.Table`:
|
| from_buffer(buffer: pyarrow.lib.Buffer)
|
| from_file(filename: str)
|
| from_pandas(*args, **kwargs)
| Convert pandas.DataFrame to an Arrow Table.
|
| The column types in the resulting Arrow Table are inferred from the
| dtypes of the pandas.Series in the DataFrame. In the case of non-object
| Series, the NumPy dtype is translated to its Arrow equivalent. In the
| case of `object`, we need to guess the datatype by looking at the
| Python objects in this Series.
|
| Be aware that Series of the `object` dtype don't carry enough
| information to always lead to a meaningful Arrow type. In the case that
| we cannot infer a type, e.g. because the DataFrame is of length 0 or
| the Series only contains `None/nan` objects, the type is set to
| null. This behavior can be avoided by constructing an explicit schema
| and passing it to this function.
|
| Args:
| df (`pandas.DataFrame`):
| schema (`pyarrow.Schema`, *optional*):
| The expected schema of the Arrow Table. This can be used to
| indicate the type of columns if we cannot infer it automatically.
| If passed, the output will have exactly this schema. Columns
| specified in the schema that are not found in the DataFrame columns
| or its index will raise an error. Additional columns or index
| levels in the DataFrame which are not specified in the schema will
| be ignored.
| preserve_index (`bool`, *optional*):
| Whether to store the index as an additional column in the resulting
| `Table`. The default of None will store the index as a column,
| except for RangeIndex which is stored as metadata only. Use
| `preserve_index=True` to force it to be stored as a column.
| nthreads (`int`, defaults to `None` (may use up to system CPU count threads))
| If greater than 1, convert columns to Arrow in parallel using
| indicated number of threads.
| columns (`List[str]`, *optional*):
| List of column to be converted. If `None`, use all columns.
| safe (`bool`, defaults to `True`):
| Check for overflows or other unsafe conversions,
|
| Returns:
| `datasets.table.Table`:
|
| Examples:
| ```python
| >>> import pandas as pd
| >>> import pyarrow as pa
| >>> df = pd.DataFrame({
| ... 'int': [1, 2],
| ... 'str': ['a', 'b']
| ... })
| >>> pa.Table.from_pandas(df)
| <pyarrow.lib.Table object at 0x7f05d1fb1b40>
| ```
|
| from_pydict(*args, **kwargs)
| Construct a Table from Arrow arrays or columns.
|
| Args:
| mapping (`Union[dict, Mapping]`):
| A mapping of strings to Arrays or Python lists.
| schema (`Schema`, defaults to `None`):
| If not passed, will be inferred from the Mapping values
| metadata (`Union[dict, Mapping]`, defaults to `None`):
| Optional metadata for the schema (if inferred).
|
| Returns:
| `datasets.table.Table`
|
| from_pylist(mapping, *args, **kwargs)
| Construct a Table from list of rows / dictionaries.
|
| Args:
| mapping (`List[dict]`):
| A mapping of strings to row values.
| schema (`Schema`, defaults to `None`):
| If not passed, will be inferred from the Mapping values
| metadata (`Union[dict, Mapping]`, defaults to `None`):
| Optional metadata for the schema (if inferred).
|
| Returns:
| `datasets.table.Table`
|
| ----------------------------------------------------------------------
| Methods inherited from Table:
|
| __deepcopy__(self, memo: dict)
|
| __eq__(self, other)
| Return self==value.
|
| __getitem__(self, i)
|
| __init__(self, table: pyarrow.lib.Table)
| Initialize self. See help(type(self)) for accurate signature.
|
| __len__(self)
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| column(self, *args, **kwargs)
| Select a column by its column name, or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the column to retrieve.
|
| Returns:
| `pyarrow.ChunkedArray`
|
| equals(self, *args, **kwargs)
| Check if contents of two tables are equal.
|
| Args:
| other ([`~datasets.table.Table`]):
| Table to compare against.
| check_metadata `bool`, defaults to `False`):
| Whether schema metadata equality should be checked as well.
|
| Returns:
| `bool`
|
| field(self, *args, **kwargs)
| Select a schema field by its column name or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the field to retrieve.
|
| Returns:
| `pyarrow.Field`
|
| itercolumns(self, *args, **kwargs)
| Iterator over all columns in their numerical order.
|
| Yields:
| `pyarrow.ChunkedArray`
|
| to_batches(self, *args, **kwargs)
| Convert Table to list of (contiguous) `RecordBatch` objects.
|
| Args:
| max_chunksize (`int`, defaults to `None`):
| Maximum size for `RecordBatch` chunks. Individual chunks may be
| smaller depending on the chunk layout of individual columns.
|
| Returns:
| `List[pyarrow.RecordBatch]`
|
| to_pandas(self, *args, **kwargs)
| Convert to a pandas-compatible NumPy array or DataFrame, as appropriate.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| Arrow MemoryPool to use for allocations. Uses the default memory
| pool is not passed.
| strings_to_categorical (`bool`, defaults to `False`):
| Encode string (UTF8) and binary types to `pandas.Categorical`.
| categories (`list`, defaults to `empty`):
| List of fields that should be returned as `pandas.Categorical`. Only
| applies to table-like data structures.
| zero_copy_only (`bool`, defaults to `False`):
| Raise an `ArrowException` if this function call would require copying
| the underlying data.
| integer_object_nulls (`bool`, defaults to `False`):
| Cast integers with nulls to objects.
| date_as_object (`bool`, defaults to `True`):
| Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype.
| timestamp_as_object (`bool`, defaults to `False`):
| Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is
| useful if you have timestamps that don't fit in the normal date
| range of nanosecond timestamps (1678 CE-2262 CE).
| If `False`, all timestamps are converted to `datetime64[ns]` dtype.
| use_threads (`bool`, defaults to `True`):
| Whether to parallelize the conversion using multiple threads.
| deduplicate_objects (`bool`, defaults to `False`):
| Do not create multiple copies Python objects when created, to save
| on memory use. Conversion will be slower.
| ignore_metadata (`bool`, defaults to `False`):
| If `True`, do not use the 'pandas' metadata to reconstruct the
| DataFrame index, if present.
| safe (`bool`, defaults to `True`):
| For certain data types, a cast is needed in order to store the
| data in a pandas DataFrame or Series (e.g. timestamps are always
| stored as nanoseconds in pandas). This option controls whether it
| is a safe cast or not.
| split_blocks (`bool`, defaults to `False`):
| If `True`, generate one internal "block" for each column when
| creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this
| can temporarily reduce memory note that various pandas operations
| can trigger "consolidation" which may balloon memory use.
| self_destruct (`bool`, defaults to `False`):
| EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow
| memory while converting the Arrow object to pandas. If you use the
| object after calling `to_pandas` with this option it will crash your
| program.
| types_mapper (`function`, defaults to `None`):
| A function mapping a pyarrow DataType to a pandas `ExtensionDtype`.
| This can be used to override the default pandas type for conversion
| of built-in pyarrow types or in absence of `pandas_metadata` in the
| Table schema. The function receives a pyarrow DataType and is
| expected to return a pandas `ExtensionDtype` or `None` if the
| default conversion should be used for that type. If you have
| a dictionary mapping, you can pass `dict.get` as function.
|
| Returns:
| `pandas.Series` or `pandas.DataFrame`: `pandas.Series` or `pandas.DataFrame` depending on type of object
|
| to_pydict(self, *args, **kwargs)
| Convert the Table to a `dict` or `OrderedDict`.
|
| Returns:
| `dict`
|
| to_pylist(self, *args, **kwargs)
| Convert the Table to a list
|
| Returns:
| `list`
|
| to_reader(self, max_chunksize: Optional[int] = None)
| Convert the Table to a RecordBatchReader.
|
| Note that this method is zero-copy, it merely exposes the same data under a different API.
|
| Args:
| max_chunksize (`int`, defaults to `None`)
| Maximum size for RecordBatch chunks. Individual chunks may be smaller depending
| on the chunk layout of individual columns.
|
| Returns:
| `pyarrow.RecordBatchReader`
|
| to_string(self, *args, **kwargs)
|
| validate(self, *args, **kwargs)
| Perform validation checks. An exception is raised if validation fails.
|
| By default only cheap validation checks are run. Pass `full=True`
| for thorough validation checks (potentially `O(n)`).
|
| Args:
| full (`bool`, defaults to `False`):
| If `True`, run expensive checks, otherwise cheap checks only.
|
| Raises:
| `pa.lib.ArrowInvalid`: if validation fails
|
| ----------------------------------------------------------------------
| Readonly properties inherited from Table:
|
| column_names
| Names of the table's columns.
|
| columns
| List of all columns in numerical order.
|
| Returns:
| `List[pa.ChunkedArray]`
|
| nbytes
| Total number of bytes consumed by the elements of the table.
|
| num_columns
| Number of columns in this table.
|
| Returns:
| int
|
| num_rows
| Number of rows in this table.
|
| Due to the definition of a table, all columns have the same number of
| rows.
|
| Returns:
| int
|
| schema
| Schema of the table and its columns.
|
| Returns:
| `pyarrow.Schema`
|
| shape
| Dimensions of the table: (#rows, #columns).
|
| Returns:
| `(int, int)`: Number of rows and number of columns.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from Table:
|
| __hash__ = None
|
| ----------------------------------------------------------------------
| Methods inherited from IndexedTableMixin:
|
| fast_gather(self, indices: Union[list[int], numpy.ndarray]) -> pyarrow.lib.Table
| Create a pa.Table by gathering the records at the records at the specified indices. Should be faster
| than pa.concat_tables(table.fast_slice(int(i) % table.num_rows, 1) for i in indices) since NumPy can compute
| the binary searches in parallel, highly optimized C
|
| fast_slice(self, offset=0, length=None) -> pyarrow.lib.Table
| Slice the Table using interpolation search.
| The behavior is the same as `pyarrow.Table.slice` but it's significantly faster.
|
| Interpolation search is used to find the start and end indexes of the batches we want to keep.
| The batches to keep are then concatenated to form the sliced Table.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from IndexedTableMixin:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class IndexedTableMixin(builtins.object)
| IndexedTableMixin(table: pyarrow.lib.Table)
|
| Methods defined here:
|
| __init__(self, table: pyarrow.lib.Table)
| Initialize self. See help(type(self)) for accurate signature.
|
| fast_gather(self, indices: Union[list[int], numpy.ndarray]) -> pyarrow.lib.Table
| Create a pa.Table by gathering the records at the records at the specified indices. Should be faster
| than pa.concat_tables(table.fast_slice(int(i) % table.num_rows, 1) for i in indices) since NumPy can compute
| the binary searches in parallel, highly optimized C
|
| fast_slice(self, offset=0, length=None) -> pyarrow.lib.Table
| Slice the Table using interpolation search.
| The behavior is the same as `pyarrow.Table.slice` but it's significantly faster.
|
| Interpolation search is used to find the start and end indexes of the batches we want to keep.
| The batches to keep are then concatenated to form the sliced Table.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class MemoryMappedTable(TableBlock)
| MemoryMappedTable(table: pyarrow.lib.Table, path: str, replays: Optional[list[tuple[str, tuple, dict]]] = None)
|
| The table is said memory mapped when it doesn't use the user's RAM but loads the data
| from the disk instead.
|
| Pickling it doesn't copy the data into memory.
| Instead, only the path to the memory mapped arrow file is pickled, as well as the list
| of transforms to "replay" when reloading the table from the disk.
|
| Its implementation requires to store an history of all the transforms that were applied
| to the underlying pyarrow Table, so that they can be "replayed" when reloading the Table
| from the disk.
|
| This is different from the `InMemoryTable` table, for which pickling does copy all the
| data in memory.
|
| `InMemoryTable` must be used when data fit in memory, while `MemoryMapped` are reserved for
| data bigger than memory or when you want the memory footprint of your application to
| stay low.
|
| Method resolution order:
| MemoryMappedTable
| TableBlock
| Table
| IndexedTableMixin
| builtins.object
|
| Methods defined here:
|
| __getstate__(self)
| Helper for pickle.
|
| __init__(self, table: pyarrow.lib.Table, path: str, replays: Optional[list[tuple[str, tuple, dict]]] = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __setstate__(self, state)
|
| add_column(self, *args, **kwargs)
| Add column to Table at position.
|
| A new table is returned with the column added, the original table
| object is left unchanged.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`: New table with the passed column added.
|
| append_column(self, *args, **kwargs)
| Append column at end of columns.
|
| Args:
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`:
| New table with the passed column added.
|
| cast(self, *args, **kwargs)
| Cast table values to another schema
|
| Args:
| target_schema (`Schema`):
| Schema to cast to, the names and order of fields must match.
| safe (`bool`, defaults to `True`):
| Check for overflows or other unsafe conversions.
|
| Returns:
| `datasets.table.Table`
|
| combine_chunks(self, *args, **kwargs)
| Make a new table by combining the chunks this table has.
|
| All the underlying chunks in the ChunkedArray of each column are
| concatenated into zero or one chunk.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| drop(self, *args, **kwargs)
| Drop one or more columns and return a new table.
|
| Args:
| columns (`List[str]`):
| List of field names referencing existing columns.
|
| Raises:
| `KeyError` : if any of the passed columns name are not existing.
|
| Returns:
| `datasets.table.Table`:
| New table without the columns.
|
| filter(self, *args, **kwargs)
| Select records from a Table. See `pyarrow.compute.filter` for full usage.
|
| flatten(self, *args, **kwargs)
| Flatten this Table. Each column with a struct type is flattened
| into one column per struct field. Other columns are left unchanged.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| remove_column(self, *args, **kwargs)
| Create new Table with the indicated column removed.
|
| Args:
| i (`int`):
| Index of column to remove.
|
| Returns:
| `datasets.table.Table`:
| New table without the column.
|
| rename_columns(self, *args, **kwargs)
| Create new table with columns renamed to provided names.
|
| replace_schema_metadata(self, *args, **kwargs)
| EXPERIMENTAL: Create shallow copy of table by replacing schema
| key-value metadata with the indicated new metadata (which may be None,
| which deletes any existing metadata.
|
| Args:
| metadata (`dict`, defaults to `None`):
|
| Returns:
| `datasets.table.Table`: shallow_copy
|
| select(self, *args, **kwargs)
| Select columns of the table.
|
| Returns a new table with the specified columns, and metadata preserved.
|
| Args:
| columns (:obj:`Union[List[str], List[int]]`):
| The column names or integer indices to select.
|
| Returns:
| :class:`datasets.table.Table`: New table with the specified columns, and metadata preserved.
|
| set_column(self, *args, **kwargs)
| Replace column in Table at position.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`:
| New table with the passed column set.
|
| slice(self, offset=0, length=None)
| Compute zero-copy slice of this Table.
|
| Args:
| offset (`int`, defaults to `0`):
| Offset from start of table to slice.
| length (`int`, defaults to `None`):
| Length of slice (default is until end of table starting from
| offset).
|
| Returns:
| `datasets.table.Table`
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_file(filename: str, replays=None)
|
| ----------------------------------------------------------------------
| Methods inherited from Table:
|
| __deepcopy__(self, memo: dict)
|
| __eq__(self, other)
| Return self==value.
|
| __getitem__(self, i)
|
| __len__(self)
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| column(self, *args, **kwargs)
| Select a column by its column name, or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the column to retrieve.
|
| Returns:
| `pyarrow.ChunkedArray`
|
| equals(self, *args, **kwargs)
| Check if contents of two tables are equal.
|
| Args:
| other ([`~datasets.table.Table`]):
| Table to compare against.
| check_metadata `bool`, defaults to `False`):
| Whether schema metadata equality should be checked as well.
|
| Returns:
| `bool`
|
| field(self, *args, **kwargs)
| Select a schema field by its column name or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the field to retrieve.
|
| Returns:
| `pyarrow.Field`
|
| itercolumns(self, *args, **kwargs)
| Iterator over all columns in their numerical order.
|
| Yields:
| `pyarrow.ChunkedArray`
|
| to_batches(self, *args, **kwargs)
| Convert Table to list of (contiguous) `RecordBatch` objects.
|
| Args:
| max_chunksize (`int`, defaults to `None`):
| Maximum size for `RecordBatch` chunks. Individual chunks may be
| smaller depending on the chunk layout of individual columns.
|
| Returns:
| `List[pyarrow.RecordBatch]`
|
| to_pandas(self, *args, **kwargs)
| Convert to a pandas-compatible NumPy array or DataFrame, as appropriate.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| Arrow MemoryPool to use for allocations. Uses the default memory
| pool is not passed.
| strings_to_categorical (`bool`, defaults to `False`):
| Encode string (UTF8) and binary types to `pandas.Categorical`.
| categories (`list`, defaults to `empty`):
| List of fields that should be returned as `pandas.Categorical`. Only
| applies to table-like data structures.
| zero_copy_only (`bool`, defaults to `False`):
| Raise an `ArrowException` if this function call would require copying
| the underlying data.
| integer_object_nulls (`bool`, defaults to `False`):
| Cast integers with nulls to objects.
| date_as_object (`bool`, defaults to `True`):
| Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype.
| timestamp_as_object (`bool`, defaults to `False`):
| Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is
| useful if you have timestamps that don't fit in the normal date
| range of nanosecond timestamps (1678 CE-2262 CE).
| If `False`, all timestamps are converted to `datetime64[ns]` dtype.
| use_threads (`bool`, defaults to `True`):
| Whether to parallelize the conversion using multiple threads.
| deduplicate_objects (`bool`, defaults to `False`):
| Do not create multiple copies Python objects when created, to save
| on memory use. Conversion will be slower.
| ignore_metadata (`bool`, defaults to `False`):
| If `True`, do not use the 'pandas' metadata to reconstruct the
| DataFrame index, if present.
| safe (`bool`, defaults to `True`):
| For certain data types, a cast is needed in order to store the
| data in a pandas DataFrame or Series (e.g. timestamps are always
| stored as nanoseconds in pandas). This option controls whether it
| is a safe cast or not.
| split_blocks (`bool`, defaults to `False`):
| If `True`, generate one internal "block" for each column when
| creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this
| can temporarily reduce memory note that various pandas operations
| can trigger "consolidation" which may balloon memory use.
| self_destruct (`bool`, defaults to `False`):
| EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow
| memory while converting the Arrow object to pandas. If you use the
| object after calling `to_pandas` with this option it will crash your
| program.
| types_mapper (`function`, defaults to `None`):
| A function mapping a pyarrow DataType to a pandas `ExtensionDtype`.
| This can be used to override the default pandas type for conversion
| of built-in pyarrow types or in absence of `pandas_metadata` in the
| Table schema. The function receives a pyarrow DataType and is
| expected to return a pandas `ExtensionDtype` or `None` if the
| default conversion should be used for that type. If you have
| a dictionary mapping, you can pass `dict.get` as function.
|
| Returns:
| `pandas.Series` or `pandas.DataFrame`: `pandas.Series` or `pandas.DataFrame` depending on type of object
|
| to_pydict(self, *args, **kwargs)
| Convert the Table to a `dict` or `OrderedDict`.
|
| Returns:
| `dict`
|
| to_pylist(self, *args, **kwargs)
| Convert the Table to a list
|
| Returns:
| `list`
|
| to_reader(self, max_chunksize: Optional[int] = None)
| Convert the Table to a RecordBatchReader.
|
| Note that this method is zero-copy, it merely exposes the same data under a different API.
|
| Args:
| max_chunksize (`int`, defaults to `None`)
| Maximum size for RecordBatch chunks. Individual chunks may be smaller depending
| on the chunk layout of individual columns.
|
| Returns:
| `pyarrow.RecordBatchReader`
|
| to_string(self, *args, **kwargs)
|
| validate(self, *args, **kwargs)
| Perform validation checks. An exception is raised if validation fails.
|
| By default only cheap validation checks are run. Pass `full=True`
| for thorough validation checks (potentially `O(n)`).
|
| Args:
| full (`bool`, defaults to `False`):
| If `True`, run expensive checks, otherwise cheap checks only.
|
| Raises:
| `pa.lib.ArrowInvalid`: if validation fails
|
| ----------------------------------------------------------------------
| Readonly properties inherited from Table:
|
| column_names
| Names of the table's columns.
|
| columns
| List of all columns in numerical order.
|
| Returns:
| `List[pa.ChunkedArray]`
|
| nbytes
| Total number of bytes consumed by the elements of the table.
|
| num_columns
| Number of columns in this table.
|
| Returns:
| int
|
| num_rows
| Number of rows in this table.
|
| Due to the definition of a table, all columns have the same number of
| rows.
|
| Returns:
| int
|
| schema
| Schema of the table and its columns.
|
| Returns:
| `pyarrow.Schema`
|
| shape
| Dimensions of the table: (#rows, #columns).
|
| Returns:
| `(int, int)`: Number of rows and number of columns.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from Table:
|
| __hash__ = None
|
| ----------------------------------------------------------------------
| Methods inherited from IndexedTableMixin:
|
| fast_gather(self, indices: Union[list[int], numpy.ndarray]) -> pyarrow.lib.Table
| Create a pa.Table by gathering the records at the records at the specified indices. Should be faster
| than pa.concat_tables(table.fast_slice(int(i) % table.num_rows, 1) for i in indices) since NumPy can compute
| the binary searches in parallel, highly optimized C
|
| fast_slice(self, offset=0, length=None) -> pyarrow.lib.Table
| Slice the Table using interpolation search.
| The behavior is the same as `pyarrow.Table.slice` but it's significantly faster.
|
| Interpolation search is used to find the start and end indexes of the batches we want to keep.
| The batches to keep are then concatenated to form the sliced Table.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from IndexedTableMixin:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class Table(IndexedTableMixin)
| Table(table: pyarrow.lib.Table)
|
| Wraps a pyarrow Table by using composition.
| This is the base class for `InMemoryTable`, `MemoryMappedTable` and `ConcatenationTable`.
|
| It implements all the basic attributes/methods of the pyarrow Table class except
| the Table transforms: `slice, filter, flatten, combine_chunks, cast, add_column,
| append_column, remove_column, set_column, rename_columns` and `drop`.
|
| The implementation of these methods differs for the subclasses.
|
| Method resolution order:
| Table
| IndexedTableMixin
| builtins.object
|
| Methods defined here:
|
| __deepcopy__(self, memo: dict)
|
| __eq__(self, other)
| Return self==value.
|
| __getitem__(self, i)
|
| __init__(self, table: pyarrow.lib.Table)
| Initialize self. See help(type(self)) for accurate signature.
|
| __len__(self)
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| add_column(self, *args, **kwargs)
| Add column to Table at position.
|
| A new table is returned with the column added, the original table
| object is left unchanged.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`: New table with the passed column added.
|
| append_column(self, *args, **kwargs)
| Append column at end of columns.
|
| Args:
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`: New table with the passed column added.
|
| cast(self, *args, **kwargs)
| Cast table values to another schema.
|
| Args:
| target_schema (`Schema`):
| Schema to cast to, the names and order of fields must match.
| safe (`bool`, defaults to `True`):
| Check for overflows or other unsafe conversions.
|
| Returns:
| `datasets.table.Table`
|
| column(self, *args, **kwargs)
| Select a column by its column name, or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the column to retrieve.
|
| Returns:
| `pyarrow.ChunkedArray`
|
| combine_chunks(self, *args, **kwargs)
| Make a new table by combining the chunks this table has.
|
| All the underlying chunks in the `ChunkedArray` of each column are
| concatenated into zero or one chunk.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| drop(self, *args, **kwargs)
| Drop one or more columns and return a new table.
|
| Args:
| columns (`List[str]`):
| List of field names referencing existing columns.
|
| Raises:
| `KeyError` : if any of the passed columns name are not existing.
|
| Returns:
| `datasets.table.Table`: New table without the columns.
|
| equals(self, *args, **kwargs)
| Check if contents of two tables are equal.
|
| Args:
| other ([`~datasets.table.Table`]):
| Table to compare against.
| check_metadata `bool`, defaults to `False`):
| Whether schema metadata equality should be checked as well.
|
| Returns:
| `bool`
|
| field(self, *args, **kwargs)
| Select a schema field by its column name or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the field to retrieve.
|
| Returns:
| `pyarrow.Field`
|
| filter(self, *args, **kwargs)
| Select records from a Table. See `pyarrow.compute.filter` for full usage.
|
| flatten(self, *args, **kwargs)
| Flatten this Table. Each column with a struct type is flattened
| into one column per struct field. Other columns are left unchanged.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| itercolumns(self, *args, **kwargs)
| Iterator over all columns in their numerical order.
|
| Yields:
| `pyarrow.ChunkedArray`
|
| remove_column(self, *args, **kwargs)
| Create new Table with the indicated column removed.
|
| Args:
| i (`int`):
| Index of column to remove.
|
| Returns:
| `datasets.table.Table`: New table without the column.
|
| rename_columns(self, *args, **kwargs)
| Create new table with columns renamed to provided names.
|
| replace_schema_metadata(self, *args, **kwargs)
| EXPERIMENTAL: Create shallow copy of table by replacing schema
| key-value metadata with the indicated new metadata (which may be None,
| which deletes any existing metadata
|
| Args:
| metadata (`dict`, defaults to `None`):
|
| Returns:
| `datasets.table.Table`: shallow_copy
|
| select(self, *args, **kwargs)
| Select columns of the table.
|
| Returns a new table with the specified columns, and metadata preserved.
|
| Args:
| columns (:obj:`Union[List[str], List[int]]`):
| The column names or integer indices to select.
|
| Returns:
| `datasets.table.Table`: table with only a subset of the columns
|
| set_column(self, *args, **kwargs)
| Replace column in Table at position.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`: New table with the passed column set.
|
| slice(self, *args, **kwargs)
| Compute zero-copy slice of this Table.
|
| Args:
| offset (`int`, defaults to `0`):
| Offset from start of table to slice.
| length (`int`, defaults to `None`):
| Length of slice (default is until end of table starting from
| offset).
|
| Returns:
| `datasets.table.Table`
|
| to_batches(self, *args, **kwargs)
| Convert Table to list of (contiguous) `RecordBatch` objects.
|
| Args:
| max_chunksize (`int`, defaults to `None`):
| Maximum size for `RecordBatch` chunks. Individual chunks may be
| smaller depending on the chunk layout of individual columns.
|
| Returns:
| `List[pyarrow.RecordBatch]`
|
| to_pandas(self, *args, **kwargs)
| Convert to a pandas-compatible NumPy array or DataFrame, as appropriate.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| Arrow MemoryPool to use for allocations. Uses the default memory
| pool is not passed.
| strings_to_categorical (`bool`, defaults to `False`):
| Encode string (UTF8) and binary types to `pandas.Categorical`.
| categories (`list`, defaults to `empty`):
| List of fields that should be returned as `pandas.Categorical`. Only
| applies to table-like data structures.
| zero_copy_only (`bool`, defaults to `False`):
| Raise an `ArrowException` if this function call would require copying
| the underlying data.
| integer_object_nulls (`bool`, defaults to `False`):
| Cast integers with nulls to objects.
| date_as_object (`bool`, defaults to `True`):
| Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype.
| timestamp_as_object (`bool`, defaults to `False`):
| Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is
| useful if you have timestamps that don't fit in the normal date
| range of nanosecond timestamps (1678 CE-2262 CE).
| If `False`, all timestamps are converted to `datetime64[ns]` dtype.
| use_threads (`bool`, defaults to `True`):
| Whether to parallelize the conversion using multiple threads.
| deduplicate_objects (`bool`, defaults to `False`):
| Do not create multiple copies Python objects when created, to save
| on memory use. Conversion will be slower.
| ignore_metadata (`bool`, defaults to `False`):
| If `True`, do not use the 'pandas' metadata to reconstruct the
| DataFrame index, if present.
| safe (`bool`, defaults to `True`):
| For certain data types, a cast is needed in order to store the
| data in a pandas DataFrame or Series (e.g. timestamps are always
| stored as nanoseconds in pandas). This option controls whether it
| is a safe cast or not.
| split_blocks (`bool`, defaults to `False`):
| If `True`, generate one internal "block" for each column when
| creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this
| can temporarily reduce memory note that various pandas operations
| can trigger "consolidation" which may balloon memory use.
| self_destruct (`bool`, defaults to `False`):
| EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow
| memory while converting the Arrow object to pandas. If you use the
| object after calling `to_pandas` with this option it will crash your
| program.
| types_mapper (`function`, defaults to `None`):
| A function mapping a pyarrow DataType to a pandas `ExtensionDtype`.
| This can be used to override the default pandas type for conversion
| of built-in pyarrow types or in absence of `pandas_metadata` in the
| Table schema. The function receives a pyarrow DataType and is
| expected to return a pandas `ExtensionDtype` or `None` if the
| default conversion should be used for that type. If you have
| a dictionary mapping, you can pass `dict.get` as function.
|
| Returns:
| `pandas.Series` or `pandas.DataFrame`: `pandas.Series` or `pandas.DataFrame` depending on type of object
|
| to_pydict(self, *args, **kwargs)
| Convert the Table to a `dict` or `OrderedDict`.
|
| Returns:
| `dict`
|
| to_pylist(self, *args, **kwargs)
| Convert the Table to a list
|
| Returns:
| `list`
|
| to_reader(self, max_chunksize: Optional[int] = None)
| Convert the Table to a RecordBatchReader.
|
| Note that this method is zero-copy, it merely exposes the same data under a different API.
|
| Args:
| max_chunksize (`int`, defaults to `None`)
| Maximum size for RecordBatch chunks. Individual chunks may be smaller depending
| on the chunk layout of individual columns.
|
| Returns:
| `pyarrow.RecordBatchReader`
|
| to_string(self, *args, **kwargs)
|
| validate(self, *args, **kwargs)
| Perform validation checks. An exception is raised if validation fails.
|
| By default only cheap validation checks are run. Pass `full=True`
| for thorough validation checks (potentially `O(n)`).
|
| Args:
| full (`bool`, defaults to `False`):
| If `True`, run expensive checks, otherwise cheap checks only.
|
| Raises:
| `pa.lib.ArrowInvalid`: if validation fails
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| column_names
| Names of the table's columns.
|
| columns
| List of all columns in numerical order.
|
| Returns:
| `List[pa.ChunkedArray]`
|
| nbytes
| Total number of bytes consumed by the elements of the table.
|
| num_columns
| Number of columns in this table.
|
| Returns:
| int
|
| num_rows
| Number of rows in this table.
|
| Due to the definition of a table, all columns have the same number of
| rows.
|
| Returns:
| int
|
| schema
| Schema of the table and its columns.
|
| Returns:
| `pyarrow.Schema`
|
| shape
| Dimensions of the table: (#rows, #columns).
|
| Returns:
| `(int, int)`: Number of rows and number of columns.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __hash__ = None
|
| ----------------------------------------------------------------------
| Methods inherited from IndexedTableMixin:
|
| fast_gather(self, indices: Union[list[int], numpy.ndarray]) -> pyarrow.lib.Table
| Create a pa.Table by gathering the records at the records at the specified indices. Should be faster
| than pa.concat_tables(table.fast_slice(int(i) % table.num_rows, 1) for i in indices) since NumPy can compute
| the binary searches in parallel, highly optimized C
|
| fast_slice(self, offset=0, length=None) -> pyarrow.lib.Table
| Slice the Table using interpolation search.
| The behavior is the same as `pyarrow.Table.slice` but it's significantly faster.
|
| Interpolation search is used to find the start and end indexes of the batches we want to keep.
| The batches to keep are then concatenated to form the sliced Table.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from IndexedTableMixin:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class TableBlock(Table)
| TableBlock(table: pyarrow.lib.Table)
|
| `TableBlock` is the allowed class inside a `ConcanetationTable`.
| Only `MemoryMappedTable` and `InMemoryTable` are `TableBlock`.
| This is because we don't want a `ConcanetationTable` made out of other `ConcanetationTables`.
|
| Method resolution order:
| TableBlock
| Table
| IndexedTableMixin
| builtins.object
|
| Methods inherited from Table:
|
| __deepcopy__(self, memo: dict)
|
| __eq__(self, other)
| Return self==value.
|
| __getitem__(self, i)
|
| __init__(self, table: pyarrow.lib.Table)
| Initialize self. See help(type(self)) for accurate signature.
|
| __len__(self)
|
| __repr__(self)
| Return repr(self).
|
| __str__(self)
| Return str(self).
|
| add_column(self, *args, **kwargs)
| Add column to Table at position.
|
| A new table is returned with the column added, the original table
| object is left unchanged.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`: New table with the passed column added.
|
| append_column(self, *args, **kwargs)
| Append column at end of columns.
|
| Args:
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`: New table with the passed column added.
|
| cast(self, *args, **kwargs)
| Cast table values to another schema.
|
| Args:
| target_schema (`Schema`):
| Schema to cast to, the names and order of fields must match.
| safe (`bool`, defaults to `True`):
| Check for overflows or other unsafe conversions.
|
| Returns:
| `datasets.table.Table`
|
| column(self, *args, **kwargs)
| Select a column by its column name, or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the column to retrieve.
|
| Returns:
| `pyarrow.ChunkedArray`
|
| combine_chunks(self, *args, **kwargs)
| Make a new table by combining the chunks this table has.
|
| All the underlying chunks in the `ChunkedArray` of each column are
| concatenated into zero or one chunk.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| drop(self, *args, **kwargs)
| Drop one or more columns and return a new table.
|
| Args:
| columns (`List[str]`):
| List of field names referencing existing columns.
|
| Raises:
| `KeyError` : if any of the passed columns name are not existing.
|
| Returns:
| `datasets.table.Table`: New table without the columns.
|
| equals(self, *args, **kwargs)
| Check if contents of two tables are equal.
|
| Args:
| other ([`~datasets.table.Table`]):
| Table to compare against.
| check_metadata `bool`, defaults to `False`):
| Whether schema metadata equality should be checked as well.
|
| Returns:
| `bool`
|
| field(self, *args, **kwargs)
| Select a schema field by its column name or numeric index.
|
| Args:
| i (`Union[int, str]`):
| The index or name of the field to retrieve.
|
| Returns:
| `pyarrow.Field`
|
| filter(self, *args, **kwargs)
| Select records from a Table. See `pyarrow.compute.filter` for full usage.
|
| flatten(self, *args, **kwargs)
| Flatten this Table. Each column with a struct type is flattened
| into one column per struct field. Other columns are left unchanged.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| For memory allocations, if required, otherwise use default pool.
|
| Returns:
| `datasets.table.Table`
|
| itercolumns(self, *args, **kwargs)
| Iterator over all columns in their numerical order.
|
| Yields:
| `pyarrow.ChunkedArray`
|
| remove_column(self, *args, **kwargs)
| Create new Table with the indicated column removed.
|
| Args:
| i (`int`):
| Index of column to remove.
|
| Returns:
| `datasets.table.Table`: New table without the column.
|
| rename_columns(self, *args, **kwargs)
| Create new table with columns renamed to provided names.
|
| replace_schema_metadata(self, *args, **kwargs)
| EXPERIMENTAL: Create shallow copy of table by replacing schema
| key-value metadata with the indicated new metadata (which may be None,
| which deletes any existing metadata
|
| Args:
| metadata (`dict`, defaults to `None`):
|
| Returns:
| `datasets.table.Table`: shallow_copy
|
| select(self, *args, **kwargs)
| Select columns of the table.
|
| Returns a new table with the specified columns, and metadata preserved.
|
| Args:
| columns (:obj:`Union[List[str], List[int]]`):
| The column names or integer indices to select.
|
| Returns:
| `datasets.table.Table`: table with only a subset of the columns
|
| set_column(self, *args, **kwargs)
| Replace column in Table at position.
|
| Args:
| i (`int`):
| Index to place the column at.
| field_ (`Union[str, pyarrow.Field]`):
| If a string is passed then the type is deduced from the column
| data.
| column (`Union[pyarrow.Array, List[pyarrow.Array]]`):
| Column data.
|
| Returns:
| `datasets.table.Table`: New table with the passed column set.
|
| slice(self, *args, **kwargs)
| Compute zero-copy slice of this Table.
|
| Args:
| offset (`int`, defaults to `0`):
| Offset from start of table to slice.
| length (`int`, defaults to `None`):
| Length of slice (default is until end of table starting from
| offset).
|
| Returns:
| `datasets.table.Table`
|
| to_batches(self, *args, **kwargs)
| Convert Table to list of (contiguous) `RecordBatch` objects.
|
| Args:
| max_chunksize (`int`, defaults to `None`):
| Maximum size for `RecordBatch` chunks. Individual chunks may be
| smaller depending on the chunk layout of individual columns.
|
| Returns:
| `List[pyarrow.RecordBatch]`
|
| to_pandas(self, *args, **kwargs)
| Convert to a pandas-compatible NumPy array or DataFrame, as appropriate.
|
| Args:
| memory_pool (`MemoryPool`, defaults to `None`):
| Arrow MemoryPool to use for allocations. Uses the default memory
| pool is not passed.
| strings_to_categorical (`bool`, defaults to `False`):
| Encode string (UTF8) and binary types to `pandas.Categorical`.
| categories (`list`, defaults to `empty`):
| List of fields that should be returned as `pandas.Categorical`. Only
| applies to table-like data structures.
| zero_copy_only (`bool`, defaults to `False`):
| Raise an `ArrowException` if this function call would require copying
| the underlying data.
| integer_object_nulls (`bool`, defaults to `False`):
| Cast integers with nulls to objects.
| date_as_object (`bool`, defaults to `True`):
| Cast dates to objects. If `False`, convert to `datetime64[ns]` dtype.
| timestamp_as_object (`bool`, defaults to `False`):
| Cast non-nanosecond timestamps (`np.datetime64`) to objects. This is
| useful if you have timestamps that don't fit in the normal date
| range of nanosecond timestamps (1678 CE-2262 CE).
| If `False`, all timestamps are converted to `datetime64[ns]` dtype.
| use_threads (`bool`, defaults to `True`):
| Whether to parallelize the conversion using multiple threads.
| deduplicate_objects (`bool`, defaults to `False`):
| Do not create multiple copies Python objects when created, to save
| on memory use. Conversion will be slower.
| ignore_metadata (`bool`, defaults to `False`):
| If `True`, do not use the 'pandas' metadata to reconstruct the
| DataFrame index, if present.
| safe (`bool`, defaults to `True`):
| For certain data types, a cast is needed in order to store the
| data in a pandas DataFrame or Series (e.g. timestamps are always
| stored as nanoseconds in pandas). This option controls whether it
| is a safe cast or not.
| split_blocks (`bool`, defaults to `False`):
| If `True`, generate one internal "block" for each column when
| creating a pandas.DataFrame from a `RecordBatch` or `Table`. While this
| can temporarily reduce memory note that various pandas operations
| can trigger "consolidation" which may balloon memory use.
| self_destruct (`bool`, defaults to `False`):
| EXPERIMENTAL: If `True`, attempt to deallocate the originating Arrow
| memory while converting the Arrow object to pandas. If you use the
| object after calling `to_pandas` with this option it will crash your
| program.
| types_mapper (`function`, defaults to `None`):
| A function mapping a pyarrow DataType to a pandas `ExtensionDtype`.
| This can be used to override the default pandas type for conversion
| of built-in pyarrow types or in absence of `pandas_metadata` in the
| Table schema. The function receives a pyarrow DataType and is
| expected to return a pandas `ExtensionDtype` or `None` if the
| default conversion should be used for that type. If you have
| a dictionary mapping, you can pass `dict.get` as function.
|
| Returns:
| `pandas.Series` or `pandas.DataFrame`: `pandas.Series` or `pandas.DataFrame` depending on type of object
|
| to_pydict(self, *args, **kwargs)
| Convert the Table to a `dict` or `OrderedDict`.
|
| Returns:
| `dict`
|
| to_pylist(self, *args, **kwargs)
| Convert the Table to a list
|
| Returns:
| `list`
|
| to_reader(self, max_chunksize: Optional[int] = None)
| Convert the Table to a RecordBatchReader.
|
| Note that this method is zero-copy, it merely exposes the same data under a different API.
|
| Args:
| max_chunksize (`int`, defaults to `None`)
| Maximum size for RecordBatch chunks. Individual chunks may be smaller depending
| on the chunk layout of individual columns.
|
| Returns:
| `pyarrow.RecordBatchReader`
|
| to_string(self, *args, **kwargs)
|
| validate(self, *args, **kwargs)
| Perform validation checks. An exception is raised if validation fails.
|
| By default only cheap validation checks are run. Pass `full=True`
| for thorough validation checks (potentially `O(n)`).
|
| Args:
| full (`bool`, defaults to `False`):
| If `True`, run expensive checks, otherwise cheap checks only.
|
| Raises:
| `pa.lib.ArrowInvalid`: if validation fails
|
| ----------------------------------------------------------------------
| Readonly properties inherited from Table:
|
| column_names
| Names of the table's columns.
|
| columns
| List of all columns in numerical order.
|
| Returns:
| `List[pa.ChunkedArray]`
|
| nbytes
| Total number of bytes consumed by the elements of the table.
|
| num_columns
| Number of columns in this table.
|
| Returns:
| int
|
| num_rows
| Number of rows in this table.
|
| Due to the definition of a table, all columns have the same number of
| rows.
|
| Returns:
| int
|
| schema
| Schema of the table and its columns.
|
| Returns:
| `pyarrow.Schema`
|
| shape
| Dimensions of the table: (#rows, #columns).
|
| Returns:
| `(int, int)`: Number of rows and number of columns.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from Table:
|
| __hash__ = None
|
| ----------------------------------------------------------------------
| Methods inherited from IndexedTableMixin:
|
| fast_gather(self, indices: Union[list[int], numpy.ndarray]) -> pyarrow.lib.Table
| Create a pa.Table by gathering the records at the records at the specified indices. Should be faster
| than pa.concat_tables(table.fast_slice(int(i) % table.num_rows, 1) for i in indices) since NumPy can compute
| the binary searches in parallel, highly optimized C
|
| fast_slice(self, offset=0, length=None) -> pyarrow.lib.Table
| Slice the Table using interpolation search.
| The behavior is the same as `pyarrow.Table.slice` but it's significantly faster.
|
| Interpolation search is used to find the start and end indexes of the batches we want to keep.
| The batches to keep are then concatenated to form the sliced Table.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from IndexedTableMixin:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
array_cast = wrapper(array, *args, **kwargs)
cast_array_to_feature = wrapper(array, *args, **kwargs)
cast_table_to_features(table: pyarrow.lib.Table, features: 'Features')
Cast a table to the arrow schema that corresponds to the requested features.
Args:
table (`pyarrow.Table`):
PyArrow table to cast.
features ([`Features`]):
Target features.
Returns:
table (`pyarrow.Table`): the casted table
cast_table_to_schema(table: pyarrow.lib.Table, schema: pyarrow.lib.Schema)
Cast a table to the arrow schema. Different from `cast_table_to_features`, this method can preserve nullability.
Args:
table (`pa.Table`):
PyArrow table to cast.
features ([`Features`]):
Target features.
Returns:
`pa.Table`: the casted table
concat_tables(tables: list[datasets.table.Table], axis: int = 0) -> datasets.table.Table
Concatenate tables.
Args:
tables (list of `Table`):
List of tables to be concatenated.
axis (`{0, 1}`, defaults to `0`, meaning over rows):
Axis to concatenate over, where `0` means over rows (vertically) and `1` means over columns
(horizontally).
<Added version="1.6.0"/>
Returns:
`datasets.table.Table`:
If the number of input tables is > 1, then the returned table is a `datasets.table.ConcatenationTable`.
Otherwise if there's only one table, it is returned as is.
embed_array_storage = wrapper(array, *args, **kwargs)
embed_table_storage(table: pyarrow.lib.Table, token_per_repo_id=None, local_files: bool = True, remote_files: bool = True)
Embed external data into a table's storage.
<Added version="2.4.0"/>
Args:
table (`pyarrow.Table`):
PyArrow table in which to embed data.
local_files (`bool`, defaults to `True`)
Whether to embed local files data in the table
<Added version="4.8.5"/>
remote_files (`bool`, defaults to `True`)
Whether to embed remote files data in the table.
E.g. files with paths that start with hf:// or https://
<Added version="4.8.5"/>
Returns:
table (`pyarrow.Table`): the table with embedded data
inject_arrow_table_documentation(arrow_table_method)
list_table_cache_files(table: datasets.table.Table) -> list[str]
Get the cache files that are loaded by the table.
Cache file are used when parts of the table come from the disk via memory mapping.
Returns:
`List[str]`:
A list of paths to the cache files loaded by the table.
read_schema_from_file(filename: str) -> pyarrow.lib.Schema
Infer arrow table schema from file without loading whole file into memory.
Useful especially while having very big files.
table_cast(table: pyarrow.lib.Table, schema: pyarrow.lib.Schema)
Improved version of `pa.Table.cast`.
It supports casting to feature types stored in the schema metadata.
Args:
table (`pyarrow.Table`):
PyArrow table to cast.
schema (`pyarrow.Schema`):
Target PyArrow schema.
Returns:
table (`pyarrow.Table`): the casted table
table_flatten(table: pyarrow.lib.Table)
Improved version of `pa.Table.flatten`.
It behaves as `pa.Table.flatten` in a sense it does 1-step flatten of the columns with a struct type into one column per struct field,
but updates the metadata and skips decodable features unless the `decode` attribute of these features is set to False.
Args:
table (`pa.Table`):
PyArrow table to flatten.
Returns:
`Table`: the flattened table
table_iter(table: datasets.table.Table, batch_size: int, drop_last_batch=False) -> collections.abc.Iterator[pyarrow.lib.Table]
Iterate over sub-tables of size `batch_size`.
Args:
table (`pyarrow.Table`):
PyArrow table to iterate over.
batch_size (`int`):
Size of each sub-table to yield.
drop_last_batch (`bool`, defaults to `False`):
Drop the last batch if it is smaller than `batch_size`.
table_visitor(table: pyarrow.lib.Table, function: Callable[[pyarrow.lib.Array], NoneType])
Visit all arrays in a table and apply a function to them.
Args:
table (`pyarrow.Table`):
PyArrow table to visit.
function (`Callable[[pa.Array], None]`):
Function to apply to each array.
DATA
Callable = typing.Callable
Deprecated alias to collections.abc.Callable.
Callable[[int], str] signifies a function that takes a single
parameter of type int and returns a str.
The subscription syntax must always be used with exactly two
values: the argument list and the return type.
The argument list must be a list of types, a ParamSpec,
Concatenate or ellipsis. The return type must be a single type.
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
Replay = tuple[str, tuple, dict]
TYPE_CHECKING = False
TableBlockContainer = ~TableBlockContainer
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
logger = <Logger datasets.table (WARNING)>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\table.py
datasets.tf_utils
Python Library Documentation: module datasets.utils.tf_utils in datasets.utils
NAME
datasets.utils.tf_utils - TF-specific utils import.
CLASSES
builtins.object
NumpyMultiprocessingGenerator
SharedMemoryContext
class NumpyMultiprocessingGenerator(builtins.object)
| NumpyMultiprocessingGenerator(dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, output_signature, shuffle, batch_size, drop_remainder, num_workers)
|
| Methods defined here:
|
| __call__(self)
| Call self as a function.
|
| __init__(self, dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, output_signature, shuffle, batch_size, drop_remainder, num_workers)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| distribute_batches(dataset, batch_size, drop_remainder, num_workers, shuffle)
|
| worker_loop(dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, columns_to_ranks, string_columns, indices, extra_batch, worker_name, array_ready_event, array_loaded_event)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class SharedMemoryContext(builtins.object)
| Methods defined here:
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_value, traceback)
|
| __init__(self)
| Initialize self. See help(type(self)) for accurate signature.
|
| get_array(self, name, shape, dtype, create)
|
| get_shm(self, name, size, create)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
ceil(x, /)
Return the ceiling of x as an Integral.
This is the smallest integer >= x.
dataset_to_tf(dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, output_signature, shuffle, batch_size, drop_remainder)
Create a tf.data.Dataset from the underlying Dataset. This is a single-process method - the multiprocess
equivalent is multiprocess_dataset_to_tf.
Args:
dataset (`Dataset`): Dataset to wrap with tf.data.Dataset.
cols_to_retain (`List[str]`): Dataset column(s) to load in the
tf.data.Dataset. It is acceptable to include column names that are created by the `collate_fn` and
that do not exist in the original dataset.
collate_fn(`Callable`): A function or callable object (such as a `DataCollator`) that will collate
lists of samples into a batch.
collate_fn_args (`Dict`): A `dict` of keyword arguments to be passed to the
`collate_fn`. Can be empty.
columns_to_np_types (`Dict[str, np.dtype]`): A `dict` mapping column names to numpy dtypes.
output_signature (`Dict[str, tf.TensorSpec]`): A `dict` mapping column names to
`tf.TensorSpec` objects.
shuffle(`bool`): Shuffle the dataset order when loading. Recommended True for training, False for
validation/evaluation.
batch_size (`int`, default `None`): Size of batches to load from the dataset. Defaults to `None`, which implies that
the dataset won't be batched, but the returned dataset can be batched later with `tf_dataset.batch(batch_size)`.
drop_remainder(`bool`, default `None`): Drop the last incomplete batch when loading. If not provided,
defaults to the same setting as shuffle.
Returns:
`tf.data.Dataset`
is_numeric_pa_type(pa_type)
minimal_tf_collate_fn(features)
minimal_tf_collate_fn_with_renaming(features)
multiprocess_dataset_to_tf(dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, output_signature, shuffle, batch_size, drop_remainder, num_workers)
Create a tf.data.Dataset from the underlying Dataset. This is a multi-process method - the single-process
equivalent is dataset_to_tf.
Args:
dataset (`Dataset`): Dataset to wrap with tf.data.Dataset.
cols_to_retain (`List[str]`): Dataset column(s) to load in the
tf.data.Dataset. It is acceptable to include column names that are created by the `collate_fn` and
that do not exist in the original dataset.
collate_fn(`Callable`): A function or callable object (such as a `DataCollator`) that will collate
lists of samples into a batch.
collate_fn_args (`Dict`): A `dict` of keyword arguments to be passed to the
`collate_fn`. Can be empty.
columns_to_np_types (`Dict[str, np.dtype]`): A `dict` mapping column names to numpy dtypes.
output_signature (`Dict[str, tf.TensorSpec]`): A `dict` mapping column names to
`tf.TensorSpec` objects.
shuffle(`bool`): Shuffle the dataset order when loading. Recommended True for training, False for
validation/evaluation.
batch_size (`int`, default `None`): Size of batches to load from the dataset. Defaults to `None`, which implies that
the dataset won't be batched, but the returned dataset can be batched later with `tf_dataset.batch(batch_size)`.
drop_remainder(`bool`, default `None`): Drop the last incomplete batch when loading. If not provided,
defaults to the same setting as shuffle.
num_workers (`int`): Number of workers to use for loading the dataset. Should be >= 1.
Returns:
`tf.data.Dataset`
np_get_batch(indices, dataset, cols_to_retain, collate_fn, collate_fn_args, columns_to_np_types, return_dict=False)
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\tf_utils.py
datasets.tqdm
Python Library Documentation: class tqdm in module datasets.utils.tqdm
class tqdm(tqdm.asyncio.tqdm_asyncio)
| tqdm(*args, **kwargs)
|
| Class to override `disable` argument in case progress bars are globally disabled.
|
| Taken from https://github.com/tqdm/tqdm/issues/619#issuecomment-619639324.
|
| Method resolution order:
| tqdm
| tqdm.asyncio.tqdm_asyncio
| tqdm.std.tqdm
| tqdm.utils.Comparable
| builtins.object
|
| Methods defined here:
|
| __delattr__(self, attr: str) -> None
| Fix for https://github.com/huggingface/datasets/issues/6066
|
| __init__(self, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from tqdm.asyncio.tqdm_asyncio:
|
| __aiter__(self)
|
| async __anext__(self)
|
| send(self, *args, **kwargs)
|
| ----------------------------------------------------------------------
| Class methods inherited from tqdm.asyncio.tqdm_asyncio:
|
| as_completed(fs, *, loop=None, timeout=None, total=None, **tqdm_kwargs)
| Wrapper for `asyncio.as_completed`.
|
| async gather(*fs, loop=None, timeout=None, total=None, **tqdm_kwargs)
| Wrapper for `asyncio.gather`.
|
| ----------------------------------------------------------------------
| Methods inherited from tqdm.std.tqdm:
|
| __bool__(self)
|
| __contains__(self, item)
|
| __del__(self)
|
| __enter__(self)
|
| __exit__(self, exc_type, exc_value, traceback)
|
| __hash__(self)
| Return hash(self).
|
| __iter__(self)
| Backward-compatibility to use: for x in tqdm(iterable)
|
| __len__(self)
|
| __reversed__(self)
|
| __str__(self)
| Return str(self).
|
| clear(self, nolock=False)
| Clear current bar display.
|
| close(self)
| Cleanup and (if leave=False) close the progressbar.
|
| display(self, msg=None, pos=None)
| Use `self.sp` to display `msg` in the specified `pos`.
|
| Consider overloading this function when inheriting to use e.g.:
| `self.some_frontend(**self.format_dict)` instead of `self.sp`.
|
| Parameters
| ----------
| msg : str, optional. What to display (default: `repr(self)`).
| pos : int, optional. Position to `moveto`
| (default: `abs(self.pos)`).
|
| moveto(self, n)
|
| refresh(self, nolock=False, lock_args=None)
| Force refresh the display of this bar.
|
| Parameters
| ----------
| nolock : bool, optional
| If `True`, does not lock.
| If [default: `False`]: calls `acquire()` on internal lock.
| lock_args : tuple, optional
| Passed to internal lock's `acquire()`.
| If specified, will only `display()` if `acquire()` returns `True`.
|
| reset(self, total=None)
| Resets to 0 iterations for repeated use.
|
| Consider combining with `leave=True`.
|
| Parameters
| ----------
| total : int or float, optional. Total to use for the new bar.
|
| set_description(self, desc=None, refresh=True)
| Set/modify description of the progress bar.
|
| Parameters
| ----------
| desc : str, optional
| refresh : bool, optional
| Forces refresh [default: True].
|
| set_description_str(self, desc=None, refresh=True)
| Set/modify description without ': ' appended.
|
| set_postfix(self, ordered_dict=None, refresh=True, **kwargs)
| Set/modify postfix (additional stats)
| with automatic formatting based on datatype.
|
| Parameters
| ----------
| ordered_dict : dict or OrderedDict, optional
| refresh : bool, optional
| Forces refresh [default: True].
| kwargs : dict, optional
|
| set_postfix_str(self, s='', refresh=True)
| Postfix without dictionary expansion, similar to prefix handling.
|
| unpause(self)
| Restart tqdm timer from last print time.
|
| update(self, n=1)
| Manually update the progress bar, useful for streams
| such as reading files.
| E.g.:
| >>> t = tqdm(total=filesize) # Initialise
| >>> for current_buffer in stream:
| ... ...
| ... t.update(len(current_buffer))
| >>> t.close()
| The last line is highly recommended, but possibly not necessary if
| `t.update()` will be called in such a way that `filesize` will be
| exactly reached and printed.
|
| Parameters
| ----------
| n : int or float, optional
| Increment to add to the internal counter of iterations
| [default: 1]. If using float, consider specifying `{n:.3f}`
| or similar in `bar_format`, or specifying `unit_scale`.
|
| Returns
| -------
| out : bool or None
| True if a `display()` was triggered.
|
| ----------------------------------------------------------------------
| Class methods inherited from tqdm.std.tqdm:
|
| external_write_mode(file=None, nolock=False)
| Disable tqdm within context and refresh tqdm when exits.
| Useful when writing to standard output stream
|
| get_lock()
| Get the global lock. Construct it if it does not exist.
|
| pandas(**tqdm_kwargs)
| Registers the current `tqdm` class with
| pandas.core.
| ( frame.DataFrame
| | series.Series
| | groupby.(generic.)DataFrameGroupBy
| | groupby.(generic.)SeriesGroupBy
| ).progress_apply
|
| A new instance will be created every time `progress_apply` is called,
| and each instance will automatically `close()` upon completion.
|
| Parameters
| ----------
| tqdm_kwargs : arguments for the tqdm instance
|
| Examples
| --------
| >>> import pandas as pd
| >>> import numpy as np
| >>> from tqdm import tqdm
| >>> from tqdm.gui import tqdm as tqdm_gui
| >>>
| >>> df = pd.DataFrame(np.random.randint(0, 100, (100000, 6)))
| >>> tqdm.pandas(ncols=50) # can use tqdm_gui, optional kwargs, etc
| >>> # Now you can use `progress_apply` instead of `apply`
| >>> df.groupby(0).progress_apply(lambda x: x**2)
|
| References
| ----------
| <https://stackoverflow.com/questions/18603270/ progress-indicator-during-pandas-operations-python>
|
| set_lock(lock)
| Set the global lock.
|
| wrapattr(stream, method, total=None, bytes=True, **tqdm_kwargs)
| stream : file-like object.
| method : str, "read" or "write". The result of `read()` and
| the first argument of `write()` should have a `len()`.
|
| >>> with tqdm.wrapattr(file_obj, "read", total=file_obj.size) as fobj:
| ... while True:
| ... chunk = fobj.read(chunk_size)
| ... if not chunk:
| ... break
|
| write(s, file=None, end='\n', nolock=False)
| Print a message via tqdm (without overlap with bars).
|
| ----------------------------------------------------------------------
| Static methods inherited from tqdm.std.tqdm:
|
| __new__(cls, *_, **__)
| Create and return a new object. See help(type) for accurate signature.
|
| format_interval(t)
| Formats a number of seconds as a clock time, [H:]MM:SS
|
| Parameters
| ----------
| t : int
| Number of seconds.
|
| Returns
| -------
| out : str
| [H:]MM:SS
|
| format_meter(n, total, elapsed, ncols=None, prefix='', ascii=False, unit='it', unit_scale=False, rate=None, bar_format=None, postfix=None, unit_divisor=1000, initial=0, colour=None, **extra_kwargs)
| Return a string-based progress bar given some parameters
|
| Parameters
| ----------
| n : int or float
| Number of finished iterations.
| total : int or float
| The expected total number of iterations. If meaningless (None),
| only basic progress statistics are displayed (no ETA).
| elapsed : float
| Number of seconds passed since start.
| ncols : int, optional
| The width of the entire output message. If specified,
| dynamically resizes `{bar}` to stay within this bound
| [default: None]. If `0`, will not print any bar (only stats).
| The fallback is `{bar:10}`.
| prefix : str, optional
| Prefix message (included in total width) [default: ''].
| Use as {desc} in bar_format string.
| ascii : bool, optional or str, optional
| If not set, use unicode (smooth blocks) to fill the meter
| [default: False]. The fallback is to use ASCII characters
| " 123456789#".
| unit : str, optional
| The iteration unit [default: 'it'].
| unit_scale : bool or int or float, optional
| If 1 or True, the number of iterations will be printed with an
| appropriate SI metric prefix (k = 10^3, M = 10^6, etc.)
| [default: False]. If any other non-zero number, will scale
| `total` and `n`.
| rate : float, optional
| Manual override for iteration rate.
| If [default: None], uses n/elapsed.
| bar_format : str, optional
| Specify a custom bar string formatting. May impact performance.
| [default: '{l_bar}{bar}{r_bar}'], where
| l_bar='{desc}: {percentage:3.0f}%|' and
| r_bar='| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, '
| '{rate_fmt}{postfix}]'
| Possible vars: l_bar, bar, r_bar, n, n_fmt, total, total_fmt,
| percentage, elapsed, elapsed_s, ncols, nrows, desc, unit,
| rate, rate_fmt, rate_noinv, rate_noinv_fmt,
| rate_inv, rate_inv_fmt, postfix, unit_divisor,
| remaining, remaining_s, eta.
| Note that a trailing ": " is automatically removed after {desc}
| if the latter is empty.
| postfix : *, optional
| Similar to `prefix`, but placed at the end
| (e.g. for additional stats).
| Note: postfix is usually a string (not a dict) for this method,
| and will if possible be set to postfix = ', ' + postfix.
| However other types are supported (#382).
| unit_divisor : float, optional
| [default: 1000], ignored unless `unit_scale` is True.
| initial : int or float, optional
| The initial counter value [default: 0].
| colour : str, optional
| Bar colour (e.g. 'green', '#00ff00').
|
| Returns
| -------
| out : Formatted meter and stats, ready to display.
|
| format_num(n)
| Intelligent scientific notation (.3g).
|
| Parameters
| ----------
| n : int or float or Numeric
| A Number.
|
| Returns
| -------
| out : str
| Formatted number.
|
| format_sizeof(num, suffix='', divisor=1000)
| Formats a number (greater than unity) with SI Order of Magnitude
| prefixes.
|
| Parameters
| ----------
| num : float
| Number ( >= 1) to format.
| suffix : str, optional
| Post-postfix [default: ''].
| divisor : float, optional
| Divisor between prefixes [default: 1000].
|
| Returns
| -------
| out : str
| Number with Order of Magnitude SI unit postfix.
|
| status_printer(file)
| Manage the printing and in-place updating of a line of characters.
| Note that if the string is longer than a line, then in-place
| updating may not work (it will print a new line at each refresh).
|
| ----------------------------------------------------------------------
| Readonly properties inherited from tqdm.std.tqdm:
|
| format_dict
| Public API for read-only member access.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from tqdm.std.tqdm:
|
| monitor = None
|
| monitor_interval = 10
|
| ----------------------------------------------------------------------
| Methods inherited from tqdm.utils.Comparable:
|
| __eq__(self, other)
| Return self==value.
|
| __ge__(self, other)
| Return self>=value.
|
| __gt__(self, other)
| Return self>value.
|
| __le__(self, other)
| Return self<=value.
|
| __lt__(self, other)
| Return self<value.
|
| __ne__(self, other)
| Return self!=value.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from tqdm.utils.Comparable:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
datasets.track
Python Library Documentation: module datasets.utils.track in datasets.utils
NAME
datasets.utils.track
CLASSES
builtins.list(builtins.object)
tracked_list
builtins.str(builtins.object)
tracked_str
collections.abc.Iterable(builtins.object)
TrackedIterableFromGenerator
class TrackedIterableFromGenerator(collections.abc.Iterable)
| TrackedIterableFromGenerator(generator, *args)
|
| Utility class to create an iterable from a generator function, in order to reset the generator when needed.
|
| Method resolution order:
| TrackedIterableFromGenerator
| collections.abc.Iterable
| builtins.object
|
| Methods defined here:
|
| __init__(self, generator, *args)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self)
|
| __reduce__(self)
| Helper for pickle.
|
| __repr__(self) -> str
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
class tracked_list(builtins.list)
| tracked_list(*args, **kwargs) -> None
|
| Method resolution order:
| tracked_list
| builtins.list
| builtins.object
|
| Methods defined here:
|
| __init__(self, *args, **kwargs) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> collections.abc.Iterator
| Implement iter(self).
|
| __repr__(self) -> str
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.list:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, index, /)
| Return self[index].
|
| __gt__(self, value, /)
| Return self>value.
|
| __iadd__(self, value, /)
| Implement self+=value.
|
| __imul__(self, value, /)
| Implement self*=value.
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __reversed__(self, /)
| Return a reverse iterator over the list.
|
| __rmul__(self, value, /)
| Return value*self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(self, /)
| Return the size of the list in memory, in bytes.
|
| append(self, object, /)
| Append object to the end of the list.
|
| clear(self, /)
| Remove all items from list.
|
| copy(self, /)
| Return a shallow copy of the list.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| extend(self, iterable, /)
| Extend list by appending elements from the iterable.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| insert(self, index, object, /)
| Insert object before index.
|
| pop(self, index=-1, /)
| Remove and return item at index (default last).
|
| Raises IndexError if list is empty or index is out of range.
|
| remove(self, value, /)
| Remove first occurrence of value.
|
| Raises ValueError if the value is not present.
|
| reverse(self, /)
| Reverse *IN PLACE*.
|
| sort(self, /, *, key=None, reverse=False)
| Sort the list in ascending order and return None.
|
| The sort is in-place (i.e. the list itself is modified) and stable (i.e. the
| order of two equal elements is maintained).
|
| If a key function is given, apply it once to each list item and sort them,
| ascending or descending, according to their function values.
|
| The reverse flag can be set to sort in descending order.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.list:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.list:
|
| __new__(*args, **kwargs) class method of builtins.list
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.list:
|
| __hash__ = None
class tracked_str(builtins.str)
| Method resolution order:
| tracked_str
| builtins.str
| builtins.object
|
| Methods defined here:
|
| __repr__(self) -> str
| Return repr(self).
|
| get_origin(self)
|
| set_origin(self, origin: str)
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| origins = {}
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.str:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __eq__(self, value, /)
| Return self==value.
|
| __format__(self, format_spec, /)
| Return a formatted version of the string as described by format_spec.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getitem__(self, key, /)
| Return self[key].
|
| __getnewargs__(...)
|
| __gt__(self, value, /)
| Return self>value.
|
| __hash__(self, /)
| Return hash(self).
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mod__(self, value, /)
| Return self%value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __rmod__(self, value, /)
| Return value%self.
|
| __rmul__(self, value, /)
| Return value*self.
|
| __sizeof__(self, /)
| Return the size of the string in memory, in bytes.
|
| __str__(self, /)
| Return str(self).
|
| capitalize(self, /)
| Return a capitalized version of the string.
|
| More specifically, make the first character have upper case and the rest lower
| case.
|
| casefold(self, /)
| Return a version of the string suitable for caseless comparisons.
|
| center(self, width, fillchar=' ', /)
| Return a centered string of length width.
|
| Padding is done using the specified fill character (default is a space).
|
| count(...)
| S.count(sub[, start[, end]]) -> int
|
| Return the number of non-overlapping occurrences of substring sub in
| string S[start:end]. Optional arguments start and end are
| interpreted as in slice notation.
|
| encode(self, /, encoding='utf-8', errors='strict')
| Encode the string using the codec registered for encoding.
|
| encoding
| The encoding in which to encode the string.
| errors
| The error handling scheme to use for encoding errors.
| The default is 'strict' meaning that encoding errors raise a
| UnicodeEncodeError. Other possible values are 'ignore', 'replace' and
| 'xmlcharrefreplace' as well as any other name registered with
| codecs.register_error that can handle UnicodeEncodeErrors.
|
| endswith(...)
| S.endswith(suffix[, start[, end]]) -> bool
|
| Return True if S ends with the specified suffix, False otherwise.
| With optional start, test S beginning at that position.
| With optional end, stop comparing S at that position.
| suffix can also be a tuple of strings to try.
|
| expandtabs(self, /, tabsize=8)
| Return a copy where all tab characters are expanded using spaces.
|
| If tabsize is not given, a tab size of 8 characters is assumed.
|
| find(...)
| S.find(sub[, start[, end]]) -> int
|
| Return the lowest index in S where substring sub is found,
| such that sub is contained within S[start:end]. Optional
| arguments start and end are interpreted as in slice notation.
|
| Return -1 on failure.
|
| format(...)
| S.format(*args, **kwargs) -> str
|
| Return a formatted version of S, using substitutions from args and kwargs.
| The substitutions are identified by braces ('{' and '}').
|
| format_map(...)
| S.format_map(mapping) -> str
|
| Return a formatted version of S, using substitutions from mapping.
| The substitutions are identified by braces ('{' and '}').
|
| index(...)
| S.index(sub[, start[, end]]) -> int
|
| Return the lowest index in S where substring sub is found,
| such that sub is contained within S[start:end]. Optional
| arguments start and end are interpreted as in slice notation.
|
| Raises ValueError when the substring is not found.
|
| isalnum(self, /)
| Return True if the string is an alpha-numeric string, False otherwise.
|
| A string is alpha-numeric if all characters in the string are alpha-numeric and
| there is at least one character in the string.
|
| isalpha(self, /)
| Return True if the string is an alphabetic string, False otherwise.
|
| A string is alphabetic if all characters in the string are alphabetic and there
| is at least one character in the string.
|
| isascii(self, /)
| Return True if all characters in the string are ASCII, False otherwise.
|
| ASCII characters have code points in the range U+0000-U+007F.
| Empty string is ASCII too.
|
| isdecimal(self, /)
| Return True if the string is a decimal string, False otherwise.
|
| A string is a decimal string if all characters in the string are decimal and
| there is at least one character in the string.
|
| isdigit(self, /)
| Return True if the string is a digit string, False otherwise.
|
| A string is a digit string if all characters in the string are digits and there
| is at least one character in the string.
|
| isidentifier(self, /)
| Return True if the string is a valid Python identifier, False otherwise.
|
| Call keyword.iskeyword(s) to test whether string s is a reserved identifier,
| such as "def" or "class".
|
| islower(self, /)
| Return True if the string is a lowercase string, False otherwise.
|
| A string is lowercase if all cased characters in the string are lowercase and
| there is at least one cased character in the string.
|
| isnumeric(self, /)
| Return True if the string is a numeric string, False otherwise.
|
| A string is numeric if all characters in the string are numeric and there is at
| least one character in the string.
|
| isprintable(self, /)
| Return True if the string is printable, False otherwise.
|
| A string is printable if all of its characters are considered printable in
| repr() or if it is empty.
|
| isspace(self, /)
| Return True if the string is a whitespace string, False otherwise.
|
| A string is whitespace if all characters in the string are whitespace and there
| is at least one character in the string.
|
| istitle(self, /)
| Return True if the string is a title-cased string, False otherwise.
|
| In a title-cased string, upper- and title-case characters may only
| follow uncased characters and lowercase characters only cased ones.
|
| isupper(self, /)
| Return True if the string is an uppercase string, False otherwise.
|
| A string is uppercase if all cased characters in the string are uppercase and
| there is at least one cased character in the string.
|
| join(self, iterable, /)
| Concatenate any number of strings.
|
| The string whose method is called is inserted in between each given string.
| The result is returned as a new string.
|
| Example: '.'.join(['ab', 'pq', 'rs']) -> 'ab.pq.rs'
|
| ljust(self, width, fillchar=' ', /)
| Return a left-justified string of length width.
|
| Padding is done using the specified fill character (default is a space).
|
| lower(self, /)
| Return a copy of the string converted to lowercase.
|
| lstrip(self, chars=None, /)
| Return a copy of the string with leading whitespace removed.
|
| If chars is given and not None, remove characters in chars instead.
|
| partition(self, sep, /)
| Partition the string into three parts using the given separator.
|
| This will search for the separator in the string. If the separator is found,
| returns a 3-tuple containing the part before the separator, the separator
| itself, and the part after it.
|
| If the separator is not found, returns a 3-tuple containing the original string
| and two empty strings.
|
| removeprefix(self, prefix, /)
| Return a str with the given prefix string removed if present.
|
| If the string starts with the prefix string, return string[len(prefix):].
| Otherwise, return a copy of the original string.
|
| removesuffix(self, suffix, /)
| Return a str with the given suffix string removed if present.
|
| If the string ends with the suffix string and that suffix is not empty,
| return string[:-len(suffix)]. Otherwise, return a copy of the original
| string.
|
| replace(self, old, new, count=-1, /)
| Return a copy with all occurrences of substring old replaced by new.
|
| count
| Maximum number of occurrences to replace.
| -1 (the default value) means replace all occurrences.
|
| If the optional argument count is given, only the first count occurrences are
| replaced.
|
| rfind(...)
| S.rfind(sub[, start[, end]]) -> int
|
| Return the highest index in S where substring sub is found,
| such that sub is contained within S[start:end]. Optional
| arguments start and end are interpreted as in slice notation.
|
| Return -1 on failure.
|
| rindex(...)
| S.rindex(sub[, start[, end]]) -> int
|
| Return the highest index in S where substring sub is found,
| such that sub is contained within S[start:end]. Optional
| arguments start and end are interpreted as in slice notation.
|
| Raises ValueError when the substring is not found.
|
| rjust(self, width, fillchar=' ', /)
| Return a right-justified string of length width.
|
| Padding is done using the specified fill character (default is a space).
|
| rpartition(self, sep, /)
| Partition the string into three parts using the given separator.
|
| This will search for the separator in the string, starting at the end. If
| the separator is found, returns a 3-tuple containing the part before the
| separator, the separator itself, and the part after it.
|
| If the separator is not found, returns a 3-tuple containing two empty strings
| and the original string.
|
| rsplit(self, /, sep=None, maxsplit=-1)
| Return a list of the substrings in the string, using sep as the separator string.
|
| sep
| The separator used to split the string.
|
| When set to None (the default value), will split on any whitespace
| character (including \n \r \t \f and spaces) and will discard
| empty strings from the result.
| maxsplit
| Maximum number of splits.
| -1 (the default value) means no limit.
|
| Splitting starts at the end of the string and works to the front.
|
| rstrip(self, chars=None, /)
| Return a copy of the string with trailing whitespace removed.
|
| If chars is given and not None, remove characters in chars instead.
|
| split(self, /, sep=None, maxsplit=-1)
| Return a list of the substrings in the string, using sep as the separator string.
|
| sep
| The separator used to split the string.
|
| When set to None (the default value), will split on any whitespace
| character (including \n \r \t \f and spaces) and will discard
| empty strings from the result.
| maxsplit
| Maximum number of splits.
| -1 (the default value) means no limit.
|
| Splitting starts at the front of the string and works to the end.
|
| Note, str.split() is mainly useful for data that has been intentionally
| delimited. With natural text that includes punctuation, consider using
| the regular expression module.
|
| splitlines(self, /, keepends=False)
| Return a list of the lines in the string, breaking at line boundaries.
|
| Line breaks are not included in the resulting list unless keepends is given and
| true.
|
| startswith(...)
| S.startswith(prefix[, start[, end]]) -> bool
|
| Return True if S starts with the specified prefix, False otherwise.
| With optional start, test S beginning at that position.
| With optional end, stop comparing S at that position.
| prefix can also be a tuple of strings to try.
|
| strip(self, chars=None, /)
| Return a copy of the string with leading and trailing whitespace removed.
|
| If chars is given and not None, remove characters in chars instead.
|
| swapcase(self, /)
| Convert uppercase characters to lowercase and lowercase characters to uppercase.
|
| title(self, /)
| Return a version of the string where each word is titlecased.
|
| More specifically, words start with uppercased characters and all remaining
| cased characters have lower case.
|
| translate(self, table, /)
| Replace each character in the string using the given translation table.
|
| table
| Translation table, which must be a mapping of Unicode ordinals to
| Unicode ordinals, strings, or None.
|
| The table must implement lookup/indexing via __getitem__, for instance a
| dictionary or list. If this operation raises LookupError, the character is
| left untouched. Characters mapped to None are deleted.
|
| upper(self, /)
| Return a copy of the string converted to uppercase.
|
| zfill(self, width, /)
| Pad a numeric string with zeros on the left, to fill a field of the given width.
|
| The string is never truncated.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.str:
|
| __new__(*args, **kwargs) class method of builtins.str
| Create and return a new object. See help(type) for accurate signature.
|
| maketrans(...)
| Return a translation table usable for str.translate().
|
| If there is only one argument, it must be a dictionary mapping Unicode
| ordinals (integers) or characters to Unicode ordinals, strings or None.
| Character keys will be then converted to ordinals.
| If there are two arguments, they must be strings of equal length, and
| in the resulting dictionary, each character in x will be mapped to the
| character at the same position in y. If there is a third argument, it
| must be a string, whose characters will be mapped to None in the result.
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\track.py
datasets.typing
Python Library Documentation: module datasets.utils.typing in datasets.utils
NAME
datasets.utils.typing
DATA
ListLike = typing.Union[list[~T], tuple[~T, ...]]
NestedDataStructureLike = typing.Union[~T, list[~T], dict[str, ~T]]
PathLike = typing.Union[str, bytes, os.PathLike]
T = ~T
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\typing.py
datasets.utils
Python Library Documentation: package datasets.utils in datasets
NAME
datasets.utils
DESCRIPTION
# Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
PACKAGE CONTENTS
_dataset_viewer
_dill
_filelock
deprecation_utils
doc_utils
experimental
extract
file_utils
filelock
hub
info_utils
json
logging
metadata
patching
py_utils
resources (package)
sharding
stratify
tf_utils
tqdm
track
typing
version
SUBMODULES
_tqdm
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\__init__.py
datasets.version
Python Library Documentation: module datasets.utils.version in datasets.utils
NAME
datasets.utils.version - Version utils.
CLASSES
builtins.object
Version
class Version(builtins.object)
| Version(version_str: str, description: Optional[str] = None, major: Union[str, int, NoneType] = None, minor: Union[str, int, NoneType] = None, patch: Union[str, int, NoneType] = None) -> None
|
| Dataset version `MAJOR.MINOR.PATCH`.
|
| Args:
| version_str (`str`):
| The dataset version.
| description (`str`):
| A description of what is new in this version.
| major (`str`):
| minor (`str`):
| patch (`str`):
|
| Example:
|
| ```py
| >>> VERSION = datasets.Version("1.0.0")
| ```
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __ge__(self, other) from functools
| Return a >= b. Computed by @total_ordering from (not a < b).
|
| __gt__(self, other) from functools
| Return a > b. Computed by @total_ordering from (not a < b) and (a != b).
|
| __hash__(self)
| Return hash(self).
|
| __init__(self, version_str: str, description: Optional[str] = None, major: Union[str, int, NoneType] = None, minor: Union[str, int, NoneType] = None, patch: Union[str, int, NoneType] = None) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __le__(self, other) from functools
| Return a <= b. Computed by @total_ordering from (a < b) or (a == b).
|
| __lt__(self, other)
| Return self<value.
|
| __post_init__(self)
|
| __repr__(self)
| Return repr(self).
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_dict(dic)
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| tuple
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'description': typing.Optional[str], 'major': typin...
|
| __dataclass_fields__ = {'description': Field(name='description',type=t...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __match_args__ = ('version_str', 'description', 'major', 'minor', 'pat...
|
| description = None
|
| major = None
|
| minor = None
|
| patch = None
DATA
Optional = typing.Optional
Optional[X] is equivalent to Union[X, None].
Union = typing.Union
Union type; Union[X, Y] means either X or Y.
On Python 3.10 and higher, the | operator
can also be used to denote unions;
X | Y means the same thing to the type checker as Union[X, Y].
To define a union, use e.g. Union[int, str]. Details:
- The arguments must be types and there must be at least one.
- None as an argument is a special case and is replaced by
type(None).
- Unions of unions are flattened, e.g.::
assert Union[Union[int, str], float] == Union[int, str, float]
- Unions of a single argument vanish, e.g.::
assert Union[int] == int # The constructor actually returns int
- Redundant arguments are skipped, e.g.::
assert Union[int, str, int] == Union[int, str]
- When comparing unions, the argument order is ignored, e.g.::
assert Union[int, str] == Union[str, int]
- You cannot subclass or instantiate a union.
- You can use Optional[X] as a shorthand for Union[X, None].
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\datasets\utils\version.py