tiktoken API Reference
Full documentation for tiktoken (0.13.0)
tiktoken.Encoding
Python Library Documentation: class Encoding in module tiktoken.core
class Encoding(builtins.object)
| Encoding(name: 'str', *, pat_str: 'str', mergeable_ranks: 'dict[bytes, int]', special_tokens: 'dict[str, int]', explicit_n_vocab: 'int | None' = None)
|
| Methods defined here:
|
| __getstate__(self) -> 'object'
| Helper for pickle.
|
| __init__(self, name: 'str', *, pat_str: 'str', mergeable_ranks: 'dict[bytes, int]', special_tokens: 'dict[str, int]', explicit_n_vocab: 'int | None' = None)
| Creates an Encoding object.
|
| See openai_public.py for examples of how to construct an Encoding object.
|
| Args:
| name: The name of the encoding. It should be clear from the name of the encoding
| what behaviour to expect, in particular, encodings with different special tokens
| should have different names.
| pat_str: A regex pattern string that is used to split the input text.
| mergeable_ranks: A dictionary mapping mergeable token bytes to their ranks. The ranks
| must correspond to merge priority.
| special_tokens: A dictionary mapping special token strings to their token values.
| explicit_n_vocab: The number of tokens in the vocabulary. If provided, it is checked
| that the number of mergeable tokens and special tokens is equal to this number.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, value: 'object') -> 'None'
|
| decode(self, tokens: 'Sequence[int]', errors: 'str' = 'replace') -> 'str'
| Decodes a list of tokens into a string.
|
| WARNING: the default behaviour of this function is lossy, since decoded bytes are not
| guaranteed to be valid UTF-8. You can control this behaviour using the `errors` parameter,
| for instance, setting `errors=strict`.
|
| ```
| >>> enc.decode([31373, 995])
| 'hello world'
| ```
|
| decode_batch(self, batch: 'Sequence[Sequence[int]]', *, errors: 'str' = 'replace', num_threads: 'int' = 8) -> 'list[str]'
| Decodes a batch (list of lists of tokens) into a list of strings.
|
| decode_bytes(self, tokens: 'Sequence[int]') -> 'bytes'
| Decodes a list of tokens into bytes.
|
| ```
| >>> enc.decode_bytes([31373, 995])
| b'hello world'
| ```
|
| decode_bytes_batch(self, batch: 'Sequence[Sequence[int]]', *, num_threads: 'int' = 8) -> 'list[bytes]'
| Decodes a batch (list of lists of tokens) into a list of bytes.
|
| decode_single_token_bytes(self, token: 'int') -> 'bytes'
| Decodes a token into bytes.
|
| NOTE: this will decode all special tokens.
|
| Raises `KeyError` if the token is not in the vocabulary.
|
| ```
| >>> enc.decode_single_token_bytes(31373)
| b'hello'
| ```
|
| decode_tokens_bytes(self, tokens: 'Sequence[int]') -> 'list[bytes]'
| Decodes a list of tokens into a list of bytes.
|
| Useful for visualising tokenisation.
| >>> enc.decode_tokens_bytes([31373, 995])
| [b'hello', b' world']
|
| decode_with_offsets(self, tokens: 'Sequence[int]') -> 'tuple[str, list[int]]'
| Decodes a list of tokens into a string and a list of offsets.
|
| Each offset is the index into text corresponding to the start of each token.
| If UTF-8 character boundaries do not line up with token boundaries, the offset is the index
| of the first character that contains bytes from the token.
|
| This will currently raise if given tokens that decode to invalid UTF-8; this behaviour may
| change in the future to be more permissive.
|
| >>> enc.decode_with_offsets([31373, 995])
| ('hello world', [0, 5])
|
| encode(self, text: 'str', *, allowed_special: "Literal['all'] | AbstractSet[str]" = set(), disallowed_special: "Literal['all'] | Collection[str]" = 'all') -> 'list[int]'
| Encodes a string into tokens.
|
| Special tokens are artificial tokens used to unlock capabilities from a model,
| such as fill-in-the-middle. So we want to be careful about accidentally encoding special
| tokens, since they can be used to trick a model into doing something we don't want it to do.
|
| Hence, by default, encode will raise an error if it encounters text that corresponds
| to a special token. This can be controlled on a per-token level using the `allowed_special`
| and `disallowed_special` parameters. In particular:
| - Setting `disallowed_special` to () will prevent this function from raising errors and
| cause all text corresponding to special tokens to be encoded as natural text.
| - Setting `allowed_special` to "all" will cause this function to treat all text
| corresponding to special tokens to be encoded as special tokens.
|
| ```
| >>> enc.encode("hello world")
| [31373, 995]
| >>> enc.encode("<|endoftext|>", allowed_special={"<|endoftext|>"})
| [50256]
| >>> enc.encode("<|endoftext|>", allowed_special="all")
| [50256]
| >>> enc.encode("<|endoftext|>")
| # Raises ValueError
| >>> enc.encode("<|endoftext|>", disallowed_special=())
| [27, 91, 437, 1659, 5239, 91, 29]
| ```
|
| encode_batch(self, text: 'list[str]', *, num_threads: 'int' = 8, allowed_special: "Literal['all'] | AbstractSet[str]" = set(), disallowed_special: "Literal['all'] | Collection[str]" = 'all') -> 'list[list[int]]'
| Encodes a list of strings into tokens, in parallel.
|
| See `encode` for more details on `allowed_special` and `disallowed_special`.
|
| ```
| >>> enc.encode_batch(["hello world", "goodbye world"])
| [[31373, 995], [11274, 16390, 995]]
| ```
|
| encode_ordinary(self, text: 'str') -> 'list[int]'
| Encodes a string into tokens, ignoring special tokens.
|
| This is equivalent to `encode(text, disallowed_special=())` (but slightly faster).
|
| ```
| >>> enc.encode_ordinary("hello world")
| [31373, 995]
|
| encode_ordinary_batch(self, text: 'list[str]', *, num_threads: 'int' = 8) -> 'list[list[int]]'
| Encodes a list of strings into tokens, in parallel, ignoring special tokens.
|
| This is equivalent to `encode_batch(text, disallowed_special=())` (but slightly faster).
|
| ```
| >>> enc.encode_ordinary_batch(["hello world", "goodbye world"])
| [[31373, 995], [11274, 16390, 995]]
| ```
|
| encode_single_token(self, text_or_bytes: 'str | bytes') -> 'int'
| Encodes text corresponding to a single token to its token value.
|
| NOTE: this will encode all special tokens.
|
| Raises `KeyError` if the token is not in the vocabulary.
|
| ```
| >>> enc.encode_single_token("hello")
| 31373
| ```
|
| encode_to_numpy(self, text: 'str', *, allowed_special: "Literal['all'] | AbstractSet[str]" = set(), disallowed_special: "Literal['all'] | Collection[str]" = 'all') -> 'npt.NDArray[np.uint32]'
| Encodes a string into tokens, returning a numpy array.
|
| Avoids the overhead of copying the token buffer into a Python list.
|
| encode_with_unstable(self, text: 'str', *, allowed_special: "Literal['all'] | AbstractSet[str]" = set(), disallowed_special: "Literal['all'] | Collection[str]" = 'all') -> 'tuple[list[int], list[list[int]]]'
| Encodes a string into stable tokens and possible completion sequences.
|
| Note that the stable tokens will only represent a substring of `text`.
|
| See `encode` for more details on `allowed_special` and `disallowed_special`.
|
| This API should itself be considered unstable.
|
| ```
| >>> enc.encode_with_unstable("hello fanta")
| ([31373], [(277, 4910), (5113, 265), ..., (8842,)])
|
| >>> text = "..."
| >>> stable_tokens, completions = enc.encode_with_unstable(text)
| >>> assert text.encode().startswith(enc.decode_bytes(stable_tokens))
| >>> assert all(enc.decode_bytes(stable_tokens + seq).startswith(text.encode()) for seq in completions)
| ```
|
| is_special_token(self, token: 'int') -> 'bool'
|
| special_tokens_set = <functools.cached_property object>
| token_byte_values(self) -> 'list[bytes]'
| Returns the list of all token byte values.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| eot_token
|
| n_vocab
| For backwards compatibility. Prefer to use `enc.max_token_value + 1`.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
tiktoken.core
Python Library Documentation: module tiktoken.core in tiktoken
NAME
tiktoken.core
CLASSES
builtins.object
Encoding
class Encoding(builtins.object)
| Encoding(name: 'str', *, pat_str: 'str', mergeable_ranks: 'dict[bytes, int]', special_tokens: 'dict[str, int]', explicit_n_vocab: 'int | None' = None)
|
| Methods defined here:
|
| __getstate__(self) -> 'object'
| Helper for pickle.
|
| __init__(self, name: 'str', *, pat_str: 'str', mergeable_ranks: 'dict[bytes, int]', special_tokens: 'dict[str, int]', explicit_n_vocab: 'int | None' = None)
| Creates an Encoding object.
|
| See openai_public.py for examples of how to construct an Encoding object.
|
| Args:
| name: The name of the encoding. It should be clear from the name of the encoding
| what behaviour to expect, in particular, encodings with different special tokens
| should have different names.
| pat_str: A regex pattern string that is used to split the input text.
| mergeable_ranks: A dictionary mapping mergeable token bytes to their ranks. The ranks
| must correspond to merge priority.
| special_tokens: A dictionary mapping special token strings to their token values.
| explicit_n_vocab: The number of tokens in the vocabulary. If provided, it is checked
| that the number of mergeable tokens and special tokens is equal to this number.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setstate__(self, value: 'object') -> 'None'
|
| decode(self, tokens: 'Sequence[int]', errors: 'str' = 'replace') -> 'str'
| Decodes a list of tokens into a string.
|
| WARNING: the default behaviour of this function is lossy, since decoded bytes are not
| guaranteed to be valid UTF-8. You can control this behaviour using the `errors` parameter,
| for instance, setting `errors=strict`.
|
| ```
| >>> enc.decode([31373, 995])
| 'hello world'
| ```
|
| decode_batch(self, batch: 'Sequence[Sequence[int]]', *, errors: 'str' = 'replace', num_threads: 'int' = 8) -> 'list[str]'
| Decodes a batch (list of lists of tokens) into a list of strings.
|
| decode_bytes(self, tokens: 'Sequence[int]') -> 'bytes'
| Decodes a list of tokens into bytes.
|
| ```
| >>> enc.decode_bytes([31373, 995])
| b'hello world'
| ```
|
| decode_bytes_batch(self, batch: 'Sequence[Sequence[int]]', *, num_threads: 'int' = 8) -> 'list[bytes]'
| Decodes a batch (list of lists of tokens) into a list of bytes.
|
| decode_single_token_bytes(self, token: 'int') -> 'bytes'
| Decodes a token into bytes.
|
| NOTE: this will decode all special tokens.
|
| Raises `KeyError` if the token is not in the vocabulary.
|
| ```
| >>> enc.decode_single_token_bytes(31373)
| b'hello'
| ```
|
| decode_tokens_bytes(self, tokens: 'Sequence[int]') -> 'list[bytes]'
| Decodes a list of tokens into a list of bytes.
|
| Useful for visualising tokenisation.
| >>> enc.decode_tokens_bytes([31373, 995])
| [b'hello', b' world']
|
| decode_with_offsets(self, tokens: 'Sequence[int]') -> 'tuple[str, list[int]]'
| Decodes a list of tokens into a string and a list of offsets.
|
| Each offset is the index into text corresponding to the start of each token.
| If UTF-8 character boundaries do not line up with token boundaries, the offset is the index
| of the first character that contains bytes from the token.
|
| This will currently raise if given tokens that decode to invalid UTF-8; this behaviour may
| change in the future to be more permissive.
|
| >>> enc.decode_with_offsets([31373, 995])
| ('hello world', [0, 5])
|
| encode(self, text: 'str', *, allowed_special: "Literal['all'] | AbstractSet[str]" = set(), disallowed_special: "Literal['all'] | Collection[str]" = 'all') -> 'list[int]'
| Encodes a string into tokens.
|
| Special tokens are artificial tokens used to unlock capabilities from a model,
| such as fill-in-the-middle. So we want to be careful about accidentally encoding special
| tokens, since they can be used to trick a model into doing something we don't want it to do.
|
| Hence, by default, encode will raise an error if it encounters text that corresponds
| to a special token. This can be controlled on a per-token level using the `allowed_special`
| and `disallowed_special` parameters. In particular:
| - Setting `disallowed_special` to () will prevent this function from raising errors and
| cause all text corresponding to special tokens to be encoded as natural text.
| - Setting `allowed_special` to "all" will cause this function to treat all text
| corresponding to special tokens to be encoded as special tokens.
|
| ```
| >>> enc.encode("hello world")
| [31373, 995]
| >>> enc.encode("<|endoftext|>", allowed_special={"<|endoftext|>"})
| [50256]
| >>> enc.encode("<|endoftext|>", allowed_special="all")
| [50256]
| >>> enc.encode("<|endoftext|>")
| # Raises ValueError
| >>> enc.encode("<|endoftext|>", disallowed_special=())
| [27, 91, 437, 1659, 5239, 91, 29]
| ```
|
| encode_batch(self, text: 'list[str]', *, num_threads: 'int' = 8, allowed_special: "Literal['all'] | AbstractSet[str]" = set(), disallowed_special: "Literal['all'] | Collection[str]" = 'all') -> 'list[list[int]]'
| Encodes a list of strings into tokens, in parallel.
|
| See `encode` for more details on `allowed_special` and `disallowed_special`.
|
| ```
| >>> enc.encode_batch(["hello world", "goodbye world"])
| [[31373, 995], [11274, 16390, 995]]
| ```
|
| encode_ordinary(self, text: 'str') -> 'list[int]'
| Encodes a string into tokens, ignoring special tokens.
|
| This is equivalent to `encode(text, disallowed_special=())` (but slightly faster).
|
| ```
| >>> enc.encode_ordinary("hello world")
| [31373, 995]
|
| encode_ordinary_batch(self, text: 'list[str]', *, num_threads: 'int' = 8) -> 'list[list[int]]'
| Encodes a list of strings into tokens, in parallel, ignoring special tokens.
|
| This is equivalent to `encode_batch(text, disallowed_special=())` (but slightly faster).
|
| ```
| >>> enc.encode_ordinary_batch(["hello world", "goodbye world"])
| [[31373, 995], [11274, 16390, 995]]
| ```
|
| encode_single_token(self, text_or_bytes: 'str | bytes') -> 'int'
| Encodes text corresponding to a single token to its token value.
|
| NOTE: this will encode all special tokens.
|
| Raises `KeyError` if the token is not in the vocabulary.
|
| ```
| >>> enc.encode_single_token("hello")
| 31373
| ```
|
| encode_to_numpy(self, text: 'str', *, allowed_special: "Literal['all'] | AbstractSet[str]" = set(), disallowed_special: "Literal['all'] | Collection[str]" = 'all') -> 'npt.NDArray[np.uint32]'
| Encodes a string into tokens, returning a numpy array.
|
| Avoids the overhead of copying the token buffer into a Python list.
|
| encode_with_unstable(self, text: 'str', *, allowed_special: "Literal['all'] | AbstractSet[str]" = set(), disallowed_special: "Literal['all'] | Collection[str]" = 'all') -> 'tuple[list[int], list[list[int]]]'
| Encodes a string into stable tokens and possible completion sequences.
|
| Note that the stable tokens will only represent a substring of `text`.
|
| See `encode` for more details on `allowed_special` and `disallowed_special`.
|
| This API should itself be considered unstable.
|
| ```
| >>> enc.encode_with_unstable("hello fanta")
| ([31373], [(277, 4910), (5113, 265), ..., (8842,)])
|
| >>> text = "..."
| >>> stable_tokens, completions = enc.encode_with_unstable(text)
| >>> assert text.encode().startswith(enc.decode_bytes(stable_tokens))
| >>> assert all(enc.decode_bytes(stable_tokens + seq).startswith(text.encode()) for seq in completions)
| ```
|
| is_special_token(self, token: 'int') -> 'bool'
|
| special_tokens_set = <functools.cached_property object>
| token_byte_values(self) -> 'list[bytes]'
| Returns the list of all token byte values.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| eot_token
|
| n_vocab
| For backwards compatibility. Prefer to use `enc.max_token_value + 1`.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
FUNCTIONS
raise_disallowed_special_token(token: 'str') -> 'NoReturn'
DATA
AbstractSet = typing.AbstractSet
A generic version of collections.abc.Set.
Collection = typing.Collection
A generic version of collections.abc.Collection.
Literal = typing.Literal
Special typing form to define literal types (a.k.a. value types).
This form can be used to indicate to type checkers that the corresponding
variable or function parameter has a value equivalent to the provided
literal (or one of several literals)::
def validate_simple(data: Any) -> Literal[True]: # always returns True
...
MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
...
open_helper('/some/path', 'r') # Passes type check
open_helper('/other/path', 'typo') # Error in type checker
Literal[...] cannot be subclassed. At runtime, an arbitrary value
is allowed as type argument to Literal[...], but type checkers may
impose restrictions.
NoReturn = typing.NoReturn
Special type indicating functions that never return.
Example::
from typing import NoReturn
def stop() -> NoReturn:
raise Exception('no way')
NoReturn can also be used as a bottom type, a type that
has no values. Starting in Python 3.11, the Never type should
be used for this concept instead. Type checkers should treat the two
equivalently.
Sequence = typing.Sequence
A generic version of collections.abc.Sequence.
TYPE_CHECKING = False
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\tiktoken\core.py
tiktoken.encoding_for_model
Python Library Documentation: function encoding_for_model in module tiktoken.model
encoding_for_model(model_name: 'str') -> 'Encoding'
Returns the encoding used by a model.
Raises a KeyError if the model name is not recognised.
tiktoken.encoding_name_for_model
Python Library Documentation: function encoding_name_for_model in module tiktoken.model
encoding_name_for_model(model_name: 'str') -> 'str'
Returns the name of the encoding used by a model.
Raises a KeyError if the model name is not recognised.
tiktoken.get_encoding
Python Library Documentation: function get_encoding in module tiktoken.registry get_encoding(encoding_name: 'str') -> 'Encoding'
tiktoken.list_encoding_names
Python Library Documentation: function list_encoding_names in module tiktoken.registry list_encoding_names() -> 'list[str]'
tiktoken.model
Python Library Documentation: module tiktoken.model in tiktoken
NAME
tiktoken.model
FUNCTIONS
encoding_for_model(model_name: 'str') -> 'Encoding'
Returns the encoding used by a model.
Raises a KeyError if the model name is not recognised.
encoding_name_for_model(model_name: 'str') -> 'str'
Returns the name of the encoding used by a model.
Raises a KeyError if the model name is not recognised.
DATA
MODEL_PREFIX_TO_ENCODING = {'chatgpt-4o-': 'o200k_base', 'ft:babbage-0...
MODEL_TO_ENCODING = {'ada': 'r50k_base', 'babbage': 'r50k_base', 'babb...
__annotations__ = {'MODEL_PREFIX_TO_ENCODING': 'dict[str, str]', 'MODE...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\tiktoken\model.py
tiktoken.registry
Python Library Documentation: module tiktoken.registry in tiktoken
NAME
tiktoken.registry
FUNCTIONS
get_encoding(encoding_name: 'str') -> 'Encoding'
list_encoding_names() -> 'list[str]'
DATA
Callable = typing.Callable
Deprecated alias to collections.abc.Callable.
Callable[[int], str] signifies a function that takes a single
parameter of type int and returns a str.
The subscription syntax must always be used with exactly two
values: the argument list and the return type.
The argument list must be a list of types, a ParamSpec,
Concatenate or ellipsis. The return type must be a single type.
There is no syntax to indicate optional or keyword arguments;
such function types are rarely used as callback types.
ENCODINGS = {}
ENCODING_CONSTRUCTORS = None
Sequence = typing.Sequence
A generic version of collections.abc.Sequence.
__annotations__ = {'ENCODINGS': 'dict[str, Encoding]', 'ENCODING_CONST...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\tiktoken\registry.py