| Name | Type | Description |
|---|---|---|
encoding_name | str | Default: 'gpt2'The name of the tiktoken encoding to use. |
model_name | str | None | Default: NoneThe name of the model to use. If provided, this will override the |
allowed_special | Literal['all'] | AbstractSet[str] | None | Default: None |
disallowed_special | Literal['all'] | Collection[str] | Default: 'all' |
Text splitter that uses tiktoken encoder to count length.
Unlike the base implementation, this also seeds the constructor with the tiktoken configuration so the splitter tokenizes on the same encoding.
Special tokens that are allowed during encoding.
Special tokens that are disallowed during encoding.
