# TokenTextSplitter

> **Class** in `langchain_text_splitters`

📖 [View in docs](https://reference.langchain.com/python/langchain-text-splitters/base/TokenTextSplitter)

Splitting text to tokens using model tokenizer.

## Signature

```python
TokenTextSplitter(
    self,
    encoding_name: str = 'gpt2',
    model_name: str | None = None,
    allowed_special: Literal['all'] | AbstractSet[str] | None = None,
    disallowed_special: Literal['all'] | Collection[str] = 'all',
    **kwargs: Any = {},
)
```

## Parameters

| Name | Type | Required | Description |
|------|------|----------|-------------|
| `encoding_name` | `str` | No | The name of the tiktoken encoding to use. (default: `'gpt2'`) |
| `model_name` | `str \| None` | No | The name of the model to use.  If provided, this will override the `encoding_name`. (default: `None`) |
| `allowed_special` | `Literal['all'] \| AbstractSet[str] \| None` | No | Special tokens that are allowed during encoding. (default: `None`) |
| `disallowed_special` | `Literal['all'] \| Collection[str]` | No | Special tokens that are disallowed during encoding. (default: `'all'`) |

## Extends

- `TextSplitter`

## Constructors

```python
__init__(
    self,
    encoding_name: str = 'gpt2',
    model_name: str | None = None,
    allowed_special: Literal['all'] | AbstractSet[str] | None = None,
    disallowed_special: Literal['all'] | Collection[str] = 'all',
    **kwargs: Any = {},
) -> None
```

| Name | Type |
|------|------|
| `encoding_name` | `str` |
| `model_name` | `str \| None` |
| `allowed_special` | `Literal['all'] \| AbstractSet[str] \| None` |
| `disallowed_special` | `Literal['all'] \| Collection[str]` |


## Methods

- [`split_text()`](https://reference.langchain.com/python/langchain-text-splitters/base/TokenTextSplitter/split_text)

---

[View source on GitHub](https://github.com/langchain-ai/langchain/blob/8fec4e7ceee2c368b068c49f9fed453276e210e7/libs/text-splitters/langchain_text_splitters/base.py#L304)