# Tokenizer

> **Class** in `langchain_text_splitters`

📖 [View in docs](https://reference.langchain.com/python/langchain-text-splitters/base/Tokenizer)

Tokenizer data class.

## Signature

```python
Tokenizer(
    self,
    chunk_overlap: int,
    tokens_per_chunk: int,
    decode: Callable[[list[int]], str],
    encode: Callable[[str], list[int]],
)
```

## Constructors

```python
__init__(
    self,
    chunk_overlap: int,
    tokens_per_chunk: int,
    decode: Callable[[list[int]], str],
    encode: Callable[[str], list[int]],
) -> None
```

| Name | Type |
|------|------|
| `chunk_overlap` | `int` |
| `tokens_per_chunk` | `int` |
| `decode` | `Callable[[list[int]], str]` |
| `encode` | `Callable[[str], list[int]]` |


## Properties

- `chunk_overlap`
- `tokens_per_chunk`
- `decode`
- `encode`

---

[View source on GitHub](https://github.com/langchain-ai/langchain/blob/fb6ab993a73180538f6cca876b3c85d46c08845f/libs/text-splitters/langchain_text_splitters/base.py#L413)