Text splitter that uses Hugging Face tokenizer to count length.
from_huggingface_tokenizer(
cls,
tokenizer: PreTrainedTokenizerBase,
**kwargs: Any = {}
) -> TextSplitter| Name | Type | Description |
|---|---|---|
tokenizer* | PreTrainedTokenizerBase | The Hugging Face tokenizer to use. |