Ask a question to get started
Enter to send•Shift+Enter new line
split_text_on_tokens( *, text: str, tokenizer: Tokenizer ) -> list[str]
text
str
The input text to be split.
tokenizer
Tokenizer
The tokenizer to use for splitting.
Split incoming text and return chunks using tokenizer.
