Ask a question to get started
Enter to sendā¢Shift+Enter new line
Tokenizer( self, chunk_overlap: int, tokens_per_chunk: int, decode: Callable[[list
Overlap in tokens between chunks
Maximum number of tokens per chunk
Function to decode a list of token IDs to a string
Function to encode a string to a list of token IDs
Tokenizer data class.
