Ask a question to get started
Enter to sendā¢Shift+Enter new line
tiktoken_enabled: bool = True
Set this to False to use HuggingFace transformers tokenization.
transformers
For non-OpenAI providers (OpenRouter, Ollama, vLLM, etc.), consider setting check_embedding_ctx_length=False instead, as it bypasses tokenization entirely.
check_embedding_ctx_length=False
