Get the number of tokens present in the text.
Useful for checking if an input fits in a model's context window.
This should be overridden by model-specific implementations to provide accurate token counts via model-specific tokenizers.
| Name | Type | Description |
|---|---|---|
text* | str | The string input to tokenize. |
