LangChain Reference home pageLangChain ReferenceLangChain Reference
  • GitHub
  • Main Docs
Deep Agents
LangChain
LangGraph
Integrations
LangSmith
  • Overview
    • Overview
    • Caches
    • Callbacks
    • Documents
    • Document loaders
    • Embeddings
    • Exceptions
    • Language models
    • Serialization
    • Output parsers
    • Prompts
    • Rate limiters
    • Retrievers
    • Runnables
    • Utilities
    • Vector stores
    MCP Adapters
    Standard Tests
    Text Splitters
    โŒ˜I

    Chat LangChain

    What can I help with?

    Enter to sendยทShift+Enter new line

    Menu

    OverviewCachesCallbacksDocumentsDocument loadersEmbeddingsExceptionsLanguage modelsSerializationOutput parsersPromptsRate limitersRetrieversRunnablesUtilitiesVector stores
    MCP Adapters
    Standard Tests
    Text Splitters
    Language
    Theme
    Pythonlangchain-coreutilsmustachetokenize
    Functionโ—Since v1.8

    tokenize

    Copy
    tokenize(
      template: str,
      def_ldel: str = '{{',
        def_rdel: str = '}}'
    View source on GitHub
    )
    ->
    Iterator
    [
    tuple
    [
    str
    ,
    str
    ]
    ]

    Parameters

    NameTypeDescription
    template*str
    def_ldelstr
    Default:'{{'
    def_rdelstr
    Default:'}}'

    Tokenize a mustache template.

    Tokenizes a mustache template in a generator fashion, using file-like objects. It also accepts a string containing the template.

    a file-like object, or a string of a mustache template

    The default left delimiter ('{{' by default, as in spec compliant mustache)

    The default right delimiter ('}}' by default, as in spec compliant mustache)