Class TokenTextSplitter

Implementation of splitter which looks at tokens.

Hierarchy (View Summary)

TextSplitter
- TokenTextSplitter

Implements

TokenTextSplitterParams

Index

Constructors

Properties

allowedSpecial chunkOverlap chunkSize disallowedSpecial encodingName keepSeparator lc_namespace lengthFunction

Methods

createDocuments mergeSplits splitDocuments splitOnSeparator splitText transformDocuments lc_name

Constructors

constructor

new TokenTextSplitter(
fields?: Partial<TokenTextSplitterParams>,
): TokenTextSplitter
Parameters
- Optionalfields: Partial<TokenTextSplitterParams>
Returns TokenTextSplitter
Overrides TextSplitter.constructor
- Defined in text_splitter.ts:738

Properties

allowedSpecial

allowedSpecial: string[] | "all"

chunkOverlap

chunkOverlap: number = 200

chunkSize

chunkSize: number = 1000

disallowedSpecial

disallowedSpecial: string[] | "all"

encodingName

encodingName: TiktokenEncoding

keepSeparator

keepSeparator: boolean = false

lc_namespace

lc_namespace: string[] = ...

lengthFunction

lengthFunction: ((text: string) => number) | ((text: string) => Promise<number>)

Methods

createDocuments

createDocuments(
    texts: string[],
    metadatas?: Record<string, any>[],
    chunkHeaderOptions?: TextSplitterChunkHeaderOptions,
): Promise<Document[]>
Parameters
- texts: string[]
- metadatas: Record<string, any>[] = []
- chunkHeaderOptions: TextSplitterChunkHeaderOptions = {}
Returns Promise<Document[]>
Inherited from TextSplitter.createDocuments
- Defined in text_splitter.ts:75

mergeSplits

mergeSplits(splits: string[], separator: string): Promise<string[]>
Parameters
- splits: string[]
- separator: string
Returns Promise<string[]>
Inherited from TextSplitter.mergeSplits
- Defined in text_splitter.ts:184

splitDocuments

splitDocuments(
documents: Document[],
chunkHeaderOptions?: TextSplitterChunkHeaderOptions,
): Promise<Document[]>
Parameters
- documents: Document[]
- chunkHeaderOptions: TextSplitterChunkHeaderOptions = {}
Returns Promise<Document[]>
Inherited from TextSplitter.splitDocuments
- Defined in text_splitter.ts:167

`Protected`splitOnSeparator

splitOnSeparator(text: string, separator: string): string[]
Parameters
- text: string
- separator: string
Returns string[]
Inherited from TextSplitter.splitOnSeparator
- Defined in text_splitter.ts:57

splitText

splitText(text: string): Promise<string[]>
Parameters
- text: string
Returns Promise<string[]>
Overrides TextSplitter.splitText
- Defined in text_splitter.ts:746

transformDocuments

transformDocuments(
documents: Document[],
chunkHeaderOptions?: TextSplitterChunkHeaderOptions,
): Promise<Document[]>
Parameters
- documents: Document[]
- chunkHeaderOptions: TextSplitterChunkHeaderOptions = {}
Returns Promise<Document[]>
Inherited from TextSplitter.transformDocuments
- Defined in text_splitter.ts:48

`Static`lc_name

lc_name(): string
Returns string
- Defined in text_splitter.ts:726