Splits the input text into smaller components by splitting text on tokens.
This method encodes the input text using a private _encode method, then
strips the start and stop token IDs from the encoded result. It returns the
processed segments as a list of strings.
| Name | Type | Description |
|---|---|---|
text* | str | The input text to be split. |
