Class SentenceTokenizer`Abstract`

Hierarchy (view full)

SentenceTokenizer
- tokenize.basic.SentenceTokenizer

Index

Constructors

Methods

stream tokenize

Constructors

constructor

new SentenceTokenizer(): tokenize.SentenceTokenizer
Returns tokenize.SentenceTokenizer

Methods

`Abstract` stream

stream(): SentenceStream
Returns a SentenceStream that can be used to push strings and receive smaller segments.

Returns SentenceStream
- Defined in agents/src/tokenize/tokenizer.ts:24

`Abstract` tokenize

tokenize(text, language?): string[]
Parameters
- text: string
- Optional language: string
Returns string[]
- Defined in agents/src/tokenize/tokenizer.ts:19