Class WordTokenizer`Abstract`

Hierarchy (view full)

WordTokenizer
- tokenize.basic.WordTokenizer

Index

Constructors

Methods

stream tokenize

Constructors

constructor

new WordTokenizer(): tokenize.WordTokenizer
Returns tokenize.WordTokenizer

Methods

`Abstract` stream

stream(): WordStream
Returns a WordStream that can be used to push words and receive smaller segments.

Returns WordStream
- Defined in agents/src/tokenize/tokenizer.ts:92

`Abstract` tokenize

tokenize(text, language?): string[]
Parameters
- text: string
- Optional language: string
Returns string[]
- Defined in agents/src/tokenize/tokenizer.ts:87