Class WordTokenizer

Hierarchy (view full)

tokenize.WordTokenizer
- WordTokenizer

Index

Constructors

Methods

stream tokenize

Constructors

constructor

new WordTokenizer(ignorePunctuation?): tokenize.basic.WordTokenizer
Parameters
- ignorePunctuation: boolean = true
Returns tokenize.basic.WordTokenizer
Overrides WordTokenizer.constructor
- Defined in agents/src/tokenize/basic/basic.ts:47

Methods

stream

stream(language?): WordStream
Returns a WordStream that can be used to push words and receive smaller segments.
Parameters
- Optional language: string
Returns WordStream
Overrides WordTokenizer.stream
- Defined in agents/src/tokenize/basic/basic.ts:58

tokenize

tokenize(text, language?): string[]
Parameters
- text: string
- Optional language: string
Returns string[]
Overrides WordTokenizer.tokenize
- Defined in agents/src/tokenize/basic/basic.ts:53