Class SentenceTokenizer

Hierarchy (view full)

tokenize.SentenceTokenizer
- SentenceTokenizer

Index

Constructors

Methods

stream tokenize

Constructors

constructor

new SentenceTokenizer(language?, minSentenceLength?, streamContextLength?): tokenize.basic.SentenceTokenizer
Parameters
- language: string = 'en-US'
- minSentenceLength: number = 20
- streamContextLength: number = 10
Returns tokenize.basic.SentenceTokenizer
Overrides SentenceTokenizer.constructor
- Defined in agents/src/tokenize/basic/basic.ts:20

Methods

stream

stream(language?): SentenceStream
Returns a SentenceStream that can be used to push strings and receive smaller segments.
Parameters
- Optional language: string
Returns SentenceStream
Overrides SentenceTokenizer.stream
- Defined in agents/src/tokenize/basic/basic.ts:35

tokenize

tokenize(text, language?): string[]
Parameters
- text: string
- Optional language: string
Returns string[]
Overrides SentenceTokenizer.tokenize
- Defined in agents/src/tokenize/basic/basic.ts:30