Class BufferedWordStream

Hierarchy (view full)

WordStream
- BufferedWordStream

Index

Constructors

Accessors

Methods

[asyncIterator] close endInput flush next pushText

Constructors

constructor

new BufferedWordStream(func, minTokenLength, minContextLength): BufferedWordStream
Parameters
- func: TokenizeFunc
- minTokenLength: number
- minContextLength: number
Returns BufferedWordStream
Overrides WordStream.constructor
- Defined in agents/src/tokenize/token_stream.ts:150

Accessors

closed

get closed(): boolean
Returns boolean
Inherited from WordStream.closed
- Defined in agents/src/tokenize/tokenizer.ts:101

Methods

[asyncIterator]

[asyncIterator](): WordStream
Returns WordStream
Inherited from WordStream.[asyncIterator]
- Defined in agents/src/tokenize/tokenizer.ts:149

close

close(): void
Close both the input and output of the tokenizer stream

Returns void
Overrides WordStream.close
- Defined in agents/src/tokenize/token_stream.ts:167

endInput

endInput(): void
Mark the input as ended and forbid additional pushes

Returns void
Overrides WordStream.endInput
- Defined in agents/src/tokenize/token_stream.ts:163

flush

flush(): void
Flush the tokenizer, causing it to process all pending text

Returns void
Overrides WordStream.flush
- Defined in agents/src/tokenize/token_stream.ts:159

next

next(): Promise<IteratorResult<TokenData, any>>
Returns Promise<IteratorResult<TokenData, any>>
Overrides WordStream.next
- Defined in agents/src/tokenize/token_stream.ts:171

pushText

pushText(text): void
Push a string of text to the tokenizer
Parameters
- text: string
Returns void
Overrides WordStream.pushText
- Defined in agents/src/tokenize/token_stream.ts:155