Class BufferedTokenStream

Implements

AsyncIterableIterator<TokenData>

Index

Constructors

Methods

[asyncIterator] close endInput flush next pushText

Constructors

constructor

new BufferedTokenStream(func, minTokenLength, minContextLength): BufferedTokenStream
Parameters
- func: TokenizeFunc
- minTokenLength: number
- minContextLength: number
Returns BufferedTokenStream
- Defined in tokenize/token_stream.ts:22

Methods

[asyncIterator]

[asyncIterator](): BufferedTokenStream
Returns BufferedTokenStream
Implementation of AsyncIterableIterator.[asyncIterator]
- Defined in tokenize/token_stream.ts:118

close

close(): void
Close both the input and output of the token stream

Returns void
- Defined in tokenize/token_stream.ts:113

endInput

endInput(): void
Mark the input as ended and forbid additional pushes

Returns void
- Defined in tokenize/token_stream.ts:100

flush

flush(): void
Flush the stream, causing it to process all pending text

Returns void
- Defined in tokenize/token_stream.ts:71

next

next(): Promise<IteratorResult<TokenData, any>>
Returns Promise<IteratorResult<TokenData, any>>
Implementation of AsyncIterableIterator.next
- Defined in tokenize/token_stream.ts:108

pushText

pushText(text): void
Push a string of text into the token stream
Parameters
- text: string
Returns void
- Defined in tokenize/token_stream.ts:31