Interface IStandardTokenizer

A tokenizer of type standard providing grammar based tokenizer that is a good tokenizer for most European language documents.

The tokenizer implements the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

Inherited Members
Namespace: OpenSearch.Client
Assembly: OpenSearch.Client.dll
Syntax
public interface IStandardTokenizer : ITokenizer

Properties

| Edit this page View Source

MaxTokenLength

The maximum token length. If a token is seen that exceeds this length then it is discarded. Defaults to 255.

Declaration
[DataMember(Name = "max_token_length")]
int? MaxTokenLength { get; set; }
Property Value
Type Description
int?

Extension Methods