Find the best answer to your question, help others answer theirs

If you are going to use a passage of Lorem Ipsum, you need to be sure there
isn't anything embarrassing hidden in the middle of text.

  • Anybody can ask a question
  • Anybody can answer
  • The best answers are voted up and rise to the top
What is the use of tokenizer?
Asked 8 वर्ष पहले
Viewed 733 times

1 Answer

2
It is used to split a stream of text into a series of tokens, where each token is a subsequence of characters in the text. The token produced are then passed through Token Filters that can add, remove or update the tokens. Later,that field is indexed by the resulting token stream.
Edit
avatar
Majed Badawi
15.5k 3 10 26
answered 8 hours ago

Leave a Comment

[named hyperlinks] (https://example.com)
**bold**
_italic_

Your Answer

Drop files here or click to upload.