ES BM25 TF-IDF相似度算法设置——
2017-02-27 11:13
871 查看
Pluggable Similarity Algorithms
Before we move on from relevance and scoring, we will finish this chapter with a more advanced subject: pluggable similarity algorithms. While Elasticsearch uses the Lucene’s Practical Scoring Function as its default similarity algorithm, it supports other algorithms out of the box, which are listed in the Similarity Modules documentation.
Both use term frequency, inverse document frequency, and field-length normalization, but the definition of each of these factors is a little different. Rather than explaining the BM25 formula in detail, we will focus on the practical advantages that BM25 offers.
However, common words occur commonly. The fact that a common word appears many times in one document is offset by the fact that the word appears many times in all documents.
However, TF/IDF was designed in an era when it was standard practice to remove the most common words (or stopwords, see Stopwords: Performance Versus Precision) from the index altogether. The algorithm didn’t need to worry about an upper limit for term frequency because the most frequent terms had already been removed.
In Elasticsearch, the
BM25, on the other hand, does have an upper limit. Terms that appear 5 to 10 times in a document have a significantly larger impact on relevance than terms that appear just once or twice. However, as can be seen in Figure 34, “Term frequency saturation for TF/IDF and BM25”, terms that appear 20 times in a document have almost the same impact as terms that appear a thousand times or more.
This is known as nonlinear term-frequency saturation.
Figure 34. Term frequency saturation for TF/IDF and BM25
BM25 also considers shorter fields to have more weight than longer fields, but it considers each field separately by taking the average length of the field into account. It can distinguish between a short
In Query-Time Boosting, we said that the
摘自:https://www.elastic.co/guide/en/elasticsearch/guide/current/pluggable-similarites.html
Before we move on from relevance and scoring, we will finish this chapter with a more advanced subject: pluggable similarity algorithms. While Elasticsearch uses the Lucene’s Practical Scoring Function as its default similarity algorithm, it supports other algorithms out of the box, which are listed in the Similarity Modules documentation.
Okapi BM25
The most interesting competitor to TF/IDF and the vector space model is called Okapi BM25, which is considered to be a state-of-the-art ranking function. BM25 originates from the probabilistic relevance model, rather than the vector space model, yet the algorithm has a lot in common with Lucene’s practical scoring function.Both use term frequency, inverse document frequency, and field-length normalization, but the definition of each of these factors is a little different. Rather than explaining the BM25 formula in detail, we will focus on the practical advantages that BM25 offers.
Term-frequency saturation
Both TF/IDF and BM25 use inverse document frequency to distinguish between common (low value) words and uncommon (high value) words. Both also recognize (see Term frequency) that the more often a word appears in a document, the more likely is it that the document is relevant for that word.However, common words occur commonly. The fact that a common word appears many times in one document is offset by the fact that the word appears many times in all documents.
However, TF/IDF was designed in an era when it was standard practice to remove the most common words (or stopwords, see Stopwords: Performance Versus Precision) from the index altogether. The algorithm didn’t need to worry about an upper limit for term frequency because the most frequent terms had already been removed.
In Elasticsearch, the
standardanalyzer—the default for
stringfields—doesn’t remove stopwords because, even though they are words of little value, they do still have some value. The result is that, for very long documents, the sheer number of occurrences of words like
theand
andcan artificially boost their weight.
BM25, on the other hand, does have an upper limit. Terms that appear 5 to 10 times in a document have a significantly larger impact on relevance than terms that appear just once or twice. However, as can be seen in Figure 34, “Term frequency saturation for TF/IDF and BM25”, terms that appear 20 times in a document have almost the same impact as terms that appear a thousand times or more.
This is known as nonlinear term-frequency saturation.
Figure 34. Term frequency saturation for TF/IDF and BM25
Field-length normalization
In Field-length norm, we said that Lucene considers shorter fields to have more weight than longer fields: the frequency of a term in a field is offset by the length of the field. However, the practical scoring function treats all fields in the same way. It will treat alltitlefields (because they are short) as more important than all
bodyfields (because they are long).
BM25 also considers shorter fields to have more weight than longer fields, but it considers each field separately by taking the average length of the field into account. It can distinguish between a short
titlefield and a
longtitle field.
In Query-Time Boosting, we said that the
titlefield has a natural boost over the
bodyfield because of its length. This natural boost disappears with BM25 as differences in field length apply only within a single field.
摘自:https://www.elastic.co/guide/en/elasticsearch/guide/current/pluggable-similarites.html
相关文章推荐
- TF-IDF提取关键词并用余弦算法计算相似度
- spark mllib 中的tf-idf算法计算文档相似度
- Solr相似度算法一:Lucene TF-IDF 相关性算分公式
- 文本相似度算法——空间向量模型的余弦算法和TF-IDF
- Solr相似度算法一:DefaultSimilarity(基于TF-IDF的默认相似度算法)
- Solr相似度算法一:Lucene TF-IDF 相关性算分公式
- Solr相似度算法一:Lucene TF-IDF 相关性算分公式
- 两篇文章相似度:TF-IDF与余弦相似性的应用
- TF-IDF算法
- TF-IDF算法及应用实例
- TF-IDF算法及其编程实现
- TF-IDF及其算法
- TF-IDF与文档相似度的应用
- TF-IDF及其算法
- TF-IDF及其算法
- TF-IDF 算法
- tf-idf算法,实现文章关键字抽取
- TF-IDF算法概念
- TF-IDF及其算法
- TF-IDF算法概念