DotLucene源码浅读笔记(1)补遗:编写简单中文分词器ChineseAnalyzer
2007-06-02 17:56
405 查看
原理部分,可以参考DotLucene源码浅读笔记(1) : Lucene.Net.Analysis ,本篇是依据上篇文章的分析,编写出的简单中文分词器(ChineseAnalyzer).
从DotLucene源码浅读笔记(1) : Lucene.Net.Analysis可以知道,与分词有关的主要是两个基类:
词法分析器(Analyzer) :词法过滤和分析的类,实际上是对分词器, 过滤器的综合包装类。
分词器(Tokenizer):对文本进行分词,可能是单字,词,二元切分等等。
而最基础的概念是Token,Token是DotLucene最基本的单位,以单字切分则每个单字为一个Token,如果以中文分词来切分则每个词为一个Token,
另外,中文分词器ChineseAnalyzer并不是中文分词,中文分词器仅仅是对中文分词的结果分析成DotLucene索引器能够认识的格式,
先实现分词器(Tokenizer),代码中用了一个第三方的分词组件做实验.
中文分析器(Analyzer)代码:
![](http://www.cnblogs.com/Images/OutliningIndicators/None.gif)
public class ChineseAnalyzer : Analyzer
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
public ChineseAnalyzer()
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
public override TokenStream TokenStream(TextReader reader)
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
TokenStream result = new ChineseTokenizer(reader);
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
result = new LowerCaseFilter(result);
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
//result = new StopFilter(result, stopSet); //你还可以自己编写过滤器。
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
return result;
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
}
上面是一个简单的DotLucene的ChineseAnalyzer,算法并不是最优的.主要用于理解如何实现ChineseAnalyzer
文中涉及到的中文分词组件:点此下载 ,中文分词组件版权归原作者所有,此出提供下载是为了大家学习之方便.用于商业目的,请自行联系作者.
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedBlock.gif)
/**//// <summary>
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
/// DotLucene中文分词器
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
/// Author :邝伟科 http://www.cnblogs.com/kwklover
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
/// dated : 2006/10/24
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
/// </summary>
![](http://www.cnblogs.com/Images/OutliningIndicators/None.gif)
public class ChineseTokenizer : Tokenizer
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private List<string> ioBuffer;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private int offSet = 0 ; //偏移量.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private int position = -1 ; //词汇在缓冲中的位置.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private int length = 0 ; //词汇的长度.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private int start = 0 ; //开始偏移量.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
public ChineseTokenizer(System.IO.TextReader input)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
: base(input)
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
//这里用了一个第三方的中文分词组件.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
ioBuffer = Sj110.Com.Chinese.Tokenizer.Tokenize(input.ReadToEnd());
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
//DotLucene的分词器简单来说,就是实现Tokenizer的Next方法,把分解出来的每一个词构造为一个Token,因为Token是DotLucene分词的基本单位。
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
public override Token Next()
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
position++;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
if (position < ioBuffer.Count)
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
length = ioBuffer[position].Length;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
start = offSet ;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
offSet += length ;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
return new Token(ioBuffer[position], start, start + length);
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
return null;
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
}
从DotLucene源码浅读笔记(1) : Lucene.Net.Analysis可以知道,与分词有关的主要是两个基类:
词法分析器(Analyzer) :词法过滤和分析的类,实际上是对分词器, 过滤器的综合包装类。
分词器(Tokenizer):对文本进行分词,可能是单字,词,二元切分等等。
而最基础的概念是Token,Token是DotLucene最基本的单位,以单字切分则每个单字为一个Token,如果以中文分词来切分则每个词为一个Token,
另外,中文分词器ChineseAnalyzer并不是中文分词,中文分词器仅仅是对中文分词的结果分析成DotLucene索引器能够认识的格式,
先实现分词器(Tokenizer),代码中用了一个第三方的分词组件做实验.
中文分析器(Analyzer)代码:
![](http://www.cnblogs.com/Images/OutliningIndicators/None.gif)
public class ChineseAnalyzer : Analyzer
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
public ChineseAnalyzer()
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
public override TokenStream TokenStream(TextReader reader)
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
TokenStream result = new ChineseTokenizer(reader);
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
result = new LowerCaseFilter(result);
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
//result = new StopFilter(result, stopSet); //你还可以自己编写过滤器。
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
return result;
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
}
上面是一个简单的DotLucene的ChineseAnalyzer,算法并不是最优的.主要用于理解如何实现ChineseAnalyzer
文中涉及到的中文分词组件:点此下载 ,中文分词组件版权归原作者所有,此出提供下载是为了大家学习之方便.用于商业目的,请自行联系作者.
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedBlock.gif)
/**//// <summary>
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
/// DotLucene中文分词器
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
/// Author :邝伟科 http://www.cnblogs.com/kwklover
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
/// dated : 2006/10/24
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
/// </summary>
![](http://www.cnblogs.com/Images/OutliningIndicators/None.gif)
public class ChineseTokenizer : Tokenizer
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private List<string> ioBuffer;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private int offSet = 0 ; //偏移量.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private int position = -1 ; //词汇在缓冲中的位置.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private int length = 0 ; //词汇的长度.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
private int start = 0 ; //开始偏移量.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
public ChineseTokenizer(System.IO.TextReader input)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
: base(input)
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
//这里用了一个第三方的中文分词组件.
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
ioBuffer = Sj110.Com.Chinese.Tokenizer.Tokenize(input.ReadToEnd());
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
//DotLucene的分词器简单来说,就是实现Tokenizer的Next方法,把分解出来的每一个词构造为一个Token,因为Token是DotLucene分词的基本单位。
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
public override Token Next()
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
position++;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
if (position < ioBuffer.Count)
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/ContractedSubBlock.gif)
![](http://www.cnblogs.com/Images/dot.gif)
{
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
length = ioBuffer[position].Length;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
start = offSet ;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
offSet += length ;
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
return new Token(ioBuffer[position], start, start + length);
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
![](http://www.cnblogs.com/Images/OutliningIndicators/InBlock.gif)
return null;
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
}
![](http://www.cnblogs.com/Images/OutliningIndicators/ExpandedBlockEnd.gif)
}
相关文章推荐
- DotLucene源码浅读笔记(1)补遗:编写简单中文分词器ChineseAnalyzer
- DotLucene源码浅读笔记(1)补遗:编写简单中文分词器ChineseAnalyzer
- DotLucene源码浅读笔记(1)补遗:编写简单中文分词器ChineseAnalyzer
- DotLucene源码浅读笔记(1)补遗:编写简单中文分词器ChineseAnalyzer
- Lucene 中文分词器SmartChineseAnalyzer
- C#中文分词算法:ChineseAnalyzer
- 笔试题13:采用UDP协议,编写一个简单发送字符串的程序(源码)
- OpenJDK源码研究笔记(四)-编写和组织可复用的工具类和方法
- solr6.1配置中文分词器IK-analyzer
- 【 Visual C++】游戏开发笔记之二——最简单的DirectX,vc窗口的编写
- Servlet学习笔记_03_使用servlet编写一个简单的helloworld程序
- 【 Visual C++】游戏开发笔记之二——最简单的DirectX,vc窗口的编写
- Common Lisp译本笔记2之第三章:简单的数据库实现(源码)
- java鬼混笔记:lucene 4、中文分词器--IKAnalyzer
- Elasticsearch笔记六之中文分词器及自定义分词器
- 【 Visual C++】游戏开发笔记之二——最简单的DirectX,vc窗口的编写
- solr中文分词器IK-analyzer安装
- solr6.0 添加中文分词器 IK-Analyzer
- IK Analyzer 3 中文分词器的学习笔记
- 自定义控件-编写笔记(简单的时间验证码和事件控件)