lucene in action 第二章（3）（索…

一、索引数值类型的数据。

在早期的lucene，数值类型的值，如“1900”是作为一个文本来对待的。他就是一个字符串，没有大小，没有范围。在实际使用当中，我们经常需要使用数字作为索引。例如，图书的价格，邮件的收发时间等等。

在lucene2.9以后提供了这种numeric
index的功能。

可以使用一个
NumericField的Field子类来实现。使用setXXValue的方法可以为Field设置numeric的值。

doc.add(new
NumericField("price").setDoubleValue(19.99));

这样这个price 的field就可以用于 搜索，sort
了。也可以像textual文本那样精确匹配。

通用
NumericField可以接受多个同名的field。

例如：doc.add(new
NumericField("price").setDoubleValue(19.99));

doc.add(new
NumericField("price").setINTValue(9.99));

在获取搜索结果的时候，NumericRangeQuery 和 NumericRangeFilter这个两个类会以
“or”
的形式来对待所有一个同名的field有多个value的情况。但是sort在这种情况下没有定义。

我们在index时间和日期的时候，就可以使用NumericField了。基本方法是将其转化为数值。

doc.add(new
NumericField("timestamp") .setLongValue(new
Date().getTime()));

Calendar cal =
Calendar.getInstance();

cal.setTime(date);

doc.add(new
NumericField("dayOfMonth") .setIntValue(cal.get(Calendar.DAY_OF_MONTH)));

二、域截断

在我们实例化一个IndexWriter的时候，会传入一个参数 IndexWriter.MaxFieldLength.UNLIMITED

new IndexWriter(dir,new
StandardAnalyzer(Version.LUCENE_36),IndexWriter.MaxFieldLength.UNLIMITED);

这个参数就是用户域截断的。当我们在索引一个Field的value的时候，我们事先不知道这个value有多长，也许很长，

或者这个value是非textual的
很长的 binary
content。这就会造成问题。我们可以使用这个参数控制index的数量。

也可使用indexWriter的setMaxFieldLength(maxFieldLength)

writer.setMaxFieldLength(maxFieldLength)

这个maxFieldLength的意思是index的最多多少个term：

The maximum
number of terms that will be indexed for a single field in a
document

如果刚开始的indexWriter的没有截断，在后面设置了设置截断长度，不影响前面的index。

！
慎用这个feature，因为可能造成索引不全，影响体验。

三、近似实时搜索

当我们频繁通过一个writer来修改一个index。现在我们想要search实时的内容时候就可以调用getReader方法。来获得一个只读的IndexReader。它会将这个writer的commit或者uncommit的change添加到index中，并返回一个read-only
indexReader。

IndexReader reader =
writer.getReader();

记住使用玩这个reader后记住关闭它。

这是lucene对它的解释。

"near real-time"
searching, in that changes made during an IndexWriter session can
be quickly made available for searching without closing the writer
nor calling

commit()

.Note
that this is functionally equivalent to calling {#flush} and then
using

IndexReader.open(org.apache.lucene.store.Directory)

to
open a new reader. But the turarnound time of this method should be
faster since it avoids the potentially
costly

commit()

.You
must close
the

IndexReader

returned
by this method once you are done using it.

四、优化index

优化原因：频繁更新indexWriter，会生成很多的segment（每个segment会有多个文件）。lucene在search的时候会search每一个segment然后合并结果，造成速度下降。而且太多的segment会消耗
更多的文件描述符。所以需要优化。

优化的目的就是将散落的多个segmet合并为N个，提高search的速度。

indexWriter提供了4个方法用于优化。

􀂃
optimize() reduces the index to a
single segment, not returning until the
operation is finished.
􀂃 optimize(int
maxNumSegments), also known as
partial optimize, reduces the
index to at most maxNumSegments segments. Because the final
merge down to
one segment is the most costly, optimizing to, say, five
segments should be quite
a bit faster than optimizing down to one segment, allowing you
to trade less
optimization time for slower search speed.
􀂃 optimize(boolean
doWait) is just like optimize,
except if doWait is false then
the call returns immediately while the necessary merges take
place in the background.
Note that doWait=false
only works for a merge scheduler
that runs
merges in background
threads, such as the default
ConcurrentMergeScheduler.
Section 2.13.6 describes merge schedulers in more
detail.
􀂃 optimize(int maxNumSegments, boolean
doWait) is a partial optimize
that
runs in the background if doWait is false.

注意！优化会使用临时文件夹，因为在合并segment后的文件是放在tmp目录下的，当合并完成后才会删除原来的segment，这就必须保证有足够的磁盘空间。

五、lucene的Directory

Directory dir =
FSDirectory.open(new File(indexDir));
writer = new IndexWriter(dir,new
StandardAnalyzer(Version.LUCENE_36),false,IndexWriter.MaxFieldLength.UNLIMITED);

todo。

六、lucene indexReader 和
IndexWriter在 多线程下工作

1、任意多个的indexReader 可以
在同一个index上打开。不管是在同一个jvm上还是多个jvm上，都没有问题。但是如果在同一个jvm上多个线程公用一个indexReader会更好滴利用资源。甚至在一个indexWriter在修改index的时候也可以创建一个indexReader。

2、在一个index上只能打开一个indexWriter。只是不能同时打开。每次只能打开一个。因为当一个indexWriter实例化后，就会创建一个write
锁。直到这个indexWriter关闭后，才释放这个锁。所以只能有一个indexWriter在一个index上。

3、但是可以有任意多个 thread 共享一个
indexReader或者indexwriter。因为这两个类是线程安全的。

4、writeIndexer的锁。Index每一个时刻只能有一个writerIndexer打开，控制这个机制就是使用write.clock来决定的。

例如 两个IndexWriter同时打开一个index，会抛出LockObtainFailedException异常。
{

IndexWriter writer
= new IndexWriter(dir,new
StandardAnalyzer(Version.LUCENE_36),IndexWriter.MaxFieldLength.UNLIMITED);

IndexWriter
new =
new IndexWriter(dir,new
StandardAnalyzer(Version.LUCENE_36),IndexWriter.MaxFieldLength.UNLIMITED);

}

Exception in thread "main"
org.apache.lucene.store.LockObtainFailedException: Lock
obtain timed out:
NativeFSLock@D:\work\eclipseworkpace\lucenelearn\charpter2-1\write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at
org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
at
org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:953)
at
charpter2.ChangeIndex.<init>(ChangeIndex.java:46)
at charpter2.ChangeIndex.main(ChangeIndex.java:177)

可以
自行设置锁的机制。使用Directory的setLockFactory方法设置锁的机制。
dir = FSDirectory.open(new
File(indexDir));

dir.setLockFactory(lockFactory)

this.writer = new
IndexWriter(dir,new
StandardAnalyzer(Version.LUCENE_36),IndexWriter.MaxFieldLength.UNLIMITED);
注意必须在调用IndexWriter构造函数之前，设置setLockFactory方法。

lucene有四种
LockFactory的实现

NativeFSLockFactory This
is the default locking for FSDirectory, using
java.nio native OS locking, which will never leave
leftover lock files when the JVM exits. But this
locking implementation may not work correctly
over certain shared file systems,
notably NFS.

SimpleFSLockFactory Uses
Java’s File.createNewFile API, which may be
more portable across different file systems
than NativeFSLockFactory. Be
aware that if the JVM crashes or IndexWriter isn’t
closed before the JVM exits, this may leave a
leftover write.lock file, which you must manually
remove.

SingleInstanceLockFactory Creates
a lock entirely in memory. This is the default
locking implementation for RAMDirectory. Use this
when you know all IndexWriters will be
instantiated in a single JVM.

NoLockFactory Disables
locking entirely. Be careful! Only use this when you
are
absolutely certain that Lucene’s normal locking
safeguard isn’t necessary—for example, when using
a private RAMDirectory with a single IndexWriter
instance.

可以使用 IndexWriter 的isLocked(Directory) 方法检查一个index是否已经上锁了。

可以使用 IndexWriter 的unlock(Directory)
给一个index解锁，但是最好不要用，可能会使一个index变得不可用。

七、 lucene的debug

this.writer = new
IndexWriter(dir,new
StandardAnalyzer(Version.LUCENE_36),IndexWriter.MaxFieldLength.UNLIMITED);
this.writer.setInfoStream(System.out);

可以使用setInfoStream（PrintStream)
lucene 会打印出所有 lucene的index过程的信息，如下
docs =
4
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: commit: start
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: commit: enter lock
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: commit: now prepare
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: prepareCommit: flush
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: now trigger flush reason=explicit flush
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: start flush:
applyAllDeletes=true
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: index before flush _0(3.6.1):C2/1
_2(3.6.1):C2/1 _4(3.6.1):C1
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: DW: flush postings as segment _5
numDocs=1
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: DW: new segment has no vectors
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: DW: flushedFiles=[_5.fdt, _5.prx, _5.fnm, _5.nrm,
_5.tis, _5.fdx, _5.frq, _5.tii]
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: DW: flush: segment=_5(3.6.1):C1
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: DW: ramUsed=0.095 MB
newFlushedSize=0 MB (0 MB w/o doc stores) docs/MB=3,898.052
new/old=0.191%
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: DW: flush: push buffered deletes startSize=98
frozenSize=1024
BD 0 [Sun Mar 18 17:54:58 CST
2012; main]: push deletes 1 deleted terms (unique
count=1) bytesUsed=1024 delGen=1 packetCount=1
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: DW: flush: delGen=1
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: DW: flush time 314 msec
IFD [Sun Mar 18 17:54:58 CST
2012; main]: now checkpoint "segments_5" [4 segments ; isCommit =
false]
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: apply all deletes during flush
BD 0 [Sun Mar 18 17:54:58 CST
2012; main]: applyDeletes: infos=[_0(3.6.1):C2/1, _2(3.6.1):C2/1,
_4(3.6.1):C1, _5(3.6.1):C1] packetCount=1
BD 0 [Sun Mar 18 17:54:58 CST
2012; main]: seg=_5(3.6.1):C1 segGen=1 segDeletes=[ 1 deleted terms
(unique count=1) bytesUsed=1024]; coalesced deletes=[null]
delCount=0
BD 0 [Sun Mar 18 17:54:58 CST
2012; main]: seg=_4(3.6.1):C1/1 segGen=0 coalesced
deletes=[CoalescedDeletes(termSets=1,queries=0)] delCount=1 100%
deleted
BD 0 [Sun Mar 18 17:54:58 CST
2012; main]: seg=_2(3.6.1):C2/1 segGen=0 coalesced
deletes=[CoalescedDeletes(termSets=1,queries=0)]
delCount=0
BD 0 [Sun Mar 18 17:54:58 CST
2012; main]: seg=_0(3.6.1):C2/1 segGen=0 coalesced
deletes=[CoalescedDeletes(termSets=1,queries=0)]
delCount=0
BD 0 [Sun Mar 18 17:54:58 CST
2012; main]: applyDeletes took 27 msec
IFD [Sun Mar 18 17:54:58 CST
2012; main]: now checkpoint "segments_5" [4 segments ; isCommit =
false]
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: drop 100% deleted segments:
[_4(3.6.1):C1/1]
IFD [Sun Mar 18 17:54:58 CST
2012; main]: now checkpoint "segments_5" [3 segments ; isCommit =
false]
IFD [Sun Mar 18 17:54:58 CST
2012; main]: delete "_4_1.del"
BD 0 [Sun Mar 18 17:54:58 CST
2012; main]: prune sis=org.apache.lucene.index.SegmentInfos@d08633
minGen=2 packetCount=1
BD 0 [Sun Mar 18 17:54:58 CST
2012; main]: pruneDeletes: prune 1 packets; 0 packets
remain
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: clearFlushPending
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: LMP: findMerges: 3 segments
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: LMP: seg=_0(3.6.1):C2/1 level=2.240549 size=0.000
MB
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: LMP: seg=_2(3.6.1):C2/1 level=2.240549 size=0.000
MB
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: LMP: seg=_5(3.6.1):C1 level=2.429752 size=0.000
MB
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: LMP: level -1.0 to 2.429752: 3
segments
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: CMS: now merge
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: CMS: index: _0(3.6.1):C2/1
_2(3.6.1):C2/1 _5(3.6.1):C1
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: CMS: no more merges pending; now
return
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: startCommit(): start
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: startCommit index=_0(3.6.1):C2/1 _2(3.6.1):C2/1
_5(3.6.1):C1 changeCount=4
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: done all syncs
IW 0 [Sun Mar 18 17:54:58 CST
2012; main]: commit: pendingCommit != null
IW 0 [Sun Mar 18 17:54:59 CST
2012; main]: commit: wrote segments file "segments_6"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: now checkpoint "segments_6" [3 segments ; isCommit =
true]
IFD [Sun Mar 18 17:54:59 CST
2012; main]: deleteCommits: now decRef commit
"segments_5"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: delete "_4.prx"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: unable to remove file "_4.prx": java.io.IOException:
Cannot delete
D:\work\eclipseworkpace\lucenelearn\charpter2-1\_4.prx; Will re-try
later.
IFD [Sun Mar 18 17:54:59 CST
2012; main]: delete "_4.fnm"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: delete "_4.fdx"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: unable to remove file "_4.fdx": java.io.IOException:
Cannot delete
D:\work\eclipseworkpace\lucenelearn\charpter2-1\_4.fdx; Will re-try
later.
IFD [Sun Mar 18 17:54:59 CST
2012; main]: delete "_4.frq"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: unable to remove file "_4.frq": java.io.IOException:
Cannot delete
D:\work\eclipseworkpace\lucenelearn\charpter2-1\_4.frq; Will re-try
later.
IFD [Sun Mar 18 17:54:59 CST
2012; main]: delete "_4.tis"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: unable to remove file "_4.tis": java.io.IOException:
Cannot delete
D:\work\eclipseworkpace\lucenelearn\charpter2-1\_4.tis; Will re-try
later.
IFD [Sun Mar 18 17:54:59 CST
2012; main]: delete "_4.tii"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: delete "_4.nrm"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: unable to remove file "_4.nrm": java.io.IOException:
Cannot delete
D:\work\eclipseworkpace\lucenelearn\charpter2-1\_4.nrm; Will re-try
later.
IFD [Sun Mar 18 17:54:59 CST
2012; main]: delete "_4.fdt"
IFD [Sun Mar 18 17:54:59 CST
2012; main]: unable to remove file "_4.fdt": java.io.IOException:
Cannot delete
D:\work\eclipseworkpace\lucenelearn\charpter2-1\_4.fdt; Will re-try
later.
IFD [Sun Mar 18 17:54:59 CST
2012; main]: delete "segments_5"
IW 0 [Sun Mar 18 17:54:59 CST
2012; main]: commit: done