您的位置:首页 > 其它

解决lucene 1.* 使用排序后内存溢出问题

2006-06-19 16:55 459 查看
使用lucene一段时间了,最近发现使用sort后服务器内存大赠,经常在800M左右。查了下资料,发现,在lucene 现有所有版本中都有这个问题。
现给出测试代码和解决方法:(非原创,是询问[韩国人]后解决的)
下面是测试代码:

1 using Lucene.Net.Analysis;
2 using Lucene.Net.Analysis.Standard;
3 using Lucene.Net.Documents;
4 using Lucene.Net.Index;
5 using Lucene.Net.Search;
6 using Lucene.Net.Store;
7 using Lucene.Net.QueryParsers;
8 using Directory = Lucene.Net.Store.Directory;
9
10
11 //Testcode:
12
13
14
15 // Create an index with 100 document. Each document has one keyword field.
16 IndexWriter oIndexWriter = new IndexWriter(@"C:\lucene_test", new StandardAnalyzer(), true);
17 for(int nCount = 1; nCount <= 100; nCount++)
18 {
19 Document oDoc = new Document();
20 oDoc.Add(Field.Keyword("field1", nCount.ToString() + "_XXXXXXXXXXX" )); // removing some XXXX results in less memory consumption
21 //oDoc.Add(Field.Keyword("field2", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx")); // a second field doesn't increase memory consumption
22 oIndexWriter.AddDocument(oDoc);
23 }
24 oIndexWriter.Close();
25
26 // create a searcher which sorts the keyword field
27 IndexSearcher oSearcher = null;
28 Sort oSort = null;
29 //oSort = new Sort("field1"); // remove this line to avoid the memory leak
30 Query oQuery = QueryParser.Parse("0", "field1", new StandardAnalyzer());
31 Directory oDirectory = FSDirectory.GetDirectory(@"C:\lucene_test", false);
32
33 // save current used memory
34 GC.Collect(); // force garbage collection
35 long lMemoryBytesStart = GC.GetTotalMemory(true);
36
37 // execute 10 searches, close the searcher every time and print out the memory used, by garbage collector
38 for(int nCount = 0; nCount < 10; nCount++)
39 {
40 oSearcher = new IndexSearcher(oDirectory);
41 Hits oHits = oSearcher.Search(oQuery, null, oSort);
42 oSearcher.Close();
43
44 GC.Collect(); // force garbage collection
45 Console.WriteLine(String.Format("{0}: {1} Hits", nCount, oHits.Length()));
46 Console.WriteLine(String.Format("{0} bytes", GC.GetTotalMemory(true) - lMemoryBytesStart));
47 Console.WriteLine("");
48 }
49
50 GC.Collect(); // force garbage collection
51
52 Console.WriteLine("End of search:");
53 Console.WriteLine(String.Format("{0} bytes", GC.GetTotalMemory(true) - lMemoryBytesStart));
54 Console.WriteLine("");
Console.ReadLine();

运行效果:

使用sort oSort = new Sort("field1") 不使用sort //oSort = new Sort("field1")
修改后,使用sort



代码修改:

1.在文件 Search\FieldSortedHitQueue.cs 中增加

1 internal static void Close(IndexReader reader)
2 {
3 lock (Comparators.SyncRoot)
4 {
5 System.Collections.Hashtable readerCache = (System.Collections.Hashtable) Comparators[reader];
6 if (readerCache != null)
7 {
8 readerCache.Clear();
9 }
10
11 Comparators.Remove(reader);
12 }
13 }
14
2.在 FieldCache.cs 中增加
void Close(IndexReader reader)

3.在 FieldCacheImpl.cs 增加

1 public virtual void Close(IndexReader reader)
2 {
3 lock (this)
4 {
5 System.Collections.Hashtable readerCache = (System.Collections.Hashtable) cache[reader];
6 if (readerCache != null)
7 {
8 readerCache.Clear();
9 readerCache = null;
10 }
11 cache.Remove(reader);
12 }
13 }
4.修改IndexSearcher.cs中Close() 方法

1 public override void Close()
2 {
3 FieldSortedHitQueue.Close(reader);
4 Lucene.Net.Search.FieldCache_Fields.DEFAULT.Close(reader);
5
6 if (closeReader)
7 reader.Close();
8 }
9
ok完成
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签: