win7下MyEclipse装Nutch1.7

一、下载安装包

下载 apache-nutch-1.7-bin.zip 和 apache-nutch-1.7-src.zip 2个包

二、基本环境搭建

1、解压apache-nutch-1.7-src.zip包到eclipse的工作目录下面，如：D:\Workspaces\MyEclipse 8.5\test\apache-nutch-1.7

2、解压apache-nutch-1.7-bin.zip包，将其lib文件夹复制到D:\Workspaces\MyEclipse 8.5\test\apache-nutch-1.7下面（因为nutchsrc包下缺少部分lib包）。

3、将nutch导入eclipse，首先点击工具栏“File”-“new”-“other”-“Java project from an existing Ant buildfile”，


，

选择next，在Ant buildfile中选中nutch中的build.xml， Project name与Ant buildfile中的build.xml前面的nutch文件夹保持一致，并勾选“link to the buildfile in the file system”。



点击“finish”后，提示nutch的build文件下缺少lib包，此时我们将nutch下面的lib文件夹复制到bulid文件下，重新执行nutch导入eclipse中步骤即可。



eclipse中已经有nutch的文件了，我们选中conf，右键“build path”---"use as source folder"



此时，发现nutch项目中存在错误“x”，选中项目后鼠标右键“properties”，将项目编码改为utf8点击“apply”，至此nutch已经导入eclipse。



三、Eclipse中配置Nutch

1、找到Crawl.java ，右键“run as ” ----"java application ", 在console中显示 “Usage: Crawl <urlDir> -solr <solrURL> [-dir d] [-threads n] [-depth i] [-topN N]”。



在nutch文件夹下新建文件夹“urls”，在urls文件夹下面建立url.txt，中文本中写入要抓取的网站地址。



此时，Crawl.java类中，鼠标右键“run as ” ----"run configuration"，在"programm argments"输入“Crawl urls -dir data1221 -threads 2 -depth 3 -topN 5”，在“vm argment”中输入“-Xms512m -Xmx800m”，点击“apply”。



点击“run”。报错，信息如下：

Injector: Converting injected urls to crawl db entries.
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-\mapred\staging\1623868107\.staging to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
at org.apache.nutch.crawl.Injector.inject(Injector.java:281)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:132)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)

这个是hadoop的包引起的，需要修改nutch下面lib中的hadoop-core-1.2.0.jar，需要修改hadoop中的FileUtil.java，注释checkReturnValue方法的内容，如图。



将hadoop重新打包。因为我不会打包操作，我是直接用解压缩的方式打开hadoop-core-1.2.0.jar，删除FileUtil的class文件，引入修改后的class文件



此时需要重新build文件，选中“build.xml”,右键 “run as” ----"Ant Build..."



选中“jar”、“job”、“runtime[default]”选项，点击“apply”，点击“run”，需要等待一段时间，才能编译好，第一次编译需要下载部分文件用于编译。



编译后，将nutch文件夹下面的build文件夹下的“plugins”文件夹、“apache-nutch-1.7.jar”、“apache-nutch-1.7.job”复制到nutch文件夹下，在eclipse中刷新nutch项目，如下图。



重新运行Crawl类，运行报错。

Fetcher: No agents listed in 'http.agent.name' property.
Exception in thread "main" java.lang.Ille
4000
galArgumentException: Fetcher: No agents listed in 'http.agent.name' property.
at org.apache.nutch.fetcher.Fetcher.checkConfiguration(Fetcher.java:1397)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1282)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:141)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)

提示没有 'http.agent.name'属性，我们找到nutch下面的conf文件夹下面的nutch-default.xml搜索'http.agent.name'，发现其中的value值为空。

<property>
<name>http.agent.name</name>
<value></value>
<description>HTTP 'User-Agent' request header. MUST NOT be empty -
please set this to a single word uniquely related to your organization.

NOTE: You should also check other related properties:

http.robots.agents
http.agent.description
http.agent.url
http.agent.email
http.agent.version

and set their values appropriately.

</description>
</property>

我们复制上面这段代码到nutch-site.xml中，填写value值。

注意，一般我们不改动nutch-default.xml的默认配置，而是修改nutch-site.xml的配置，覆盖nutch-default.xml的配置。



重新运行Crawl类，报错信息如下：

Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
file:/D:/Workspaces/MyEclipse 8.5/test/apache-nutch-1.7/data1221/segments/20131221154928/parse_data
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:197)
at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:40)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1081)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1073)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:180)
at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:151)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)

上面的报错信息是因为没有找到对应的parse_data文件，由于之前运行过程中报错，但当时已经生成了segments下的部分信息导致的。

此时我们发现nutch下面已经生成了data1221文件，我们删除“data1221”这个文件，重新跑Crawl类，运行正常。

发现nutch下面重新生成了“data1221”文件夹，且文件夹下面生成了“crawldb”、“linkdb”、”segments“文件夹。

需要补充的是，我们在这个过程没有改动nutch抓取网页的配置信息，因为nutch下面conf文件下”automaton-urlfilter.txt“默认允许抓取网址如下：

# accept anything else
+.*

至此，nutch抓取网页演示完毕。

附eclipse的console中的输出信息如下：

solrUrl is not set, indexing will be skipped...
crawl started in: data1221
rootUrlDir = urls
threads = 2
depth = 3
solrUrl=null
topN = 5
Injector: starting at 2013-12-21 16:05:08
Injector: crawlDb: data1221/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 0
Injector: total number of urls injected after normalization and filtering: 1
Injector: Merging injected urls into crawl db.
Injector: finished at 2013-12-21 16:05:11, elapsed: 00:00:02
Generator: starting at 2013-12-21 16:05:11
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: data1221/segments/20131221160513
Generator: finished at 2013-12-21 16:05:14, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2013-12-21 16:05:14
Fetcher: segment: data1221/segments/20131221160513
Using queue mode : byHost
Fetcher: threads: 2
Fetcher: time-out divisor: 2
QueueFeeder finished: total 1 records + hit by time limit :0
Using queue mode : byHost
Using queue mode : byHost
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
fetching http://www.163.com/ (queue crawl delay=5000ms)
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2013-12-21 16:05:16, elapsed: 00:00:02
ParseSegment: starting at 2013-12-21 16:05:16
ParseSegment: segment: data1221/segments/20131221160513
Parsed (8ms):http://www.163.com/
ParseSegment: finished at 2013-12-21 16:05:17, elapsed: 00:00:01
CrawlDb update: starting at 2013-12-21 16:05:17
CrawlDb update: db: data1221/crawldb
CrawlDb update: segments: [data1221/segments/20131221160513]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: 404 purging: false
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2013-12-21 16:05:18, elapsed: 00:00:01
Generator: starting at 2013-12-21 16:05:18
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: data1221/segments/20131221160520
Generator: finished at 2013-12-21 16:05:21, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2013-12-21 16:05:21
Fetcher: segment: data1221/segments/20131221160520
Using queue mode : byHost
Fetcher: threads: 2
Fetcher: time-out divisor: 2
QueueFeeder finished: total 2 records + hit by time limit :0
Using queue mode : byHost
fetching http://m.163.com/ (queue crawl delay=5000ms)
Using queue mode : byHost
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1
* queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613126729
now           = 1387613122697
0. http://m.163.com/newsapp/ -activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1
* queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613126729
now           = 1387613123697
0. http://m.163.com/newsapp/ -activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1
* queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613126729
now           = 1387613124697
0. http://m.163.com/newsapp/ -activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1
* queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613126729
now           = 1387613125698
0. http://m.163.com/newsapp/ -activeThreads=2, spinWaiting=2, fetchQueues.totalSize=1
* queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613126729
now           = 1387613126698
0. http://m.163.com/newsapp/ fetching http://m.163.com/newsapp/ (queue crawl delay=5000ms)
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2013-12-21 16:05:28, elapsed: 00:00:07
ParseSegment: starting at 2013-12-21 16:05:28
ParseSegment: segment: data1221/segments/20131221160520
Parsed (1ms):http://m.163.com/newsapp/
ParseSegment: finished at 2013-12-21 16:05:29, elapsed: 00:00:01
CrawlDb update: starting at 2013-12-21 16:05:29
CrawlDb update: db: data1221/crawldb
CrawlDb update: segments: [data1221/segments/20131221160520]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: 404 purging: false
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2013-12-21 16:05:30, elapsed: 00:00:01
Generator: starting at 2013-12-21 16:05:30
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: data1221/segments/20131221160532
Generator: finished at 2013-12-21 16:05:33, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2013-12-21 16:05:33
Fetcher: segment: data1221/segments/20131221160532
Using queue mode : byHost
Fetcher: threads: 2
Fetcher: time-out divisor: 2
QueueFeeder finished: total 5 records + hit by time limit :0
Using queue mode : byHost
Using queue mode : byHost
fetching http://digi.163.com/13/0719/10/9450M2MJ00162659.html (queue crawl delay=5000ms)
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
fetching http://help.3g.163.com/13/1216/17/9G81M68M0096400O.html (queue crawl delay=5000ms)
fetching http://m.163.com/newsapp/download.html (queue crawl delay=5000ms)
-activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2
* queue: http://help.3g.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139083
now           = 1387613135003
0. http://help.3g.163.com/13/1127/15/9EMS17RN0096400O.html * queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139117
now           = 1387613135003
0. http://m.163.com/newsapp/zhinan.html -activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2
* queue: http://help.3g.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139083
now           = 1387613136012
0. http://help.3g.163.com/13/1127/15/9EMS17RN0096400O.html * queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139117
now           = 1387613136012
0. http://m.163.com/newsapp/zhinan.html -activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2
* queue: http://help.3g.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139083
now           = 1387613137013
0. http://help.3g.163.com/13/1127/15/9EMS17RN0096400O.html * queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139117
now           = 1387613137013
0. http://m.163.com/newsapp/zhinan.html -activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2
* queue: http://help.3g.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139083
now           = 1387613138014
0. http://help.3g.163.com/13/1127/15/9EMS17RN0096400O.html * queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139117
now           = 1387613138014
0. http://m.163.com/newsapp/zhinan.html -activeThreads=2, spinWaiting=2, fetchQueues.totalSize=2
* queue: http://help.3g.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139083
now           = 1387613139015
0. http://help.3g.163.com/13/1127/15/9EMS17RN0096400O.html * queue: http://m.163.com maxThreads    = 1
inProgress    = 0
crawlDelay    = 5000
minCrawlDelay = 0
nextFetchTime = 1387613139117
now           = 1387613139015
0. http://m.163.com/newsapp/zhinan.html fetching http://help.3g.163.com/13/1127/15/9EMS17RN0096400O.html (queue crawl delay=5000ms)
fetching http://m.163.com/newsapp/zhinan.html (queue crawl delay=5000ms)
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2013-12-21 16:05:40, elapsed: 00:00:07
ParseSegment: starting at 2013-12-21 16:05:40
ParseSegment: segment: data1221/segments/20131221160532
Parsed (0ms):http://digi.163.com/13/0719/10/9450M2MJ00162659.html
Parsed (0ms):http://help.3g.163.com/13/1127/15/9EMS17RN0096400O.html
Parsed (0ms):http://help.3g.163.com/13/1216/17/9G81M68M0096400O.html
Parsed (0ms):http://m.163.com/newsapp/download.html
Parsed (0ms):http://m.163.com/newsapp/zhinan.html
ParseSegment: finished at 2013-12-21 16:05:42, elapsed: 00:00:01
CrawlDb update: starting at 2013-12-21 16:05:42
CrawlDb update: db: data1221/crawldb
CrawlDb update: segments: [data1221/segments/20131221160532]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: 404 purging: false
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2013-12-21 16:05:44, elapsed: 00:00:02
LinkDb: starting at 2013-12-21 16:05:44
LinkDb: linkdb: data1221/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: internal links will be ignored.
LinkDb: adding segment: file:/D:/Workspaces/MyEclipse 8.5/test/apache-nutch-1.7/data1221/segments/20131221160513
LinkDb: adding segment: file:/D:/Workspaces/MyEclipse 8.5/test/apache-nutch-1.7/data1221/segments/20131221160520
LinkDb: adding segment: file:/D:/Workspaces/MyEclipse 8.5/test/apache-nutch-1.7/data1221/segments/20131221160532
LinkDb: finished at 2013-12-21 16:05:45, elapsed: 00:00:01
crawl finished: data1221