nutch集成solr和中文分词

int32位 posted @ Mar 22, 2016 03:43:22 PM in hadoop/spark , 1696 阅读
转载请注明：http://krystism.is-programmer.com/若有错误，请多多指正，谢谢！

一、构建nutch环境

1. 设置代理

由于nutch使用ant构建，ant调用ivy，会从maven仓库中下载依赖包，因此若公司需要代理才能上网，需要设置代理，如果公司不需要代理即可上网，此步骤直接省略.

总结设置代理遇到的几个坑：

  • 强制使用系统代理,即 ant -autoproxy , 结果失败!
  • 修改 build.xml , 增加 setproxy ,设置代理,结果失败!
  • 设置 ANT_OPTS ,添加选项
  • -Dhttp.proxyHost=http://proxy.xxx.com
  • -Dhttp.proxyPort=xxx , 结果失败!
  • 传递 http_proxy ,结果失败!

失败原因: proxyHost只需要包括主机名，而不需要指定协议

成功编译为修改build.xml文件，设置代理，内容如下:

<target name="myproxy">
      <setproxy proxyhost="proxy.xxx.com" proxyport="913" />
</target>

并修改ivy-init指令，如下:

<target name="ivy-init" depends="ivy-probe-antlib, ivy-init-antlib,myproxy" description="--> initialise Ivy settings">
    <ivy:settings file="${ivy.dir}/ivysettings.xml" />
</target>

2. 设置hadoop环境

若只是构建单机模式，此步骤省略!

否则需要设置并export HADOOP_HOME环境变量:

export HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop}
echo $HADOOP_HOME

3. nutch配置

在编译之前首先需要配置nutch，每次更新配置文件，都需要重新编译！ 配置文件位于$NUTCH_HOME/conf下，务必设置http.agent.name,否则编译后不能运行, 编辑conf/nutch-site.xml, 内容为:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>http.agent.name</name>
        <value>My Spider</value>
    </property>
</configuration>

如果需要设置nutch运行时代理，需要配置代理,修改文件$NUTCH_HOME/conf/nutch-default.xml, 修改http.proxyHost和http.proxyPort值，内容如下:

<property>
  <name>http.proxy.host</name>
  <value>proxy.xxx.com</value>
  <description>The proxy hostname.  If empty, no proxy is used.</description>
</property>

<property>
  <name>http.proxy.port</name>
  <value>xxx</value>
  <description>The proxy port.</description>
</property>

4.编译构建nutch

在此之前，请确保以上步骤已经正确完成，否则会导致编译失败或者运行失败！ 编译:

ant -v runtime

构建开始，大约需要30分钟的时间

二、集成solr

目前solr的版本是5.x,但好像5.x版本差别较大，nutch没有集成支持！因此我们使用当前的4.x版本，目前该版本的latest是4.10.4,点击此处下载.

1.初始化

解压缩，复制example/solr/collection1下的core collection1为nutch:

cp -rf collection1 nutch

并修改$SOLR_HOME/example/solr/nutch/core.properties文件，设置name为nutch:

name=nutch

把$NUTCH_HOME/conf/schema-solr4.xml复制到$SOLR_HOME/example/solr/nutch/conf下,并重命名为schema.xml:

cp $NUTCH_HOME/conf/schema-solr4.xml  $SOLR_HOME/example/solr/nutch/conf/schema.xml

2. 修改配置

此时启动solr会出现以下错误:

org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: copyField dest :'location' is not an explicit field and doesn't match a dynamicField

应该是配置文件schema.xml的一个bug, 修复办法为在<fields>下增加一个location field， 内容为:

  <field name="location" type="location" stored="false" indexed="true" multiValued="true"/>

若没有_version属性，则增加_version_属性:

<field name="_version_" type="long" indexed="true" stored="true"/>

3. 增加中文分词

首先从google code上下载IKAnalyzer,下载版本为IK Analyzer 2012FF_hf1.zip,如果被GFW， 点击此处下载. 解压缩文件，把IKAnalyzer2012FF_u1.jar文件复制到$SOLR_HOME/example/solr-webapp/webapp/WEB-INF/lib,把IKAnalyzer.cfg.xml和stopword.dic复制到$SOLR_HOME/example/solr/nutch/conf,与schema.xml一个目录下:

cp IKAnalyzer2012FF_u1.jar $SOLR_HOME/example/solr-webapp/webapp/WEB-INF/lib
cp IKAnalyzer.cfg.xml stopword.dic $SOLR_HOME/example/solr/nutch/conf

修改core的schema.xml，在配置项间加一段如下配置：

<fieldType name="text_cn" class="solr.TextField">   
     <analyzer class="org.wltea.analyzer.lucene.IKAnalyzer"/>   
</fieldType>

在这个core的schema.xml里面配置field类型的时候就可以使用text_cn

<field name="name"      type="text_cn"   indexed="true"  stored="true"  multiValued="true" />

启动solr服务

cd $SOLR_HOME/example
java -jar start.jar

浏览器访问http://172.16.182.23:8983/solr/#/nutch，在左下边点击Analysis连接，选择Analyse Fieldname / FieldType为text_cn,在Field Value (Index)下输入:

我喜欢solr

然后点击蓝色按钮Analyse Values,查看效果，看是否正确分词！

三、单机运行测试

具体过程可以查看官方教程 。

总结过程如下:

1. 创建种子列表

mkdir urls
cd urls
touch seeds.txt
echo "http://nutch.apache.org/" >> seeds.txt # 每行一个URL

2.使用crawl脚本运行

bin/crawl -i -D solr.server.url=http://localhost:8983/solr/nutch urls/ TestCrawl/  2

3. 验证结果

打开http://localhost:8983/solr/，点击solr admin, 在Querty String输入nutch, 点击Search查看效果

四、分布式运行

与单机过程类型，首先创建urls目录作为种子url，然后上传到hdfs上:

mkdir urls
cd urls
echo "http://apache.org" >>seeds.txt
hdfs dfs -put urls

假定solr已经配置完成，url为localhost:8983

运行以下命令运行：

cd $NUTCH_HOME/runtime/runtime/deploy/
bin/crawl -i -D solr.server.url=http://localhost:8983/solr/nutch urls/ TestCrawl/  2

 

转载请注明：http://krystism.is-programmer.com/若有错误，请多多指正，谢谢！
  • 无匹配
  • 无匹配
Small_feed 评论 (66)
1
[回复]
Emma 说:
2023年1月19日 18:29

This integration of Nutch and Solr with Chinese word segmentation is an incredibly powerful tool that can help users create highly optimized and accurate searches. This will make it cbd effects much easier for individuals to find the information they need quickly and efficiently. Additionally, the ability to segment words in Chinese will ensure that users are getting accurate results that match their search query. This is a great advancement in technology that will be of great benefit to users all over the world.

2
[回复]
Bushra 说:
2023年4月06日 19:51

Exclusive Placements helps with lead capture, website design, review management, Google Ads, and so much more! Increase visibility and boost your online presence today! internet advertising for lawyers

3
[回复]
Bushra 说:
2023年4月07日 01:20

I high appreciate this post. It’s hard to find the good from the bad sometimes, but I think you’ve nailed it! would you mind updating your blog with more information? buy followers

4
[回复]
Bushra 说:
2023年4月07日 17:54

Thanks for the valuable information and insights you have so provided here... agenzia di traduzioni Milano

5
[回复]
Bushra 说:
2023年4月07日 21:29

This is such a great resource that you are providing and you give it away for free. pressure washing services near Cheswick PA

6
[回复]
Bushra 说:
2023年4月09日 07:58

Thanks For sharing this Superb article.I use this Article to show my assignment in college.it is useful For me Great Work. Visit This Website

7
[回复]
Bushra 说:
2023年4月10日 00:36

Thanks for the valuable information and insights you have so provided here... Visit This Website

8
[回复]
Bushra 说:
2023年4月10日 01:31

Thanks for the valuable information and insights you have so provided here... Visit This Website

9
[回复]
Bushra 说:
2023年4月10日 04:13

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. local citations

10
[回复]
asim 说:
2023年4月10日 08:37

Thanks for the valuable information and insights you have so provided here... Adam McManus

11
[回复]
asim 说:
2023年4月10日 20:06

Nice Informative Blog having nice sharing.. w88 rezence

12
[回复]
Bushra 说:
2023年4月10日 23:54

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It is the old what goes around comes around routine. home buying for millennials

13
[回复]
Bushra 说:
2023年4月11日 00:47

Thanks for a wonderful share. Your article has proved your hard work and experience you have got in this field. Brilliant .i love it reading. Granite countertops Sarasota

14
[回复]
Bushra 说:
2023年4月11日 02:16

Thanks For sharing this Superb article.I use this Article to show my assignment in college.it is useful For me Great Work. Click the Following Post

15
[回复]
Bushra 说:
2023年4月11日 08:35

I appreciated your work very thanks Tubidy

16
[回复]
Bushra 说:
2023年4月11日 09:41

thanks for this usefull article, waiting for this article like this again. Click the Following Post

17
[回复]
Bushra 说:
2023年4月11日 19:56

That is really nice to hear. thank you for the update and good luck. Mp3 Juice

18
[回复]
Bushra 说:
2023年4月11日 23:50

Great article Lot's of information to Read...Great Man Keep Posting and update to People..Thanks Click the Following Post

19
[回复]
Bushra 说:
2023年4月12日 01:09

Thanks for sharing the info, keep up the good work going.... I really enjoyed exploring your site. good resource... mini truck

20
[回复]
Bushra 说:
2023年4月14日 00:09

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. See More Tips

21
[回复]
Bushra 说:
2023年4月14日 06:18

Thanks for sharing nice information with us. i like your post and all you share with us is uptodate and quite informative, i would like to bookmark the page so i can come here again to read you, as you have done a wonderful job. Check Out These Helpful Tips

22
[回复]
Bushra 说:
2023年4月14日 20:13

Wow what a Great Information about World Day its very nice informative post. thanks for the post. Tubidy

23
[回复]
Bushra 说:
2023年4月15日 01:15

Thanks for the post and great tips..even I also think that hard work is the most important aspect of getting success.. Car Accident Lawyer New Orleans

24
[回复]
Bushra 说:
2023年4月15日 02:54

Great article Lot's of information to Read...Great Man Keep Posting and update to People..Thanks Click the Following Post

25
[回复]
Bushra 说:
2023年4月15日 05:16

Great article Lot's of information to Read...Great Man Keep Posting and update to People..Thanks Click the Following Post

26
[回复]
Bushra 说:
2023年4月15日 08:40

New web site is looking good. Thanks for the great effort. Click the Following Article

27
[回复]
Bushra 说:
2023年4月15日 20:02

Thank you very much for this useful article. I like it. earthquake retrofitting Los Angeles, CA

28
[回复]
Bushra 说:
2023年4月15日 23:28

Thanks For sharing this Superb article.I use this Article to show my assignment in college.it is useful For me Great Work. http://www.cccum.org/when-your-muffler-needs-repair/

29
[回复]
Bushra 说:
2023年4月16日 01:15

Most of the time I don’t make comments on websites, but I'd like to say that this article really forced me to do so. Really nice post! Click the Following Post

30
[回复]
Bushra 说:
2023年4月16日 08:40

thanks this is good blog. Discover

31
[回复]
Bushra 说:
2023年4月16日 19:12

I am hoping the same best effort from you in the future as well. In fact your creative writing skills has inspired me. Sunrise Fl

32
[回复]
Bushra 说:
2023年4月16日 22:38

Thank you very much for this useful article. I like it. investing in a precious metals ira

33
[回复]
Bushra 说:
2023年4月16日 23:43

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!THANKS!!!!!! click now

34
[回复]
Bushra 说:
2023年4月17日 01:09

thanks for this usefull article, waiting for this article like this again. visit the blog

35
[回复]
Bushra 说:
2023年4月18日 01:06

Thanks for your information, it was really very helpfull.. Apartment complex cleaning services near Monroeville PA 15146

36
[回复]
Bushra 说:
2023年4月19日 19:29

very interesting keep posting. seismic retrofit

37
[回复]
Bushra 说:
2023年4月19日 20:43

Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!THANKS!!!!!! click this link

38
[回复]
Bushra 说:
2023年4月19日 22:21

very interesting keep posting. browse around this site

39
[回复]
Bushra 说:
2023年4月22日 23:47

Most of the time I don’t make comments on websites, but I'd like to say that this article really forced me to do so. Really nice post! Scot French

40
[回复]
Bushra 说:
2023年5月01日 04:30

I found your this post while searching for some related information on blog search...Its a good post..keep posting and update the information. Toronto Raptors Tickets

42
[回复]
naveed 说:
2023年5月02日 12:45

I am hoping the same best effort from you in the future as well. In fact your creative writing skills has inspired me. 클릭하기

43
[回复]
Bushra 说:
2023年5月04日 00:40

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It is the old what goes around comes around routine. mississippi.butcherblockcountertop.net

44
[回复]
naveed 说:
2023年5月05日 13:28

Great! It sounds good. Thanks for sharing.. mississippi.butcherblockcountertop.org

46
[回复]
naveed 说:
2023年5月07日 14:38

Thanks for sharing the info, keep up the good work going.... I really enjoyed exploring your site. good resource... mississippi.complexkitchens.com

47
[回复]
Bushra 说:
2023年5月07日 15:16

This is very educational content and written well for a change. It's nice to see that some people still understand how to write a quality post! best curry pie recipe

48
[回复]
Bushra 说:
2023年5月17日 22:49

That is really nice to hear. thank you for the update and good luck. click here

49
[回复]
asim 说:
2023年5月18日 20:46

That is really nice to hear. thank you for the update and good luck. <a href="http://www.valley-fellowship.com/how-to-download-and-install-mp3-songs/">Download Mp3</a>

50
[回复]
asim 说:
2023年5月18日 20:48

Great article Lot's of information to Read...Great Man Keep Posting and update to People..Thanks Download Mp3

51
[回复]
Bushra 说:
2023年5月18日 21:41

Love to read it,Waiting For More new Update and I Already Read your Recent Post its Great Thanks. <a href="http://www.gabrieldesign.co.uk/way-to-get-content-off-of-facebook/">http://www.gabrieldesign.co.uk/way-to-get-content-off-of-facebook/</a>

52
[回复]
asim 说:
2023年5月19日 23:49

Thanks for a wonderful share. Your article has proved your hard work and experience you have got in this field. Brilliant .i love it reading. Lake of the Ozarks landscape companies

53
[回复]
Bushra 说:
2023年5月21日 18:02

Great! It sounds good. Thanks for sharing.. visit twinspinslots

54
[回复]
Bushra 说:
2023年5月21日 20:20

Love to read it,Waiting For More new Update and I Already Read your Recent Post its Great Thanks. batterie iphone

55
[回复]
Bushra 说:
2023年5月21日 21:57

Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging, Antique Wiring

56
[回复]
Bushra 说:
2023年5月23日 02:23

Thank you for taking the time to publish this information very useful! Check This Out

57
[回复]
Bushra 说:
2023年5月23日 03:22

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. Read This Article

58
[回复]
Bushra 说:
2023年5月23日 14:41

Thank you for taking the time to publish this information very useful! See This Helpful Information

59
[回复]
Naveed 说:
2023年5月25日 04:34

Most of the time I don’t make comments on websites, but I'd like to say that this article really forced me to do so. Really nice post! loc hair care

60
[回复]
asim 说:
2023年5月27日 04:26

This is very educational content and written well for a change. It's nice to see that some people still understand how to write a quality post! Get More Info

61
[回复]
Naveed 说:
2023年5月27日 10:57

Thanks for the post and great tips..even I also think that hard work is the most important aspect of getting success.. mobile bbq catering

64
[回复]
Bushra 说:
2023年5月31日 23:53

Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging, spray foam attic insulation

65
[回复]
Bushra 说:
2023年6月01日 00:36

I really appreciate the kind of topics you post here. Thanks for sharing us a great information that is actually helpful. Good day! masonry wall

66
[回复]
Bushra 说:
2023年6月01日 01:32

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. stump removal


登录 *


loading captcha image...
(输入验证码)
or Ctrl+Enter