sqoop2安装

int32位 posted @ Mar 22, 2016 03:42:06 PM in hadoop/spark , 1747 阅读
转载请注明:http://krystism.is-programmer.com/若有错误,请多多指正,谢谢!

 

1. 下载解压缩

此次安装版本为1.99.6

# Decompress Sqoop distribution tarball
tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz


ln -s sqoop-<version>-bin-hadoop<hadoop version>.tar.gz sqoop

export SQOOP_HOME=`pwd`/sqoop
# Change working directory
cd $SQOOP_HOME

2. 配置服务

配置文件在$SQOOP_HOME/server/conf下,大多数默认配置即可。需要注意的配置是catalina.properties文件下common_loader需要正确配置hadoop库文件,包括hdfs、mapreduce、hive等所有jar包. 我的配置为:

common.loader=${catalina.base}/lib,\ 
${catalina.base}/lib/*.jar,\ 
${catalina.home}/lib,\ 
${catalina.home}/lib/*.jar,\ 
${catalina.home}/../lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop/lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/lib/*.jar,\ /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-yarn/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hive/lib/*.jar

sqoop.properties文件的org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/配置hadoop 配置文件路径,默认为/etc/hadoop/conf

3. 测试配置是否正确

运行

sqoop2-tool verify

若正确,则输出为:

Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly

4. 下载derby 包

自带derby版本过低运行时会由于不兼容而出现以下错误:

org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)
    at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:85)
    at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)
    at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:127)
    at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)
    at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
    at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
    at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:75)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.sqoop.tomcat.TomcatToolRunner.main(TomcatToolRunner.java:77)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.catalina.startup.Tool.main(Tool.java:225)
Caused by: java.sql.SQLException: No suitable driver found for 

解决办法为下载derby最新版本,先删除/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib下的derby旧包,然后把新下载的derby目录下的lib下的jar包拷贝到/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib

5. 验证

运行bin/sqoop2-shell,进入sqoop shell模式, 运行show version --all,若能正确输出server版本,则安装成功:

sqoop:000> show version --all
client version:
  Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8
  Compiled by root on Wed Apr 29 10:40:43 CST 2015
server version:
  Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8
  Compiled by root on Wed Apr 29 10:40:43 CST 2015
API versions:
  [v1]
转载请注明:http://krystism.is-programmer.com/若有错误,请多多指正,谢谢!
  • 无匹配
  • 无匹配
cleaning companies i 说:
2019年9月18日 16:11

Still, if you're looking for “regular housecleaning, ” you have to pick maid housecleaning services in which charge usually while using number about maids, and therefore the time requested. Maid housecleaning companies will probably price most of the services located at 25 to make sure you 35 AED each hour.

handi sports 说:
2019年11月15日 01:10

Credit card debt financing is just about the most common solution to finance your organization. You sign up for a bank loan and consent to repay over a lot of time. The financial institution charges attention and tends to make a profit like that. The most frequent business lenders will be the following:

star lite shopping m 说:
2020年3月24日 20:37

On the net shopping would be the perfect solution with the home limited and if you find shopping a challenging task. Online looking has turned out to be a boon if you live with rural regions too. Whether you will be buying directly at a business on the net retailer, anyone online looking site, or maybe an World-wide-web auction web page, shopping online is usually fun, uncomplicated, time protecting and cost effective.

shopping batalha 说:
2020年3月24日 20:37

Accomplishing this prior to a exchange online is possibly the most critical into a shopping cart's achievements. If this buying practice causes annoyance, confusion or maybe insecurity, the user will abandon the shopping cart software package, never to return again.

elimperio travel 说:
2020年3月24日 20:37

In order to make the the vast majority of your management and business travel funds, it is vital to cover leveraging ones program for everyone it might be priced at. Telling travelers to pick out the most competitive logical airfare is not enough. Here are classified as the elements to be considered as soon as planning or maybe evaluating ones travel method.

travelling 2 peru 说:
2020年3月24日 20:38

An incredible option is to get a 'personalised' on the net booking tool that was custom made in line with you company's particular traveling policies in addition to needs. Although taking your own traveling tech possesses its strengths - like the ability to increase proficiency with inbuilt approval operations and coping with your travel whenever they want - here are a few disadvantages to be noted.

rapidity news 说:
2020年3月24日 20:38

Telly is a different medium connected with news in addition to current affairs in a country or maybe locality; more reguarily news is it being brought dwell and real-time. However, internet may bring such advantages for news people and followers. Aside by full articles or blog posts on favorite news, you will discover websites which often enable viewers to see news including what they will find with TV.

monthly maid service 说:
2020年4月29日 18:28

Also you can enquire towards previous buyers about if thez maid provides her unique vacuums, cleansing agents, mops, towels, pails in addition to polishes or it is advisable to provide these individuals. Make sure to supply a exclusive training or maybe advice n the best way to clean this antique pieces of furniture, oriental carpeting and rugs, and very sensitive show portions; also reveal them the way to handle with them.

construction paintin 说:
2020年4月29日 18:28

Symbol painting manufactured with the effort in addition to talent connected with portrait painters. Without these individuals, this style of art can never exist. Really, portrait painters have many contribution from the art marketplace. Agree or maybe not, i am the people that harvest the goods of the skills. That's why, we experience the appearance connected with portraits particularly when they usually are imitated on the works connected with famous symbol painters.


登录 *


loading captcha image...
(输入验证码)
or Ctrl+Enter