sqoop2安装

int32位 posted @ Mar 22, 2016 03:42:06 PM in hadoop/spark , 1319 阅读
转载请注明:http://krystism.is-programmer.com/若有错误,请多多指正,谢谢!

 

1. 下载解压缩

此次安装版本为1.99.6

# Decompress Sqoop distribution tarball
tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz


ln -s sqoop-<version>-bin-hadoop<hadoop version>.tar.gz sqoop

export SQOOP_HOME=`pwd`/sqoop
# Change working directory
cd $SQOOP_HOME

2. 配置服务

配置文件在$SQOOP_HOME/server/conf下,大多数默认配置即可。需要注意的配置是catalina.properties文件下common_loader需要正确配置hadoop库文件,包括hdfs、mapreduce、hive等所有jar包. 我的配置为:

common.loader=${catalina.base}/lib,\ 
${catalina.base}/lib/*.jar,\ 
${catalina.home}/lib,\ 
${catalina.home}/lib/*.jar,\ 
${catalina.home}/../lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop/lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/lib/*.jar,\ /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-yarn/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/*.jar,\ 
/opt/cloudera/parcels/CDH/lib/hive/lib/*.jar

sqoop.properties文件的org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/配置hadoop 配置文件路径,默认为/etc/hadoop/conf

3. 测试配置是否正确

运行

sqoop2-tool verify

若正确,则输出为:

Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly

4. 下载derby 包

自带derby版本过低运行时会由于不兼容而出现以下错误:

org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)
    at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:85)
    at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)
    at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:127)
    at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)
    at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
    at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
    at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:75)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.sqoop.tomcat.TomcatToolRunner.main(TomcatToolRunner.java:77)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.catalina.startup.Tool.main(Tool.java:225)
Caused by: java.sql.SQLException: No suitable driver found for 

解决办法为下载derby最新版本,先删除/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib下的derby旧包,然后把新下载的derby目录下的lib下的jar包拷贝到/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib

5. 验证

运行bin/sqoop2-shell,进入sqoop shell模式, 运行show version --all,若能正确输出server版本,则安装成功:

sqoop:000> show version --all
client version:
  Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8
  Compiled by root on Wed Apr 29 10:40:43 CST 2015
server version:
  Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8
  Compiled by root on Wed Apr 29 10:40:43 CST 2015
API versions:
  [v1]
转载请注明:http://krystism.is-programmer.com/若有错误,请多多指正,谢谢!
  • 无匹配
  • 无匹配
cleaning companies i 说:
2019年9月18日 16:11

Still, if you're looking for “regular housecleaning, ” you have to pick maid housecleaning services in which charge usually while using number about maids, and therefore the time requested. Maid housecleaning companies will probably price most of the services located at 25 to make sure you 35 AED each hour.

handi sports 说:
2019年11月15日 01:10

Credit card debt financing is just about the most common solution to finance your organization. You sign up for a bank loan and consent to repay over a lot of time. The financial institution charges attention and tends to make a profit like that. The most frequent business lenders will be the following:


登录 *


loading captcha image...
(输入验证码)
or Ctrl+Enter