sqoop2实战 常见错误

🧙 Questions

• com.google.common.base.Preconditions.checkArgument
• org.apache.derby.impl.jdbc.authentication
• configuration directory

☄️Ideas

com.google.common.base.Preconditions.checkArgument

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1380)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1361)
        at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:37)
        at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
        at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
        at org.apache.sqoop.server.SqoopJettyServer.<init>(SqoopJettyServer.java:67)
        at org.apache.sqoop.server.SqoopJettyServer.main(SqoopJettyServer.java:177)

解决方案

hadoop和hive版本过高,需要替换掉guava-27.0-jre.jar文件

# server
rm /data/sqoop/sqoop/server/lib/guava-11.0.2.jar 
cp /data/apache-hive-3.1.2-bin/lib/guava-27.0-jre.jar /data/sqoop/sqoop/server/lib/

# tools
rm /data/sqoop/sqoop/shell/lib/guava-11.0.2.jar 
cp /data/apache-hive-3.1.2-bin/lib/guava-27.0-jre.jar /data/sqoop/sqoop/shell/lib/

# shell
rm /data/sqoop/sqoop/tools/lib/guava-11.0.2.jar 
cp /data/apache-hive-3.1.2-bin/lib/guava-27.0-jre.jar /data/sqoop/sqoop/tools/lib/

org.apache.derby.impl.jdbc.authentication

Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.impl.jdbc.authentication is sealed
        at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:400)
        at java.net.URLClassLoader.definePackageInternal(URLClassLoader.java:420)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:452)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.derby.impl.services.monitor.BaseMonitor.getImplementations(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.getDefaultImplementations(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.runWithState(Unknown Source)
        at org.apache.derby.impl.services.monitor.FileMonitor.<init>(Unknown Source)
        at org.apache.derby.iapi.services.monitor.Monitor.startMonitor(Unknown Source)
        at org.apache.derby.iapi.jdbc.JDBCBoot.boot(Unknown Source)
        at org.apache.derby.jdbc.EmbeddedDriver.boot(Unknown Source)
        at org.apache.derby.jdbc.EmbeddedDriver.<clinit>(Unknown Source)
        ... 11 more

解决方案

derby jar包重复,删除hive或者sqoop的低版本的derby(最好备份) derby-10.14.1.0.jar

rm /data/sqoop/sqoop/server/lib/derby-10.8.2.2.jar

configuration directory

Caused by: org.apache.sqoop.common.SqoopException: MAPREDUCE_0002:Failure on submission engine initialization - Invalid Hadoop configuration directory (not a directory or permission issues): /etc/hadoop/conf/
        at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:97)
        at org.apache.sqoop.driver.JobManager.initialize(JobManager.java:257)
        at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:64)
        ... 2 more

解决方案

sqoop配置中的hadoop配置异常,没有配置hadoop配置文件

vim /conf/sqoop.properties

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/data/hadoop-3.3.0/etc/hadoop

🔗 Links