sqoop1实战 spring集成sqoop

🧙 Questions

spring集成sqoop1

☄️ Ideas

安装依赖

1. 下载sqoop.jar

scp ispong@isxcode:/data/sqoop/sqoop1/sqoop-1.4.7.jar ./

2. 安装依赖

mvn install:install-file -DgroupId=org.apache.sqoop -DartifactId=sqoop -Dversion=1.4.7 -Dpackaging=jar -Dfile=sqoop-1.4.7.jar

3. 导入依赖

<dependency>
	<groupId>org.apache.sqoop</groupId>
    <artifactId>sqoop</artifactId>
    <version>1.4.7</version>
</dependency>

mysql到hive

public void test( ){
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS", "hdfs://isxcode:30110");

    String curClasspath = System.getProperty("java.class.path");
    curClasspath = curClasspath
        + File.pathSeparator
        + "/data/sqoop/sqoop1/sqoop-1.4.7.jar"
        + File.pathSeparator
        + "/data/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0.jar"
        + File.pathSeparator
        + "/data/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.0.jar";

   String[] codeGenArgs = new String[]{
       "codegen",
       "--connect", "jdbc:mysql://isxcode:30002/data_assets",
       "--driver", "com.mysql.cj.jdbc.Driver",
       "--username", "ispong_data",
       "--password", "define2021",
       "--table", "ispong_demo",
       "--bindir","/data/sqoop/sqoop1/lib",
       "--outdir","/data/sqoop/sqoop1/lib"
   };

   Sqoop.runTool(codeGenArgs, conf);

    // 同步数据
    String[] syncArgs = new String[]{
        "export",
        "--connect", "jdbc:mysql://isxcode:30002/data_assets",
        "--driver", "com.mysql.cj.jdbc.Driver",
        "--username", "ispong_data",
        "--password", "define2021",
        "--table", "ispong_demo",
        "--export-dir", "/data/hive-data/warehouse/ispong_demo",
        "--m", "1"
    };

    Sqoop.runTool(syncArgs, conf);
}

常见错误

• java.lang.ClassNotFoundException: org.apache.avro.LogicalType

Caused by: java.lang.ClassNotFoundException: org.apache.avro.LogicalType
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[na:1.8.0_271]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[na:1.8.0_271]
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[ispong-admin-0.0.1.jar:0.0.1]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[na:1.8.0_271]
        ... 75 common frames omitted

解决方案
缺少依赖,项目中加入avro

<dependency>
     <groupId>org.apache.avro</groupId>
     <artifactId>avro</artifactId>
     <version>1.8.1</version>
</dependency>

• org.apache.hadoop.io不存在

/tmp/sqoop-isponger/compile/680e9f62d9a357db896da1ce931b60df/ispong_demo.java:7: 错误: 程序包org.apache.hadoop.io不存在
import org.apache.hadoop.io.BytesWritable;

解决方案
将hadoop的包放入 全局ClassPath中

sudo vim /etc/profile

export CLASSPATH=.:/data/hadoop-3.3.0/share/hadoop/common/hadoop-common-3.3.0.jar:/data/sqoop/sqoop1/sqoop-1.4.7.jar:/data/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.0.jar

source /etc/profile

• Input path does not exist

2021-03-13 17:11:50.848 ERROR 6307 --- [io-30104-exec-1] org.apache.sqoop.tool.ExportTool         : Encountered IOException running export job: 

org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/data/hive-data/warehouse/ispong_demo
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:340)

解决方案:
权限不够

🔗 Links

• sqoop website