doris hive同步到doris

🧙 Questions

将hive中的数据导入doris中

☄️ Ideas

创建hive中的表

-- beeline -n ispong -p ispong123 -u jdbc:hive2://localhost:10000

DROP TABLE default.demo_table;
CREATE TABLE IF NOT EXISTS default.demo_table(
    username   STRING  COMMENT 'comment for username',
    age        INT     COMMENT 'comment for age'
)
ROW FORMAT 
DELIMITED FIELDS TERMINATED BY '\001'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;

INSERT INTO default.demo_table ( username, age ) VALUES ('ispong', 218);

创建doris中的表

-- docker exec -it isxcode-mysql bash
-- mysql -u root -h 127.0.0.1 -P 9030

CREATE DATABASE demo;
CREATE TABLE demo.demo_table_doris
(
  username  VARCHAR (32) ,
  age       BIGINT
)
UNIQUE KEY (username)
DISTRIBUTED BY HASH(username) BUCKETS 10
PROPERTIES("replication_num" = "1");

创建load开始同步

Note: hive中的\001 –> \\x01
默认分隔符 \t

-- 查看可用broker
SHOW PROC '/brokers';

-- 开始同步
LOAD LABEL ispong_label_14 (
    DATA INFILE("hdfs://localhost:9000/user/hive/warehouse/demo_table/*")
    INTO TABLE demo_table_doris
    COLUMNS TERMINATED BY '\\x01'
    (
      username,
      age
    )
    SET (
      username = username,
      age = age
    ) where username = 'ispong'
) WITH BROKER 'ispong_broker' (
  "username" = "",
  "password" = ""  
) PROPERTIES (
  "timeout"="3600",
  "max_filter_ratio"="0.1",
  "timezone"="Asia/Shanghai"
);

-- 查看任务是否成功
SHOW LOAD WHERE LABEL = "ispong_label_8";

-- 取消运行过程中
CANCEL LOAD WHERE LABEL = "ispong_label";

🔗 Links

• doris docs
• doris load docs