cdh教程 集群安装

Last updated on September 15, 2024 pm

🧙 Questions

安装cdh(6.2.0)

服务器信息
name ip(public) ip(private)
main 47.99.126.247 172.18.183.230
node1 121.41.114.73 172.19.17.27
node2 120.26.58.239 172.30.31.232
安装前提
  • 创建cdh用户,赋予password-less sudo权限
sudo useradd cdh
sudo passwd cdh
sudo vim /etc/sudoers
# === sudo vim /etc/sudoers ===
cdh   ALL =(ALL) NOPASSWD: ALL
# === sudo vim /etc/sudoers ===
  • 下载并上传cdh安装包
#链接: https://pan.baidu.com/s/1Fg7YxcWubrhJx-YjETaGMQ 
#提取码: 9iy8
scp cdh6.2.0.zip cdh@47.99.126.247:~/
  • 下载并上传rpm相关安装包(自行百度下载离线安装包)
# 主机点安装
sudo yum install -y unzip
sudo yum install -y bind-utils
sudo yum install -y httpd
sudo yum install -y createrepo
  • 安装java环境(最好使用openjdk-1.8.0)

Note: 自己手动下载安装jdk，cdh可能无法识别，所以直接使用openjdk安装 

sudo yum install -y java-1.8.0-openjdk-devel java-1.8.0-openjdk
  • 下载并上传mysql驱动
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.23.tar.gz
tar vzxf mysql-connector-java-8.0.23.tar.gz
sudo cp mysql-connector-java-8.0.23/mysql-connector-java-8.0.23.jar /usr/share/java/mysql-connector-java.jar
  • 安装并提供mysql数据库(自行百度)
sudo mkdir -p /data/mysql/data
sudo mkdir -p /data/mysql/conf.d
sudo docker run \
  --name cdh-mysql \
  --privileged=true \
  --restart=always \
  -d \
  -p 30102:3306 \
  -e MYSQL_ROOT_PASSWORD=cdh2021 \
  -e MYSQL_DATABASE=cdh \
  -v /data/mysql/data:/var/lib/mysql \
  -v /data/mysql/conf.d:/etc/mysql/conf.d \
  mysql
  • 安装python环境(一般系统自带最好python2.7.5)
sudo yum install python

修改hostname

# 三个节点都要配置
sudo vim /etc/hosts
# === sudo vim /etc/hosts ===
172.18.183.230 main
172.19.17.27   node1
172.30.31.232  node2
# === sudo vim /etc/hosts ===

# 对应各自的节点做配置
sudo hostnamectl set-hostname ${hostName}
sudo vim /etc/sysconfig/network 
# === vim /etc/sysconfig/network ===
HOSTNAME=${hostName}
# === vim /etc/sysconfig/network ===

关闭防火墙

# sudo systemctl status firewalld
sudo systemctl disable firewalld
sudo systemctl stop firewalld
sudo systemctl status firewalld

关闭selinux

Note: 防止重启后，自动启动防火墙

sudo vim /etc/selinux/config

# === sudo vim /etc/selinux/config ===
SELINUX=disabled
# === sudo vim /etc/selinux/config ===

# getenforce
setenforce 0

安装scm

# 主节点安装 按顺序运行
sudo yum install -y cdh6.2.0/cloudera-repos-6.2.0/cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm 
sudo yum install -y cdh6.2.0/cloudera-repos-6.2.0/cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
sudo yum install -y cdh6.2.0/cloudera-repos-6.2.0/cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm

创建scm数据库

-- docker exec -it cdh-mysql /bin/bash
-- mysql -u  root -p
CREATE DATABASE cdh_scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE cdh_hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE cdh_monitor DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

初始化scm数据库

# sudo /opt/cloudera/cm/schema/scm_prepare_database.sh -h main -P 30102 mysql cdh_scm root cdh2021 
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh -h ${host} -P ${port} mysql ${dbName} ${username} ${password}

安装httpd

Note: 如果报错先不用管,可能因为80端口被占用

# 切换到su用户下
systemctl start httpd
# 修改端口号
vim /etc/httpd/conf/httpd.conf
# / Listen
# === vim /etc/httpd/conf/httpd.conf ===
#Listen 12.34.56.78:80
Listen 30108
# === vim /etc/httpd/conf/httpd.conf ===
systemctl restart httpd

上传cdh6_parcel文件

sudo mkdir -p /var/www/html/cdh6_parcel
sudo cp cdh6.2.0/parcel-6.2.0/* /var/www/html/cdh6_parcel
sudo mkdir -p /var/www/html/cdh6_parcel/cm6
sudo cp cdh6.2.0/cloudera-repos-6.2.0/* /var/www/html/cdh6_parcel/cm6/

# 测试安装
# parcel镜像地址 http://172.18.183.230:30108/cdh6_parcel/
# cm6镜像地址 http://172.18.183.230:30108/cdh6_parcel/cm6

创建本地镜像仓库

cd /var/www/html/cdh6_parcel/cm6/
sudo createrepo .
cd /etc/yum.repos.d
sudo vim cloudera-manager.repo
# === sudo vim cloudera-manager.repo ===
[cloudera-manager]
name=Cloudera Manager 6.2.0
baseurl=http://172.18.183.230:30108/cdh6_parcel/cm6
gpgcheck=0
enabled=1
# === sudo vim cloudera-manager.repo ===
yum clean all && yum makecache

启动scm

sudo systemctl enable cloudera-scm-server
sudo systemctl start cloudera-scm-server
# sudo systemctl status cloudera-scm-server

查看scm日志

sudo tail -F /var/log/cloudera-scm-server/cloudera-scm-server.log

修改scm访问端口

-- INSERT INTO CONFIGS (CONFIG_ID, ATTR, VALUE, CONFIG_CONTAINER_ID) VALUES (4, 'http_port', '30107', 2);
INSERT INTO CONFIGS (CONFIG_ID, ATTR, VALUE, CONFIG_CONTAINER_ID) VALUES (4, 'http_port', '${customPort}', 2);
# sudo systemctl stop cloudera-scm-server
# sudo systemctl stop cloudera-scm-agent
# sudo systemctl restart cloudera-scm-agent
sudo systemctl restart cloudera-scm-server

测试

默认端口号 7180

cloudera
cdh教程 集群安装
https://ispong.isxcode.com/hadoop/cloudera/cdh教程 集群安装/
Author
ispong
Posted on
January 20, 2022
Licensed under