spark 单节点极速安装

Last updated on September 15, 2024 pm

🧙 Questions

spark 3.4.0 版本单节点极速安装

☄️ Ideas

前提

下载

cd /tmp 
nohup wget https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.tgz >> download_spark.log 2>&1 &  
tail -f download_spark.log

创建安装目录

sudo mkdir -p /data/spark
sudo chown -R ispong:ispong /data/spark

解压并创建软连接

tar -vzxf spark-3.4.0-bin-hadoop3.tgz -C /data/spark
sudo ln -s /data/spark/spark-3.4.0-bin-hadoop3 /opt/spark

配置环境变量

sudo tee -a /etc/profile <<-'EOF'
export SPARK_HOME=/opt/spark 
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
EOF
source /etc/profile

查看版本信息

spark-submit --version

配置spark-defaults.conf

cp /opt/spark/conf/spark-defaults.conf.template /opt/spark/conf/spark-defaults.conf
tee -a /opt/spark/conf/spark-defaults.conf <<-'EOF'
spark.master          spark://isxcode:7077
spark.master.web.url  http://isxcode:8081
EOF

配置spark-env.sh

cp /opt/spark/conf/spark-env.sh.template /opt/spark/conf/spark-env.sh
tee -a /opt/spark/conf/spark-env.sh <<-'EOF'
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8081
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
EOF

配置worker

cp /opt/spark/conf/workers.template /opt/spark/conf/workers
vim /opt/spark/conf/workers

isxcode

配置免密登录

# 一路回车
ssh-keygen
ssh-copy-id ispong@isxcode

启动服务

最好到sbin到目录下运行，避免和hadoop的命令冲突
停止
cd /opt/spark/sbin
bash stop-all.sh
单独开启
bash start-master.sh
bash start-worker.sh spark://ispong-mac.local:7077

cd /opt/spark/sbin
bash start-all.sh

http://isxcode:8081

测试

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --deploy-mode cluster \
  /opt/spark/examples/jars/spark-examples_2.12-3.4.0.jar \
  100