网站链接: element-ui dtcms
当前位置: 首页 > 技术博文  > 技术博文

spark安装教程

2021/6/28 14:37:45 人评论

1.下载Spark安装包 # 下载Spark cd /export/softwares wget https://archive.apache.org/dist/spark .将spark上传到opt目录中 解压缩:tar -zxvf spark-3.1.1-bin-hadoop2.7.tgz ​ 查看:ll 2.将1000权限改成root权限 chown -R root /opt/spark-3.1…

1.下载Spark安装包

# 下载Spark
cd /export/softwares
wget https://archive.apache.org/dist/spark

.将spark上传到opt目录中

解压缩:tar -zxvf spark-3.1.1-bin-hadoop2.7.tgz
​
查看:ll

2.将1000权限改成root权限

chown -R root /opt/spark-3.1.1-bin-hadoop2.7
​
chgrp -R root /opt/spark-3.1.1-bin-hadoop2.7

3.改名

ln -s /opt/spark-3.1.1-bin-hadoop2.7 /opt/spark

4.测试

cd conf/   
​
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
export SCALA_HOME=/usr/local/scala/scala-2.13.5
export JAVA_HOME=/usr/local/java/jdk1.8.0_162
export SPARK_MASTER_IP=192.168.109.137
export SPARK_HOME=/opt/spark
export HADOOP_HOME=/opt/hadoop
export HADOOP_CONF_DIR=/opt/hadoop-2.6.4/etc/hadoop
export SPARK_MASTER_HOST=node1

export SPARK_MASTER_PORT=7077 

export SPARK_MASTER_WEBUI_PORT=8080

export SPARK_WORKER_CORES=1

export SPARK_WORKER_MEMORY=1g 

5.在conf目录下打开workers.template或者(slaves.template)

vi workers.template
​
添加:
worker1
worker2

6.启动spark交互式窗口

cd bin/
/opt/spark/bin/spark-shell  (./spark-shell
​
进入http://master22:4040

7.添加数据

(1)准备数据:vim  /root/words.txt
​
hello me you her
hello me you
hello me
hello
(2)查看数据
cat words.txt

8.执行WorkCount

val textFile = sc.textFile("file:///root/words.txt")
val counts = textFile.flatMap(_.split("")).map((_,1))reduceByKey(_+_)
counts.collect

添加环境(回到spark目录)

vim /etc/profile
​
export SPARK_HOME=/opt/spark
export PATH=${SPARK_HOME}/bin:$PATH
​
source /etc/profile

相关资讯

    暂无相关的数据...

共有条评论 网友评论

验证码: 看不清楚?