livy 部署

环境信息

使用的 hadoop 完全分布式集群

192.168.2.241 hadoop01
192.168.2.242 hadoop02
192.168.2.243 hadoop03

简介

Livy 是一个基于 Spark 的开源 REST 服务，它能够通过 REST 的方式将代码片段或是序列化的二进制代码提交到 Spark 集群中去执行

Livy 安装

官网 https://livy.incubator.apache.org/get-started/

hadoop03 节点执行 (不兼容 spark-3.2.1，0.8.0版本可用)

wget https://dlcdn.apache.org/incubator/livy/0.7.1-incubating/apache-livy-0.7.1-incubating-bin.zip  --no-check-certificate

mkdir -p /opt/bigdata/livy
unzip apache-livy-0.7.1-incubating-bin.zip  -d /opt/bigdata/livy
cd /opt/bigdata/livy/
ln -s apache-livy-0.7.1-incubating-bin current
cd /opt/bigdata/livy/current
cp conf/livy.conf.template conf/livy.conf
cp conf/livy-env.sh.template conf/livy-env.sh

源码编译 后,传文件到 hadoop03 上

mkdir -p /opt/bigdata/livy
unzip apache-livy-0.8.0-incubating-SNAPSHOT-bin.zip  -d /opt/bigdata/livy
cd /opt/bigdata/livy/
ln -s apache-livy-0.8.0-incubating-SNAPSHOT-bin current
cd /opt/bigdata/livy/current
cp conf/livy.conf.template conf/livy.conf
cp conf/livy-env.sh.template conf/livy-env.sh

修改配置文件
/opt/bigdata/livy/current/conf/livy-env.sh

# export HADOOP_CONF_DIR=/etc/hadoop/conf
# export SPARK_HOME=/opt/bigdata/spark/current
# 变量名要用下划线，参数是 -Xmx（小写 x）；而且下面那行如果再 export 一次会把这里整个覆盖掉，所以合成一条写：

export HADOOP_HOME=/opt/bigdata/hadoop/current
export HADOOP_CONF_DIR=/opt/bigdata/hadoop/current/etc/hadoop
export SPARK_HOME=/opt/bigdata/spark/current
export SPARK_CONF_DIR=/opt/bigdata/spark/current/conf
export PYSPARK_PYTHON=/usr/bin/python
export LIVY_SERVER_JAVA_OPTS="-Xmx2g -Djava.library.path=$HADOOP_HOME/lib/native"

修改配置文件
/opt/bigdata/livy/current/conf/livy.conf

livy.spark.master = yarn
# cluster 模式会报错，这里用 client
# 注意 livy.conf 是 Java properties 格式（Properties.load 读取），不支持行内注释：
# 写成 "client # cluster" 的话 # 后面的内容会成为值的一部分，
# 实际取到 "client # cluster  会报错，"，传给 spark-submit 会 Unknown deploy mode 失败。
livy.spark.deployMode = client
livy.environment = production
livy.impersonation.enabled = true
livy.server.csrf_protection.enabled false
livy.server.port = 8998
livy.server.session.timeout = 3600000
livy.server.recovery.mode = recovery
livy.server.recovery.state-store=filesystem
livy.server.recovery.state-store.url=/tmp/livy
livy.repl.enable-hive-context = true
livy.rsc.rpc.server.address=192.168.2.243

注意事项, 需要给 spark 添加如下配置

${SPARK_HOME}/conf/spark-defaults.conf # 使用 IP地址

spark.driver.host 192.168.2.243

使用 IP地址原因

livy 会 用到 域名.cluster.local 并没有配置，可能默认使用到了

spark 用 域名 会被 yarn kill 掉暂未 定位 原因

启动

bash bin/livy-server stop && bash bin/livy-server start

验证

curl -X POST --data '{"kind": "spark", "numExecutors": 2, "executorMemory": "1G"}' -H "Content-Type: application/json" -H "X-Requested-By: user" hadoop03:8998/sessions


curl -X POST --data '{"code":"sc.textFile(\"/tmp/a.txt\").collect().foreach(println)"}' -H "Content-Type:application/json" hadoop03:8998/sessions/0/statements

curl -X GET hadoop03:8998/sessions/0/statements/0

测试 spark-hive

[root@imwl-02 ~]# curl -X POST --data '{"kind": "spark", "numExecutors": 2, "executorMemory": "1G"}' -H "Content-Type: application/json" -H "X-Requested-By: user" 192.168.2.132:8998/sessions
{"id":0,"name":null,"appId":null,"owner":null,"proxyUser":null,"state":"starting","kind":"spark","appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: "]}


[root@imwl-02 ~]#   curl -X POST -H "Content-Type:application/json" -d '{"code":"spark.sql(\"show databases\").show()"}' http://192.168.2.132:8998/sessions/0/statements
{"id":0,"code":"spark.sql(\"show databases\").show()","state":"waiting","output":null,"progress":0.0,"started":0,"completed":0}

[root@imwl-02 ~]#   curl -X POST -H "Content-Type:application/json" -d '{"code":"spark.sql(\"use dw_label\").show()"}' http://192.168.2.132:8998/sessions/0/statements
{"id":1,"code":"spark.sql(\"use dw_label\").show()","state":"waiting","output":null,"progress":0.0,"started":0,"completed":0}

[root@imwl-02 ~]#   curl -X POST -H "Content-Type:application/json" -d '{"code":"spark.sql(\"show create table dwd_dls_cust_group\").show()"}' http://192.168.2.132:8998/sessions/0/statements
{"id":2,"code":"spark.sql(\"show create table dwd_dls_cust_group\").show()","state":"waiting","output":null,"progress":0.0,"started":0,"completed":0}


[root@imwl-02 ~]#   curl -X GET 192.168.2.132:8998/sessions/0/statements/2
{"id":2,"code":"spark.sql(\"show create table dwd_dls_cust_group\").show()","state":"available","output":{"status":"ok","execution_count":3,"data":{"text/plain":"+--------------------+\n|      createtab_stmt|\n+--------------------+\n|CREATE TABLE `dw_...|\n+--------------------+\n\n"}},"progress":1.0,"started":1680770029776,"completed":1680770030162}

遇到的问题

1. java.lang.IllegalArgumentException: Code type should be specified if session kind is shared

使用共享会话时，需要明确指定代码的类型。ExecuteRequest 中只有 4 个已知字段：msg_type、kind、code、timeoutMs

code

Scala (kind: "spark")："val df = spark.read.json(\"path_to_file.json\"); df.show()"
PySpark (kind: "pyspark")："df = spark.read.json('path_to_file.json'); df.show()"
SparkR (kind: "sparkr")："df <- read.df('path_to_file.json', source = 'json'); showDF(df)"
SQL (kind: "sql")："SHOW DATABASES"

修改如下

curl -X POST -H "Content-Type:application/json" -d '{"code":"spark.sql(\"show databases\").show()", kind": "spark"}' http://http://192.168.2.132:8998/sessions/70/statements

2. livyserver session 失败，shut_down，报错空指针

删除 rsc-jars 目录多余的文件和目录，只留使用到的jar 包

livy-*.jar
netty-all*.jar
gree-*.jar
*jdbc*jar

适配FI

livy适配华为FI大数据集群

前提：已安装华为FI客户端

一、准备Spark客户端

拷贝华为FIspark客户端
执行：cp -R 华为FI大数据客户端路径/Spark2x/spark   /opt/bigdata/spark_hw
将conf目录下非必要的配置如：java-opts,hdfs-site.xml,jdbc-log4j.properties，jets3t.properties等删除



二、准备Spark镜像

使用如下Dockerfile：



FROM spark-py:3.1.2
ENV TIME_ZONE Asia/Shanghai
RUN mkdir /var/test && groupadd --gid 3000 test && useradd --uid 3000 --gid test --shell /bin/bash --home-dir /var/test test  && chown 3000:3000 /var/test && usermod -a -G root test && ln -snf /usr/share/zoneinfo/$TIME_ZONE /etc/localtime && echo $TIME_ZONE > /etc/timezone && ln -s /usr/local/bin/python3  /usr/local/bin/python

RUN /bin/bash -c 'rm -rf /opt/spark/jars/*'
RUN /bin/bash -c 'rm -rf /opt/spark/bin/*'
RUN /bin/bash -c 'rm -rf /opt/spark/sbin/*'

##注意：Dockerfile文件的同级目录jars bin sbin是从华为FI spark客户端拷贝过来的
##jar包冲突处理：华为FI spark客户端jar包中是scala2.12.14,okhttp4.9.3;参照开源版本使用scala2.12.10，okio1.14.0对应使用低版本okhttp2.7.5或者okhttp3.12.12
## 华为spark的jars中gsjdbc4-V100R003C10SPC125.jar与开源pg驱动包postgresql-x.jar互相冲突
COPY jars/  /opt/spark/jars/
COPY bin/  /opt/spark/bin/
COPY sbin/  /opt/spark/sbin/
##官方开源镜像是jdk11为适配华为相关包需要降级为java8
ADD jdk-8u332-linux-x64.tar.gz /opt/jdk8

ENV JAVA_HOME /opt/jdk8/jdk1.8.0_332
ENV CLASSPATH $JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH $PATH:$JAVA_HOME/bin

USER test
到Dockerfile所在目录,执行：
docker build -t spark-py-java8:3.1.2  .
docker save  -o /mnt/test/spark-py-java8-2-3.1.2.tar spark-py-java8:3.1.2

集群内其它机器导入镜像执行：
docker load -i /mnt/test/spark-py-java8-2-3.1.2.tar



三、修改spark客户端的配置

配置spark-defaults.conf到/opt/bigdata/spark_hw/conf
修改：spark.kubernetes.container.image  spark-py-java8:3.1.2
配置内容为步骤2打出来的镜像文件



四、配置livy

配置：livy-env.sh

export HADOOP_HOME=华为FI大数据客户端路径/HDFS/hadoop
export HADOOP_CONF_DIR=华为FI大数据客户端路径/HDFS/hadoop/etc/hadoop
export SPARK_HOME=/opt/bigdata/spark_hw

注意：使用的kerberos相关的注意路径及指定krb5.conf
export LIVY_SERVER_JAVA_OPTS="-Djava.security.krb5.conf=/mnt/test/krb/krb5.conf"

spark-conf 下的 hive-site.xml 文件必须为Spark2X/spark/conf 目录下的hive-site.xml 文件，不能为 Hive/conf 目录下的hive-site.xml 文件，否则spark适配上的问题



注意：不要source过华为客户端的环境变量后去启动停止livy。