调度器——DolphinScheduler讲解及安装教程

调度器——DolphinScheduler讲解及安装教程

一：基本讲解

Dolphin Scheduler

1、开源的分布式任务调度系统
2、支持多种任务类型，包括Shell、Spark、Hive等
3、灵活的任务调度功能和友好的Web界面，方便管理和监控任务的执行情况

架构

操作系统版本要求

硬件要求

二：安装教程

前期准备工作

• jdk 1.8+
• mysql 5+
• zookeeper 3.4.6+
• hadoop 2.6+
• psmisc yum -y install psmisc

解压安装包

# 将安装包apache-dolphinscheduler-2.0.8-bin.tar.gz放置/opt/download目录下
# 解压缩
tar -zxvf apache-dolphinscheduler-2.0.8-bin.tar.gz -C /opt/software

重命名

mv apache-dolphinscheduler-2.0.8-bin/ dolphinscheduler-2.0.8

配置一键部署脚本

vim conf/config/install_config.conf 

ips="single" 
# 将要部署任一 DolphinScheduler 服务的服务器主机名或 ip 列表sshPort="22"masters="single" 
# master 所在主机名列表，必须是 ips 的子集workers="single:default" 
# worker 主机名及队列，此处的 ip 必须在 ips 列表中alertServer="single"
# 告警服务所在服务器主机名apiServerPort="12345"
apiServers="single"
# api服务所在服务器主机名installPath="/opt/software/dolphinscheduler-2.0.8-install/install"
# DS 安装路径，如果不存在会创建deployUser="root"
# 部署用户dataBasedirPath="/opt/software/dolphinscheduler-2.0.8-install/data"
# 前文配置的所有节点的本地数据存储路径，需要确保部署用户拥有该【目录】的读写权限javaHome="/opt/software/jdk1.8.0_171"
# JAVA_HOME 路径DATABASE_TYPE="mysql"
# 数据库类型SPRING_DATASOURCE_URL="jdbc:mysql://single:3306/dolphinscheduler?createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=UTF-8"
# 数据库 URLSPRING_DATASOURCE_USERNAME="用户名"
# 数据库用户名，有远程访问权限（远程连接）SPRING_DATASOURCE_PASSWORD="密码"
# 数据库密码（远程连接）registryPluginName="zookeeper"
# 注册中心插件名称，DS 通过注册中心来确保集群配置的一致性
registryServers="single:2181"
# 注册中心地址，即 Zookeeper 集群的地址
registryNamespace="dolphinscheduler"
# DS 在 Zookeeper 的结点名称# ---------------------------------------------------------
# Worker Task Server
# ---------------------------------------------------------
taskPluginDir="lib/plugin/task"resourceStorageType="HDFS"  
# 资源存储类型
resourceUploadPath="/dolphinscheduler"
# 资源上传路径,请确保该路径在HDFS已创建
defaultFS="hdfs://single:9000"
# 默认文件系统，此处为单机，集群默认端口为8020# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"
# yarn RM http 访问端口# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
yarnHaIps=
# Yarn RM 高可用 ip，若未启用 RM 高可用，则将该值置空# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="single"
# Yarn RM 主机名，若启用了 HA 或未启用 RM，保留默认值# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="root"
# 拥有 HDFS 根目录操作权限的用户# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username,watch out the @ sign should followd by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"
# kerberos认证，防止恶意用户伪装成其他用户侵入集群或提交作业# use sudo or not
sudoEnable="true"# worker tenant auto create
workerTenantAutoCreate="false"

初始化数据库

因 SPRING_DATASOURCE_URL 配置中追加 createDatabaseIfNotExist=true，所以无需手动建库

拷贝MySQL驱动

cp /opt/download/mysql-connector-j-8.0.33.jar /opt/software/dolphinscheduler-2.0.8/lib

执行数据库初始化脚本

/opt/software/dolphinscheduler-2.0.8/script/create-dolphinscheduler.sh

一键部署

启动ZK后一键部署并启动

# 启动zookeeper
zkServer.sh start
# 启动dolphinscheduler
/opt/software/dolphinscheduler-2.0.8/install.sh

查看进程

# 单机
3668 QuorumPeerMain
4614 AlertServer
4520 WorkerServer
4662 ApiApplicationServer
4567 LoggerServer
4473 MasterServer

访问 Web UI

http://single:12345/dolphinscheduler

初始用户：admin
初始密码：dolphinscheduler123

一键开关命令

/opt/software/dolphinscheduler-2.0.8/bin/start-all.sh
/opt/software/dolphinscheduler-2.0.8/bin/stop-all.sh