Hadoop生态圈-Azkaban部署实战
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.Azkaban的优势
二.Azkaban的搭建过程
1>.上传azkaban程序并创建解压目录
[yinzhengjie@s101 data]$ pwd /home/yinzhengjie/data [yinzhengjie@s101 data]$ ll total 401716 -rw-r--r-- 1 yinzhengjie yinzhengjie 16664491 Apr 16 15:37 azkaban-exec-server-3.46.0.tar.gz -rw-r--r-- 1 yinzhengjie yinzhengjie 24333 Apr 16 15:18 azkaban-hadoop-security-plugin-3.46.0.tar.gz -rw-r--r-- 1 yinzhengjie yinzhengjie 349929173 Apr 16 15:11 azkaban-plugins-master.zip -rw-r--r-- 1 yinzhengjie yinzhengjie 23819895 Apr 16 15:18 azkaban-solo-server-3.46.0.tar.gz -rw-r--r-- 1 yinzhengjie yinzhengjie 20899667 Apr 16 15:37 azkaban-web-server-3.46.0.tar.gz -rw-r--r-- 1 yinzhengjie yinzhengjie 11194 Nov 30 2017 create-all-sql-3.46.0.sql [yinzhengjie@s101 data]$ [yinzhengjie@s101 data]$ mkdir /soft/azkaban [yinzhengjie@s101 data]$
2>.分别解压文件到/soft/azkaban中
[yinzhengjie@s101 data]$ tar -xzf azkaban-exec-server-3.46.0.tar.gz -C /soft/azkaban/ [yinzhengjie@s101 data]$ tar -xzf azkaban-web-server-3.46.0.tar.gz -C /soft/azkaban/ [yinzhengjie@s101 data]$
3>.进入到mysql命令行并创建数据库
[yinzhengjie@s101 data]$ mysql -uroot -pyinzhengjie Warning: Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.6.38 MySQL Community Server (GPL) Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement. mysql> create database yinzhengjie_azkaban; Query OK, 1 row affected (0.00 sec) mysql> use yinzhengjie_azkaban; Database changed mysql> source /home/yinzhengjie/data/create-all-sql-3.46.0.sql Query OK, 0 rows affected (0.03 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.06 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.13 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.22 sec) Query OK, 0 rows affected (0.15 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.18 sec) Query OK, 0 rows affected (0.13 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.17 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.03 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.06 sec) Query OK, 0 rows affected (0.33 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.22 sec) Query OK, 0 rows affected (0.12 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.09 sec) Query OK, 0 rows affected (0.10 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.17 sec) Query OK, 0 rows affected (0.18 sec) Query OK, 0 rows affected (0.14 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.12 sec) Query OK, 0 rows affected (0.04 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.08 sec) Query OK, 0 rows affected (0.17 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.16 sec) Query OK, 0 rows affected (0.13 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.13 sec) Query OK, 0 rows affected (0.10 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.04 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected (0.19 sec) Query OK, 0 rows affected (0.23 sec) Query OK, 0 rows affected (0.08 sec) Query OK, 0 rows affected (0.13 sec) Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.06 sec) Query OK, 0 rows affected (0.19 sec) Query OK, 0 rows affected (0.27 sec) Query OK, 0 rows affected (0.18 sec) Query OK, 0 rows affected (0.16 sec) Query OK, 0 rows affected (0.17 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.19 sec) mysql> exit; Bye [yinzhengjie@s101 data]$
4>.创建SSL配置（完成下述工作后,将在当前目录生成 keystore 证书文件）
[yinzhengjie@s101 data]$ keytool -keystore keystore -alias jetty -genkey -keyalg RSA Enter keystore password: Re-enter new password: What is your first and last name? [Unknown]: What is the name of your organizational unit? [Unknown]: What is the name of your organization? [Unknown]: What is the name of your City or Locality? [Unknown]: What is the name of your State or Province? [Unknown]: What is the two-letter country code for this unit? [Unknown]: CN Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=CN correct? [no]: y Enter key password for <jetty> (RETURN if same as keystore password): [yinzhengjie@s101 data]$ [yinzhengjie@s101 data]$ ll | grep keystore -rw-rw-r-- 1 yinzhengjie yinzhengjie 2232 Jun 24 02:48 keystore [yinzhengjie@s101 data]$
5>.将keystore文件拷贝到web文件夹下
[yinzhengjie@s101 data]$ ll | grep keystore -rw-rw-r-- 1 yinzhengjie yinzhengjie 2232 Jun 24 02:48 keystore [yinzhengjie@s101 data]$ cp keystore /soft/azkaban/azkaban-web-server-3.46.0/ [yinzhengjie@s101 data]$
6>.编辑配置文件【azkaban.properties】
[yinzhengjie@s101 data]$ more /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban.properties #Azkaban Personalization Settings #服务器UI名称,用于服务器上方显示的名字 azkaban.name=yinzhengjie #描述 azkaban.label=My Local Azkaban #UI颜色 azkaban.color=#FF3601 #指定首页路径 azkaban.default.servlet.path=/index #默认根web目录 web.resource.dir=/soft/azkaban/azkaban-web-server-3.46.0/web/ #默认时区,已改为亚洲/上海 默认为美国 default.timezone.id=Asia/Shanghai #Azkaban UserManager class #用户权限管理默认类 user.manager.class=azkaban.user.XmlUserManager #用户配置,具体配置参加下文 user.manager.xml.file=/soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml #Loader for projects # global配置文件所在位置 executor.global.properties=/soft/azkaban/azkaban-web-server-3.46.0/conf/global.properties azkaban.project.dir=projects #数据库类型 database.type=mysql #端口号 mysql.port=3306 #数据库连接IP mysql.host=s101 #数据库实例名 mysql.database=yinzhengjie_azkaban #数据库用户名 mysql.user=root #数据库密码 mysql.password=yinzhengjie #最大连接数 mysql.numconnections=100 # Velocity dev mode #关闭快速开发模式 velocity.dev.mode=false # Jetty服务器属性. #最大线程数 jetty.maxThreads=25 #Jetty SSL端口 jetty.ssl.port=8443 #Jetty端口 jetty.port=8081 #SSL文件名 jetty.keystore=keystore #SSL文件密码 jetty.password=yinzhengjie #Jetty主密码 与 keystore文件相同 jetty.keypassword=yinzhengjie #SSL文件名 jetty.truststore=keystore # SSL文件密码 jetty.trustpassword=yinzhengjie # 执行服务器属性 #执行服务器端口 executor.port=12321 # 邮件设置 #发送邮箱 mail.sender=xxxxxxxx@163.com #发送邮箱smtp地址 mail.host=smtp.163.com #发送邮件时显示的名称 mail.user=xxxxxxxx #邮箱密码 mail.password=********** #任务失败时发送邮件的地址 job.failure.email=xxxxxxxx@163.com #任务成功时发送邮件的地址 job.success.email=xxxxxxxx@163.com lockdown.create.projects=false #缓存目录 cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban JobTypes 插件配置 azkaban.jobtype.plugin.dir=plugins/jobtypes [yinzhengjie@s101 data]$
7>.修改用户配置文件【azkaban-users.xml】
[yinzhengjie@s101 data]$ more /soft/azkaban/azkaban-web-server-3.46.0/conf/azkaban-users.xml <azkaban-users> <user username="azkaban" password="yinzhengjie" roles="admin" groups="azkaban" /> <user username="metrics" password="yinzhengjie" roles="metrics"/> <user username="admin" password="yinzhengjie" roles="admin,metrics" /> <role name="admin" permissions="ADMIN" /> <role name="metrics" permissions="METRICS"/> </azkaban-users> [yinzhengjie@s101 data]$
8>.创建global.properties
[yinzhengjie@s101 data]$ touch /soft/azkaban/azkaban-web-server-3.46.0/conf/global.properties [yinzhengjie@s101 data]$
9>.编写启动脚本
[yinzhengjie@s101 data]$ more /usr/local/bin/azweb.sh #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判断用户是否传参 if [ $# -ne 1 ];then echo "无效参数，用法为: $0 {start|stop}" exit fi #获取用户输入的命令 cmd=$1 case $cmd in start) /soft/azkaban/azkaban-web-server-3.46.0/bin/start-web.sh echo s$i "服务已启动" ;; stop) /soft/azkaban/azkaban-web-server-3.46.0/bin/shutdown-web.sh echo s$i "服务已停止" ;; *) echo "无效参数，用法为: $0 {start|stop}" exit ;; esac echo ========= azweb $cmd ============= [yinzhengjie@s101 data]$ [yinzhengjie@s101 data]$ [yinzhengjie@s101 data]$ sudo chmod a+x /usr/local/bin/azweb.sh [yinzhengjie@s101 data]$
[yinzhengjie@s101 data]$ more /usr/local/bin/azexec.sh #!/bin/bash #@author :yinzhengjie #blog:http://www.cnblogs.com/yinzhengjie #EMAIL:y1053419035@qq.com #判断用户是否传参 if [ $# -ne 1 ];then echo "无效参数，用法为: $0 {start|stop}" exit fi #获取用户输入的命令 cmd=$1 case $cmd in start) /soft/azkaban/azkaban-exec-server-3.46.0/bin/start-exec.sh echo s$i "服务已启动" ;; stop) /soft/azkaban/azkaban-exec-server-3.46.0/bin/shutdown-exec.sh echo s$i "服务已停止" ;; *) echo "无效参数，用法为: $0 {start|stop}" exit ;; esac echo ========= azexec $cmd ============= [yinzhengjie@s101 data]$ [yinzhengjie@s101 data]$ sudo chmod a+x /usr/local/bin/azexec.sh [yinzhengjie@s101 data]$
10>.启动服务（启动web和executor）
11>.
原文：https://www.cnblogs.com/yinzhengjie/p/9221131.html
