当前位置: 代码迷 >> 综合 >> 黑猴子的家:Centos7 安装部署 Azkaban 3.84.4
  详细解决方案

黑猴子的家:Centos7 安装部署 Azkaban 3.84.4

热度:102   发布时间:2023-09-14 21:51:04.0

1、准备好安装包

[alex@hadoop102 azkaban_tar.gz]$ pwd
/opt/software/azkaban_tar.gz
[alex@hadoop102 azkaban_tar.gz]$ ls -l
-rw-rw-r-- 1 alex alex     3876 May 20 13:32 azkaban-db-3.84.4.tar.gz
-rw-rw-r-- 1 alex alex 15766958 May 20 13:32 azkaban-exec-server-3.84.4.tar.gz
-rw-rw-r-- 1 alex alex 20003240 May 20 13:33 azkaban-web-server-3.84.4.tar.gz

选择Mysql作为Azkaban数据库,因为Azkaban建立了一些Mysql连接增强功能,以方便Azkaban设置,并增强服务可靠性。

2、创建安装目录

[alex@hadoop102 module]$ mkdir azkaban

3、解压安装包

[alex@hadoop102 software]$ tar -xzvf azkaban_tar.gz/azkaban-web-server-3.84.4.tar.gz -C ../module/azkaban
[alex@hadoop102 software]$ tar -xzvf azkaban_tar.gz/azkaban-exec-server-3.84.4.tar.gz -C ../module/Azkaban
[alex@hadoop102 software]$ tar -xzvf azkaban_tar.gz/azkaban-db-3.84.4.tar.gz -C ../module/azkaban

4、修改名称

[alex@hadoop102 azkaban]$ mv azkaban-db-3.84.4 azkaban-db
[alex@hadoop102 azkaban]$ mv azkaban-web-server-3.84.4 azkaban-web-server
[alex@hadoop102 azkaban]$ mv azkaban-exec-server-3.84.4 azkaban-exec-server

5、发送sql文件

[alex@hadoop102 azkaban-db]$ scp -r create-all-sql-3.84.4.sql alex@hadoop103:/opt/software/

6、azkaban脚本导入

进入mysql,创建azkaban数据库,并将解压的脚本导入到azkaban数据库。

[alex@hadoop103 ~]$ vim /etc/my.cnf
[mysqld]
max_allowed_packet=1024M
[alex@hadoop103 ~]$ sudo systemctl restart mysqld
[alex@hadoop103 ~]$ mysql -hhadoop103 -P3306 -uroot -p000000
mysql> drop database if exists azkaban;
mysql> create database azkaban;
mysql> show databases;
mysql> use azkaban;
mysql> source /opt/software/create-all-sql-0.1.0-SNAPSHOT.sql
mysql> show tables;

尖叫提示:这里可以使用navicat 连接一下试一试
source后跟.sql文件,用于批量处理.sql文件中的sql语句

7、生成密钥库

keystore 是java 的一个工具,生成密钥库后,拷贝到azkaban 的web 服务里面
1)生成 keystore的密码及相应信息的密钥库

[alex@hadoop102 azkaban]$ keytool -keystore keystore -alias jetty -genkey -keyalg RSA

注意:
密钥库的密码至少必须6个字符,可以是纯数字或者字母或者数字和字母的组合等等
密钥库的密码最好和<jetty> 的密钥相同,方便记忆

2)将keystore 拷贝到 azkaban web服务器根目录中

[alex@hadoop102 azkaban]$ mv keystore azkaban-web-server/

Keytool是java数据证书的管理工具,使用户能够管理自己的公/私钥对及相关证书。
keystore 指定密钥库的名称及位置(产生的各类信息将不在.keystore文件中)
-genkey 在用户主目录中创建一个默认文件".keystore"
-alias 对我们生成的.keystore 进行指认别名;如果没有默认是mykey
-keyalg 指定密钥的算法 RSA/DSA 默认是DSA

8、时间同步配置

先配置好服务器节点上的时区
1)tzselect
如果在/usr/share/zoneinfo/这个目录下不存在时区配置文件Asia/Shanghai,就要用 tzselect 生成。

[alex@hadoop102 azkaban]$ date -R
Thu, 20 May 2021 15:25:30 +0800[alex@hadoop102 azkaban]$ tzselect
Please identify a location so that time zone rules can be set correctly.
Please select a continent or ocean.1) Africa2) Americas3) Antarctica4) Arctic Ocean5) Asia6) Atlantic Ocean7) Australia8) Europe9) Indian Ocean
10) Pacific Ocean
11) none - I want to specify the time zone using the Posix TZ format.
#? 5
Please select a country.1) Afghanistan           18) Israel                35) Palestine2) Armenia               19) Japan                 36) Philippines3) Azerbaijan            20) Jordan                37) Qatar4) Bahrain               21) Kazakhstan            38) Russia5) Bangladesh            22) Korea (North)         39) Saudi Arabia6) Bhutan                23) Korea (South)         40) Singapore7) Brunei                24) Kuwait                41) Sri Lanka8) Cambodia              25) Kyrgyzstan            42) Syria9) China                 26) Laos                  43) Taiwan
10) Cyprus                27) Lebanon               44) Tajikistan
11) East Timor            28) Macau                 45) Thailand
12) Georgia               29) Malaysia              46) Turkmenistan
13) Hong Kong             30) Mongolia              47) United Arab Emirates
14) India                 31) Myanmar (Burma)       48) Uzbekistan
15) Indonesia             32) Nepal                 49) Vietnam
16) Iran                  33) Oman                  50) Yemen
17) Iraq                  34) Pakistan
#? 9
Please select one of the following time zone regions.
1) Beijing Time
2) Xinjiang Time
#? 1The following information has been given:ChinaBeijing TimeTherefore TZ='Asia/Shanghai' will be used.
Local time is now:      Thu Oct 18 16:24:23 CST 2018.
Universal Time is now:  Thu Oct 18 08:24:23 UTC 2018.
Is the above information OK?
1) Yes
2) No
#? 1You can make this change permanent for yourself by appending the lineTZ='Asia/Shanghai'; export TZ
to the file '.profile' in your home directory; then log out and log in again.Here is that TZ value again, this time on standard output so that you
can use the /usr/bin/tzselect command in shell scripts:
Asia/Shanghai

2)拷贝该时区文件,覆盖系统本地时区配置

[alex@hadoop102 azkaban]$ cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

3)集群时间同步(同时发给三个窗口)

[alex@hadoop102 azkaban]$ sudo date -s '2021-05-20 16:39:30'

9、azkaban-web-server 修改配置文件

1)azkaban.properties

[alex@hadoop102 ~]$ cd /opt/module/azkaban/azkaban-web-server/conf/
[alex@hadoop102 conf]$ pwd
/opt/module/azkaban/azkaban-web-server/conf[alex@hadoop102 conf]$ ll
-rw-rw-r-- 1 alex alex 1946 Sep 25  2018 azkaban.properties
-rw-rw-r-- 1 alex alex  267 Sep 25  2018 azkaban-users.xml
-rw-rw-r-- 1 alex alex    0 Sep 25  2018 global.properties
-rw-rw-r-- 1 alex alex  644 Sep 25  2018 log4j.properties[alex@hadoop102 conf]$ vim azkaban.propertiesdefault.timezone.id=Asia/Shanghaidatabase.type=mysql
mysql.port=3306
mysql.host=hadoop103
mysql.database=azkaban
mysql.user=root
mysql.password=000000
mysql.numconnections=100azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus

2)azkaban-users.xml

[alex@hadoop102 conf]$ pwd
/opt/module/azkaban/azkaban-web-server/conf[alex@hadoop102 conf]$ vim azkaban-users.xml
<azkaban-users><user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/><user password="metrics" roles="metrics" username="metrics"/><user username="admin" password="admin" roles="admin,metrics" /><role name="admin" permissions="ADMIN"/><role name="metrics" permissions="METRICS"/>
</azkaban-users>

尖叫提示:增加管理员用户

3)log4j.properties

[alex@hadoop102 conf]$ pwd
/opt/module/azkaban/azkaban-web-server/conf[alex@hadoop102 conf]$ vim log4j.properties
log4j.appender.server.File=/opt/module/azkaban/azkaban-web-server/logs/azkaban-webserver.log

10、azkaban-exec-server修改配置文件

1)azkaban.properties

[alex@hadoop102 conf]$ pwd
/opt/module/azkaban/azkaban-exec-server/conf[alex@hadoop102 conf]$ vim azkaban.propertiesdefault.timezone.id=Asia/Shanghaiazkaban.webserver.url=http://hadoop102:8081database.type=mysql
mysql.port=3306
mysql.host=hadoop103
mysql.database=azkaban
mysql.user=root
mysql.password=000000
mysql.numconnections=100executor.port=12321
executor.metric.reports=true
executor.metric.milisecinterval.default=60000

2)log4j.properties

[alex@hadoop102 conf]$ vim log4j.properties
log4j.appender.server.File=/opt/module/azkaban/azkaban-exec-server/logs/azkaban-execserver.log

3)commonprivate.properties

[alex@hadoop102 jobtypes]$ pwd
/opt/module/azkaban/azkaban-exec-server/plugins/jobtypes
[alex@hadoop102 jobtypes]$ vim commonprivate.properties 
azkaban.native.lib=false
memCheck.enabled=false

若不添加这个参数,在执行Job时会报错

azkaban.utils.UndefinedPropertyException: Missing required property 'azkaban.native.lib'at azkaban.utils.Props.getString(Props.java:450)at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:242)at azkaban.execapp.JobRunner.runJob(JobRunner.java:823)at azkaban.execapp.JobRunner.doRun(JobRunner.java:602)at azkaban.execapp.JobRunner.run(JobRunner.java:563)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)

11、分发

[alex@hadoop102 azkabn]$ xsync.sh azkaban-exec-server/

12、启动Azkaban

先启动azkaban-exec-serve,再启动azkaban-web-server
因为要避免Web Server因为找不到执行器而启动失败
1)启动azkaban-exec-serve

[alex@hadoop102 azkaban-exec-server]$ bin/start-exec.sh
[alex@hadoop103 azkaban-exec-server]$ bin/start-exec.sh
[alex@hadoop104 azkaban-exec-server]$ bin/start-exec.sh

尖叫提示:如果在/opt/module/azkaban/azkaban-exec-serve目录下出现executor.port文件,说明启动成功

2)需要激活executor

[alex@hadoop102 azkaban-exec-server]$ curl -G "hadoop102:$(<./executor.port)/executor?action=activate" && echo
{"status":"success"}[alex@hadoop103 azkaban-exec-server]$ curl -G "hadoop103:$(<./executor.port)/executor?action=activate" && echo
{"status":"success"}[alex@hadoop104 azkaban-exec-server]$ curl -G "hadoop104:$(<./executor.port)/executor?action=activate" && echo
{"status":"success"}

尖叫提示:每次重新启动azkaban-exec-server的时候,都需要激活一下

3)修改executors表 active字段的状态

mysql> use azkaban;
mysql> update executors set active=1;
Query OK, 3 rows affected (0.00 sec)
Rows matched: 3  Changed: 3  Warnings: 0mysql> select * from executors;
+----+-----------+-------+--------+
| id | host      | port  | active |
+----+-----------+-------+--------+
|  1 | hadoop102 | 42680 |      1 |
|  4 | hadoop103 | 43817 |      1 |
|  5 | hadoop104 | 42601 |      1 |
+----+-----------+-------+--------+
3 rows in set (0.00 sec)

尖叫提示:azkaban 不能使用shutdown-exec.sh 和 shutdown-web.sh 来关闭服务,因为关闭服务的时候,会修改active状态为0,所以推荐使用kill -9 pid来关闭服务

4)启动azkaban-web-server服务

[alex@hadoop102 azkaban-web-server]$ bin/start-web.sh

5)浏览器访问

https://hadoop102:8443/
http://hadoop102:8081/
黑猴子的家:Centos7 安装部署 Azkaban 3.84.4

尖叫提示:用户名和密码都是admin
6)登陆成功

黑猴子的家:Centos7 安装部署 Azkaban 3.84.4

7)控制台查看

[alex@hadoop102 azkaban-web-server]$ jps
22114 Jps
15396 AzkabanExecutorServer
18276 AzkabanWebServer