建站
咨詢
售后
建站咨詢
新聞中心
這篇文章主要介紹“如何在CDH集群啟用Kerberos”，在日常操作中，相信很多人在如何在CDH集群啟用Kerberos問題上存在疑惑，小編查閱了各式資料，整理出簡單好用的操作方法，希望對大家解答”如何在CDH集群啟用Kerberos”的疑惑有所幫助！接下來，請跟著小編一起來學(xué)習(xí)吧！
1.文檔編寫目的
本文檔講述如何在CDH集群啟用及配置Kerberos，您將學(xué)習(xí)到以下知識：
1.如何安裝及配置KDC服務(wù)
2.如何通過CDH啟用Kerberos
3.如何登錄Kerberos并訪問Hadoop相關(guān)服務(wù)
文檔主要分為以下幾步：
1.安裝及配置KDC服務(wù)
2.CDH集群啟用Kerberos
3.Kerberos使用
這篇文檔將重點(diǎn)介紹如何在CDH集群啟用及配置Kerberos，并基于以下假設(shè)：
1.CDH集群運(yùn)行正常
2.集群未啟用Kerberos
3.MySQL 5.1.73
以下是本次測試環(huán)境，但不是本操作手冊的必需環(huán)境：
1.操作系統(tǒng)：CentOS 6.5
2.CDH和CM版本為5.12.0
3.采用root用戶進(jìn)行操作
2.KDC服務(wù)安裝及配置
本文檔中將KDC服務(wù)安裝在Cloudera Manager Server所在服務(wù)器上（KDC服務(wù)可根據(jù)自己需要安裝在其他服務(wù)器）
1.在Cloudera Manager服務(wù)器上安裝KDC服務(wù)
[root@ip-172-31-6-148~]# yum -y install krb5-serverkrb5-libs krb5-auth-dialog krb5-workstation
2.修改/etc/krb5.conf配置
[root@ip-172-31-6-148 fayson_r]# vim /etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = FAYSON.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
[realms]
FAYSON.COM = {
kdc = ip-172-31-6-148.fayson.com
admin_server = ip-172-31-6-148.fayson.com
}
[domain_realm]
.ip-172-31-6-148.fayson.com = FAYSON.COM
ip-172-31-6-148.fayson.com = FAYSON.COM標(biāo)紅部分為需要修改的信息。
3.修改/var/kerberos/krb5kdc/kadm5.acl配置
[root@ip-172-31-6-148~]# vim /var/kerberos/krb5kdc/kadm5.acl */[email protected] *
4.修改/var/kerberos/krb5kdc/kdc.conf配置
[root@ip-172-31-6-148 ~]# vim /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88
[realms]
FAYSON.COM= {
#master_key_type = aes256-cts
max_renewable_life= 7d 0h 0m 0s
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:n
ormal des-cbc-md5:normal des-cbc-crc:normal
}標(biāo)紅部分為需要修改的配置。
5.創(chuàng)建Kerberos數(shù)據(jù)庫
[root@ip-172-31-6-148 ~]# kdb5_util create –r FAYSON.COM -s Loading random data Initializing database '/var/kerberos/krb5kdc/principal' for realm 'FAYSON.COM', master key name 'K/[email protected]' You will be prompted for the database Master Password. It is important that you NOT FORGET this password. Enter KDC database master key: Re-enter KDC database master key to verify:
此處需要輸入Kerberos數(shù)據(jù)庫的密碼。
6.創(chuàng)建Kerberos的管理賬號
[root@ip-172-31-6-148 ~]# kadmin.local Authenticating as principal fayson/[email protected] with password. kadmin.local: addprinc admin/[email protected] WARNING: no policy specified for admin/[email protected]; defaulting to no policy Enter password for principal "admin/[email protected]": Re-enter password for principal "admin/[email protected]": Principal "admin/[email protected]" created. kadmin.local: exit [root@ip-172-31-6-148 ~]#
標(biāo)紅部分為Kerberos管理員賬號，需要輸入管理員密碼。
7.將Kerberos服務(wù)添加到自啟動(dòng)服務(wù)，并啟動(dòng)krb5kdc和kadmin服務(wù)
[root@ip-172-31-6-148~]# chkconfig krb5kdc on [root@ip-172-31-6-148 ~]# chkconfig kadmin on [root@ip-172-31-6-148 ~]# service krb5kdc start Starting Kerberos 5 KDC: [ OK ] [root@ip-172-31-6-148 ~]# service kadmin start Starting Kerberos 5 Admin Server: [ OK ] [root@ip-172-31-6-148 ~]#
8.測試Kerberos的管理員賬號
[root@ip-172-31-6-148 ~]# kinit admin/[email protected] Password for admin/[email protected]: [root@ip-172-31-6-148 ~]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: admin/[email protected] Valid starting Expires Service principal 09/05/17 16:39:17 09/06/17 16:39:17 krbtgt/[email protected] renew until 09/12/17 16:39:17 [root@ip-172-31-6-148 ~]#
9.為集群安裝所有Kerberos客戶端，包括Cloudera Manager
[root@ip-172-31-6-148 cdh-shell-master]# yum -y install krb5-libs krb5-workstation
10.在Cloudera Manager Server服務(wù)器上安裝額外的包
[root@ip-172-31-6-148cdh-shell-master]# yum -y install openldap-clients
11.將KDC Server上的krb5.conf文件拷貝到所有Kerberos客戶端
[root@ip-172-31-6-148cdh-shell-master]# scp -r /etc/krb5.conf [email protected]:/etc/
此處使用腳本進(jìn)行拷貝
[root@ip-172-31-6-148cdh-shell-master]# sh b.sh node.list /etc/krb5.conf /etc/ krb5.conf 100% 451 0.4KB/s 00:00 krb5.conf 100% 451 0.4KB/s 00:00 krb5.conf 100% 451 0.4KB/s 00:00 krb5.conf 100% 451 0.4KB/s 00:00 [root@ip-172-31-6-148 cdh-shell-master]#
3.CDH集群啟用Kerberos
1.在KDC中給Cloudera Manager添加管理員賬號
[root@ip-172-31-6-148 cdh-shell-bak]# kadmin.local Authenticating as principal admin/[email protected] with password. kadmin.local: addprinc cloudera-scm/[email protected] WARNING: no policy specified for cloudera-scm/[email protected]; defaulting to no policy Enter password for principal "cloudera-scm/[email protected]": Re-enter password for principal "cloudera-scm/[email protected]": Principal "cloudera-scm/[email protected]" created. kadmin.local: exit [root@ip-172-31-6-148 cdh-shell-bak]#
2.進(jìn)入Cloudera Manager的“管理”-> “安全”界面
3.選擇“啟用Kerberos”，進(jìn)入如下界面
確保如下列出的所有檢查項(xiàng)都已完成
4.點(diǎn)擊“繼續(xù)”，配置相關(guān)的KDC信息，包括類型、KDC服務(wù)器、KDC Realm、加密類型以及待創(chuàng)建的Service Principal（hdfs，yarn,，hbase，hive等）的更新生命期等
5.點(diǎn)擊“繼續(xù)”
6.不建議讓Cloudera Manager來管理krb5.conf, 點(diǎn)擊“繼續(xù)”
7.輸入Cloudera Manager的Kerbers管理員賬號，必須和之前創(chuàng)建的賬號一致，點(diǎn)擊“繼續(xù)”
8.等待啟用Kerberos完成，點(diǎn)擊“繼續(xù)”
9.點(diǎn)擊“繼續(xù)”
10.勾選重啟集群，點(diǎn)擊“繼續(xù)”
11.等待集群重啟成功，點(diǎn)擊“繼續(xù)”
至此已成功啟用Kerberos。
4.Kerberos使用
使用fayson用戶運(yùn)行MapReduce任務(wù)及操作Hive，需要在集群所有節(jié)點(diǎn)創(chuàng)建fayson用戶。
1.使用kadmin創(chuàng)建一個(gè)fayson的principal
[root@ip-172-31-6-148 cdh-shell-bak]# kadmin.local Authenticating as principal admin/[email protected] with password. kadmin.local: addprinc [email protected] WARNING: no policy specified for [email protected]; defaulting to no policy Enter password for principal "[email protected]": Re-enter password for principal "[email protected]": Principal "[email protected]" created. kadmin.local: exit [root@ip-172-31-6-148 cdh-shell-bak]#
2.使用fayson用戶登錄Kerberos
[root@ip-172-31-6-148 cdh-shell-bak]# kdestroy [root@ip-172-31-6-148 cdh-shell-bak]# kinit fayson Password for [email protected]: [root@ip-172-31-6-148 cdh-shell-bak]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: [email protected] Valid starting Expires Service principal 09/05/17 17:19:08 09/06/17 17:19:08 krbtgt/[email protected] renew until 09/12/17 17:19:08 [root@ip-172-31-6-148 cdh-shell-bak]#
3.運(yùn)行MapReduce作業(yè)
[root@ip-172-31-6-148~]# hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 1 ... Starting Job 17/09/02 20:10:43 INFO mapreduce.Job: Running job: job_1504383005209_0001 17/09/02 20:10:56 INFO mapreduce.Job: Job job_1504383005209_0001 running in ubermode : false 17/09/02 20:10:56 INFO mapreduce.Job: map0% reduce 0% 17/09/02 20:11:09 INFO mapreduce.Job: map20% reduce 0% 17/09/02 20:11:12 INFO mapreduce.Job: map40% reduce 0% 17/09/02 20:11:13 INFO mapreduce.Job: map50% reduce 0% 17/09/02 20:11:15 INFO mapreduce.Job: map60% reduce 0% 17/09/02 20:11:16 INFO mapreduce.Job: map70% reduce 0% 17/09/02 20:11:19 INFO mapreduce.Job: map80% reduce 0% 17/09/02 20:11:21 INFO mapreduce.Job: map100% reduce 0% 17/09/02 20:11:26 INFO mapreduce.Job: map100% reduce 100% 17/09/02 20:11:26 INFO mapreduce.Job: Job job_1504383005209_0001 completedsuccessfully
4.使用beeline連接hive進(jìn)行測試
[root@ip-172-31-6-148 cdh-shell-bak]# beeline Beeline version 1.1.0-cdh6.12.1 by Apache Hive beeline> !connect jdbc:hive2://localhost:10000/;principal=hive/[email protected] ... Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://localhost:10000/> show tables; ... INFO : OK +-------------+--+ | tab_name | +-------------+--+ | test_table | +-------------+--+ 1 row selected (0.194 seconds) 0: jdbc:hive2://localhost:10000/> select * from test_table; ... INFO : OK +----------------+----------------+--+ | test_table.s1 | test_table.s2 | +----------------+----------------+--+ | 4 | lisi | | 1 | test | | 2 | fayson | | 3 | zhangsan | +----------------+----------------+--+ 4 rows selected (0.144 seconds) 0: jdbc:hive2://localhost:10000/>
運(yùn)行Hive MapReduce作業(yè)
0: jdbc:hive2://localhost:10000/> select count(*) from test_table; ... INFO : OK +------+--+ | _c0 | +------+--+ | 4 | +------+--+ 1 row selected (35.779 seconds) 0: jdbc:hive2://localhost:10000/>
5.常見問題
1.使用Kerberos用戶身份運(yùn)行MapReduce作業(yè)報(bào)錯(cuò)
main : run as user is fayson main : requested yarn user is fayson Requested user fayson is not whitelisted and has id 501,whichis below the minimum allowed 1000 Failing this attempt. Failing the application. 17/09/02 20:05:04 INFO mapreduce.Job: Counters: 0 Job Finished in 6.184 seconds java.io.FileNotFoundException: File does not exist:hdfs://ip-172-31-6-148:8020/user/fayson/QuasiMonteCarlo_1504382696029_1308422444/out/reduce-out at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266) at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1258) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1258) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1820) at org.apache.hadoop.io.SequenceFile$Reader. (SequenceFile.java:1844) at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314) at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363) at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) atorg.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
問題原因：是由于Yarn限制了用戶id小于10000的用戶提交作業(yè)；
解決方法：修改Yarn的min.user.id來解決
到此，關(guān)于“如何在CDH集群啟用Kerberos”的學(xué)習(xí)就結(jié)束了，希望能夠解決大家的疑惑。理論與實(shí)踐的搭配能更好的幫助大家學(xué)習(xí)，快去試試吧！若想繼續(xù)學(xué)習(xí)更多相關(guān)知識，請繼續(xù)關(guān)注創(chuàng)新互聯(lián)網(wǎng)站，小編會(huì)繼續(xù)努力為大家?guī)砀鄬?shí)用的文章！
另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)scvps.cn，海內(nèi)外云服務(wù)器15元起步，三天無理由+7*72小時(shí)售后在線，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡單易用、服務(wù)可用性高、性價(jià)比高”等特點(diǎn)與優(yōu)勢，專為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應(yīng)用場景需求。
網(wǎng)頁名稱：如何在CDH集群啟用Kerberos-創(chuàng)新互聯(lián)
分享網(wǎng)址：http://fisionsoft.com.cn/article/djegch.html
