hbase加kerberos 后报错hbase master 起不来

背景

公司测试环境，好多人用，各种瞎改，以前的问题也不好改，我就没管，把kerberos装上后，hbase master起不来了

问题

habse master重启后进入standby，一会自己挂掉，查看日志文件如下

ERROR [main] regionserver.HRegionServer: Failed construction RegionServer
org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hbase/master.am.com@AM.COM from keytab /etc/security/keytabs/hbase.service.keytab javax.security.auth.login.LoginException: Unable to obtain password from user

        at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1215)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1008)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:313)
        at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:344)
        at org.apache.hadoop.hbase.security.User.login(User.java:231)
        at org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:198)
        at org.apache.hadoop.hbase.master.HMaster.login(HMaster.java:663)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:582)
        at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:478)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3093)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3111)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
        at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
        ... 19 more
2022-09-14 16:23:18,536 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3100)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
......

解决

看日志能看出来，是kerberos认证的问题，让他自动重建一份还是不行，就手动重建

sudo kadmin -padmin/admin -wadmin -q"xst -k /etc/security/keytabs/hbase.service.keytab  hbase/master.am.com@AM.COM"

依然出错

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-sYENbeUs-1664610753369)(https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/bea783c8d8e644efb4a0bea7559fbcab~tplv-k3u1fbpfcp-watermark.image?)]

判断是权限问题
修改keytab文件权限

chown -R hbase:hadoop /etc/security/keytabs/hbase.service.keytab

重启服务问题解决

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-JgUL62sm-1664610753370)(https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/3b4dda3ceb0140fa9329bde8d1ae2f0a~tplv-k3u1fbpfcp-watermark.image?)]

总结

1.需要手动重新创建keytab文件
2.注意keytab文件权限问题
3.解决问题命令如下：

sudo kadmin -padmin/admin -wadmin -q"xst -k /etc/security/keytabs/hbase.service.keytab  hbase/master.am.com@AM.COM"
chown -R hbase:hadoop /etc/security/keytabs/hbase.service.keytab