未加星标

Securing Hadoop Web Interfaces with Kerberos

字体大小 | |
[数据库(综合) 所属分类 数据库(综合) | 发布者 店小二05 | 时间 2016 | 作者 红领巾 ] 0人收藏点击收藏

By default HDFS, Yarn, MapReduce Hadoop web interfaces are not “Kerberized”. In IOP, access to Hadoop web interfaces is through Apache Knox. Apache Knox is a very powerful proxy optimized for Hadoop components access. Without Knox Apache we would need to access Hadoop web interfaces directly without any authentication protection. To turn such authentication protection end-to-end it is necessary to enable Kerberos in the Hadoop Web interfaces.

This means to enable SPNEGO protocol since Web and REST access means HTTP access. Knox provides an authenticated access to the Hadoop Web Interfaces using either PAM or LDAP through Basic Authentication, then in turn Knox proxy accesses the Hadoop Web interfaces using SPNEGO.

Following are the steps and some discussion on enabling Hadoop Web interfaces with Kerberos. Do this after you have enable your Kerberos cluster

1. Create a “secret” file to be used by “Hadoop SPNEGO” filter.

dd if=/dev/urandom of=/etc/security/your_secret_data_file bs=1024 count=1
chown hdfs:hadoop /etc/security/your_secret_data_file
chmod 440 /etc/security/your_secret_data_file

Your file your_secret_data_file, is only needed on the namenode servers. This is the file used by the Hadoop HTTP SPNEGO, see reference at “Hadoop Auth,Java HTTP SPNEGO” from Hadoop documentation, to sign the HTTP cookie used for the SPNEGO protocol.

2. Change HDFS configuration

The following entry gives proxy knox user access to Hadoop servlets secure paths. You will need to change your hdfs-site (through Ambari advanced hdfs-site).

dfs.cluster.administrators hdfs,knox

3. The following entries setup Hadoop SPNEGO filter.

You need to add these settings to the core-site file (using Ambari Custom hdfs core-site)

hadoop.http.authentication.simple.anonymous.allowed false
hadoop.http.authentication.signature.secret.file /etc/security/your_secret_data_file
hadoop.http.authentication.type kerberos
hadoop.http.authentication.kerberos.keytab /etc/security/keytabs/spnego.service.keytab
hadoop.http.authentication.kerberos.principal [email protected]_REALM
hadoop.http.filter.initializers org.apache.hadoop.security.AuthenticationFilterInitializer
hadoop.http.authentication.cookie.domain YOUR_REALM

YOUR_REALM e.g. MYSAMPLE.COM

4. Change Yarn configurations in either of the following options.

The following entry gives user knox access to YARN REST/UI API when Hadoop web interface is secure. ( this versus dr.who Hadoop static default user when not authenticated).

To make the following changes you need to modify the Yarn Resource Manager settings.


yarn.admin.acl yarn,knox

Another alternative is to use (done in the Ambari Yarn custom yarn-site settings).:

yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled true

The second alternative relies on the RMAuthenticationFilter class which extends the Hadoop security class DelegationTokenAuthenticationFilter. This accomplishes a couple of things: First it allows the Yarn Console to use Delegation Token authentication (vs Kerberos SPNEGO), and second it permits the "user" name doing the request to show on the "Logged in as" and not the proxy super user "knox". Using RMAuthenticationFilter maybe a more secure option since yarn.admin.acl may grant super user knox or group more access than what it is desired.

5. Restart from Ambari admin server all servers involved: HDFS, MapReduce, Yarn, KNOX.

Comments

One thing to notice is that if the Hadoop console uses Kerberos authentication (not Delegation token) thus it shows as user "knox" which is the proxy "super user" since to Hadoop web code it shows the user making the request (request remote user), which in the case of Knox is the proxy user "knox".

An alternative for the case of Yarn is to use a Delegation Token filter which means we are not using SPNEGO protocol authentication but Delegation token Authentication in a Kerberos channel (It is assumed that we are running ona Kerberized installation). In this alternative case the requesting user is setthe request wrapper of the DelegationTokenAuthenticationFilter (parent class of RMAuthenticationFilter) which uses the logged user short name (the user shortname of the Kerberos principal).

It would be argued that the current solution of using "dfs.cluster.administrators: hdfs,knox,hadoop" is not as secure since this setup ACL controls who can access the default servlets in HDFS UI. In addition, since user "knox" being the proxy super user would have access to the web resources like "/logs" any user authenticated through Knox would have access to these logs. We should explore with the HDFS community whether it makes sense to provide such a setting to provide Delegation Token on the Namenodes web UI and/or to give finer access authorization to the logs from the HDFS UI.

We re-opened a Hadoop Jira 13119, that is related to our solution above on using "dfs.cluster.administrators" to gain access to "/logs" servlet. This web path is not secure since it is not included on the secure section of the HDFS web descriptor thus is doesn't take advantage of the web authentication on HDFS web application but it relies on dfs.cluster.administrators membership. This would seem an inconsistency on HDFS web application that we would need to reconcile through the open source community.

Finally, another option my team discovered is that both the proxyed user name as well as the access to "/logs" issues can be fixed by using the org.apache.hadoop.security.token.delegation.web.DelegationAuthenticationHandler class which is also an AuthenticationHandler. This class implements Kerberos and supports Delegation Token authentication, thus in section 3. we could set up the "hadoop.http.filter.initializers" to the DelegationTokenAuthenticationFilter which when Kerberos is enable uses the DelegationAuthenticationHandler. In this case the authentication with Kerberos is through Delegation Token using RPC sasl rather than the SPNEGO protocol.

hadoop.http.authentication.simple.anonymous.allowed false
hadoop.http.authentication.signature.secret.file /etc/security/your_secret_data_file
hadoop.http.authentication.type kerberos
hadoop.http.authentication.kerberos.keytab /etc/security/keytabs/spnego.service.keytab
hadoop.http.authentication.kerberos.principal [email protected]_REALM
hadoop.http.filter.initializers org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter
hadoop.http.authentication.cookie.domain YOUR_REALM References

Hadoop Auth, Java HTTP SPNEGO - Server Side Configuration

Apache Knox REST API Gateway for the Apache Hadoop Ecosystem

本文数据库(综合)相关术语:系统安全软件

主题: HadoopHDFSRESTJavaMapReduceRPC
分页:12
转载请注明
本文标题:Securing Hadoop Web Interfaces with Kerberos
本站链接:http://www.codesec.net/view/480932.html
分享请点击:


1.凡CodeSecTeam转载的文章,均出自其它媒体或其他官网介绍,目的在于传递更多的信息,并不代表本站赞同其观点和其真实性负责;
2.转载的文章仅代表原创作者观点,与本站无关。其原创性以及文中陈述文字和内容未经本站证实,本站对该文以及其中全部或者部分内容、文字的真实性、完整性、及时性,不作出任何保证或承若;
3.如本站转载稿涉及版权等问题,请作者及时联系本站,我们会及时处理。
登录后可拥有收藏文章、关注作者等权限...
技术大类 技术大类 | 数据库(综合) | 评论(0) | 阅读(47)