未加星标

Cloudera Altus Director SDX Integration

字体大小 | |
[数据库(综合) 所属分类 数据库(综合) | 发布者 店小二04 | 时间 2018 | 作者 红领巾 ] 0人收藏点击收藏

Cloudera now provides a pathway for migrating an Altus Director managed cluster to a cluster that takes advantage of other Cloudera Altus cloud services. This blog post will outline how to do that.

SDX for Cloudera Altus persists both Apache Hive metadata and Apache Sentry data access policies independently from clusters in SDX namespaces. In this way, SDX for Cloudera Altus provides the missing link to share Altus Director managed workload with Altus cloud services. Separating metadata from the compute resources enables transient workloads and users never have to worry about losing data context, even when a cluster is terminated. Whenever new clusters are created and attached to the same SDX namespace, the existing table metadata and access policies apply to the new cluster right away.

The followingexample demonstrates how to share metadata between a cluster created by Altus Director with an Altus Data Warehouse cluster.


Cloudera Altus Director SDX Integration

Before getting started:

Ensure that an Altus Director instance is running. Work with an Altus admin to create a user and allow that user to create and manage the following resources SDX namespaces Altus environments Data Warehouse clusters Altus Director: Creating a Cluster with an External Database

SDX configured namespaces, helpful for migrating from Altus Director to other Altus cloud services, require Apache Hive and Apache Sentry to be set upto usean external database server. Altus Director provides the ability to create a CDH deployment and attach an external database to the cluster. This database can be either mysql or PostgreSQL. Altus Director also has the ability to create an RDS database in AWS on behalf of the user to be utilized as an external database. An external database in Altus Director operates the same as an Altus SDX namespace.

The Altus Director team provides sample configurations to quickly get started. Here is a sample SDX configuration. This configuration includes Hive and Impala, but not Spark. If the use case in question requiresother services, there are several reference configurations in this same git repository.

In order to take advantage of RDS, users need to modify the above configuration to create an RDS server instead of using an existing database. This configuration provides an example of specifying RDS database servers.

The following example is asnippet which should be included in the Altus Director configuration that shows the steps to combine Sentry with anRDS database:

# define a database name for reuse later rds { name: "jheyming-mysql1" } databaseServers { # the name of the RDS database that will be created in AWS by Director jheyming-mysql1 { type: mysql user: root password: <redacted> instanceClass: db.m3.medium dbSubnetGroupName: my-subnet-group vpcSecurityGroupIds: "sg-12345678" allocatedStorage: 10 engineVersion: 5.5.53 tags { owner: ${?USER} } } } # cluster configuration cluster { databaseTemplates: { HIVE { name: hivetemplate databaseServerName: ${rds.name} databaseNamePrefix: hive usernamePrefix: hiveu } # ... repeat for HUE, OOZIE, SENTRY # Sentry admin groups # important for later when trying to administer databases in the Hue query editor configs { SENTRY { sentry_service_admin_group: "hive,impala,hue,solr,svc_admin" sentry_service_allow_connect: "hive,impala,hue,hdfs,solr,svc_admin" } } } # cloudera manager can use the database too cloudera-manager { databaseTemplates { CLOUDERA_MANAGER { name: cmtemplate databaseServerName: ${rds.name} databaseNamePrefix: cm usernamePrefix: cmu } # repeat for ACTIVITYMONITOR, REPORTSMANAGER, NAVIGATOR, NAVIGATORMETASERVER… }

Once the configuration is ready, use the Altus Director bootstrap-remote CLI to execute it:

$ cloudera-director bootstrap-remote \ /Users/jheyming/sdx-with-rds.conf \ --lp.remote.hostAndPort=localhost:7189 \ --lp.remote.username=admin \ --lp.remote.password=admin

The user who runs this command must have the ability to create EC2 instances as well as create RDS databases in AWS. This usermust alsohave access to the VPC subnets and security groups in this configuration.Learn more here.

The progress of the bootstrap command can be viewed in the Altus Director UI:


Cloudera Altus Director SDX Integration
Cloudera Manager: Working with Hue

Now that Cloudera Manager is up and running Sentry permissions must be configured via Impala SQL using Hue. To do that, add a user as a Sentry admin and create a parallel Hue user (in this case I’ll create a user jheyming which we’ll use as the administrator). Navigate to Cloudera Manager via Altus Director:


Cloudera Altus Director SDX Integration

Log into the Cloudera Manager server and navigate to the Hue service. There, find out the IP address for the Hue server.

Making note of the IP address, go to that address at port 8888. Cloudera Manager provides quick links to navigate to the host. Before logging into Hue, create an admin user in order to create some tables.

One way to set up this user for Hue is to log into each host in Cloudera Manager and run the useradd command. Hue with Sentry can authorize these users to have access to the Hue interface. This user also needs to be a Sentryadmin. Look back to the Director configuration; there were Sentry admin groups defined there. In that configuration, there was an admin group defined for Impala. Use this same group and assign this user to the group.

To create a user on each host quickly, the following command can be used to batch SSH log into each host in the Cloudera Manager. Use the IP addresses of each host in Cloudera Manager and then use SSH to create a user. Here is an example of doing this in bulk:

for host in 10.38.0.250 10.38.1.128 10.38.3.107 10.38.3.141 10.38.4.231 10.38.5.181 10.38.7.181 10.38.7.206 do; ssh -t -o UserKnownHostsFile=/dev/null \ -o StrictHostKeyChecking=no \ -i /Users/jheyming/.ssh/my.pem \ centos@$host \ "sudo useradd -g 479 jheyming --password='encrypted'"; done; # @see https://serverfault.com/questions/367559/how-to-add-a-user-without-knowing-the-encrypted-form-of-the-password # 479 was found to be the group id for impala by inspecting /etc/group. # I chose to use the 'impala' group because this group was allocated # administrative rights for the Sentry service in our Altus Director # conf file. If you don't want to use the built-in impala group, you # can recreate your cluster with your own custom admin group.

Then on the Hue server, log in as admin user with admin as the password. Then create a user with the same name (for example, jheyming ) using the User Management page.


Cloudera Altus Director SDX Integration

Log in as that user and run the following queries:

create role admin; grant role admin to group `impala`; grant all on server server1 to role admin; create database jheyming_director; Group impala was added as an admin Sentry group in the Altus Direct

本文数据库(综合)相关术语:系统安全软件

代码区博客精选文章
分页:12
转载请注明
本文标题:Cloudera Altus Director SDX Integration
本站链接:https://www.codesec.net/view/621283.html


1.凡CodeSecTeam转载的文章,均出自其它媒体或其他官网介绍,目的在于传递更多的信息,并不代表本站赞同其观点和其真实性负责;
2.转载的文章仅代表原创作者观点,与本站无关。其原创性以及文中陈述文字和内容未经本站证实,本站对该文以及其中全部或者部分内容、文字的真实性、完整性、及时性,不作出任何保证或承若;
3.如本站转载稿涉及版权等问题,请作者及时联系本站,我们会及时处理。
登录后可拥有收藏文章、关注作者等权限...
技术大类 技术大类 | 数据库(综合) | 评论(0) | 阅读(137)