网站营销主管岗位职责江苏建站
目录
一、Ranger简介
1、组件列表
2、支持的数据引擎服务
二、主机环境准备
1、关闭防火墙
2、关闭SELINUX
3、安装docker
4、下载Ranger源码包
5、下载Maven安装包
三、编译Ranger源码
1、修改官方包中的build_ranger_using_docker.sh
2、运行脚本编译
3、编译检查
一、Ranger简介
Apache Ranger提供一个集中式安全管理框架, 并解决授权和审计。它可以对Hadoop生态的组件如HDFS、Yarn、Hive、Hbase等进行细粒度的数据访问控制。通过操作Ranger控制台,管理员可以轻松的通过配置策略来控制用户访问权限。
1、组件列表
| # | Service Name | Listen Port | Core Ranger Service |
|---|---|---|---|
| 1 | ranger | 6080/tcp | Y (ranger engine - 3.0.0-SNAPSHOT version) |
| 2 | ranger-postgres | 5432/tcp | Y (ranger datastore) |
| 3 | ranger-solr | 8983/tcp | Y (audit store) |
| 4 | ranger-zk | 2181/tcp | Y (used by solr) |
| 5 | ranger-usersync | - | Y (user/group synchronization from Local Linux/Mac) |
| 6 | ranger-kms | 9292/tcp | N (needed only for Encrypted Storage / TDE) |
| 7 | ranger-tagsync | - | N (needed only for Tag Based Policies to be sync from ATLAS) |
2、支持的数据引擎服务
| # | Service Name | Listen Port | Service Description |
|---|---|---|---|
| 1 | Hadoop | 8088/tcp | Apache Hadoop 3.3.0 |
| 2 | HBase | 16000/tcp 16010/tcp 16020/tcp 16030/tcp | Apache HBase 2.4.6 Protected by Apache Ranger's HBase Plugin |
| 3 | Hive | 10000/tcp | Apache Hive 3.1.2 Protected by Apache Ranger's Hive Plugin |
| 4 | Kafka | 6667/tcp | Apache Kafka 2.8.1 Protected by Apache Ranger's Kafka Plugin |
| 5 | Knox | 8443/tcp | Apache Knox 1.4.0 Protected by Apache Ranger's Knox Plugin |
二、主机环境准备
1、关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
2、关闭SELINUX
sed -i.bak$DATE '/^SELINUX=/c SELINUX=disabled' /etc/selinux/config
setenforce 0
3、安装docker
yum install -y docker
systemctl start docker
systemctl enable docker
4、下载Ranger源码包
Apache Ranger官网没有可以直接部署的安装包，必须通过源码进行编译。
官网地址：Apache Ranger - Download Apache Ranger?
wget https://www.apache.org/dist/ranger/2.4.0/apache-ranger-2.4.0.tar.gz --no-check-certificate
5、下载Maven安装包
wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificate
三、编译Ranger源码
1、修改官方包中的build_ranger_using_docker.sh
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License#This script creates the Docker image (if not already created) and runs maven in the container
#1. Install Docker
#2. Checkout Ranger source and go to the root directory
#3. Run this script. If host is linux, then run this script as "sudo $0 ..."
#4. If you are running on Mac, then you don't need to use "sudo"
#5. To delete the image, run "[sudo] docker rmi ranger_dev"#Usage: [sudo] ./build_ranger_using_docker.sh [-build_image] mvn <build params>
#Example 1 (default no param): (mvn -Pall -DskipTests=true clean compile package install)
#Example 2 (Regular build): ./build_ranger_using_docker.sh mvn -Pall clean install -DskipTests=true
#Example 3 (Recreate Docker image): ./build_ranger_using_docker.sh mvn -Pall -build_image clean install -DskipTests=true
#Notes: To remove build image manually, run "docker rmi ranger_dev" or "sudo docker rmi ranger_dev"default_command="mvn -Pall -DskipTests=true clean compile package install"
build_image=0
if [ "$1" = "-build_image" ]; then
build_image=1
shift
fiparams=$*
if [ $# -eq 0 ]; then
params=$default_command
fiimage_name="ranger_dev"
remote_home="$HOME"
container_name="--name ranger_build"if [ ! -d security-admin ]; then
echo "ERROR: Run the script from root folder of source. e.g. $HOME/git/ranger"
exit 1
fiimages=`docker images | cut -f 1 -d " "`
[[ $images =~ $image_name ]] && found_image=1 || build_image=1if [ $build_image -eq 1 ]; then
echo "Creating image $image_name ..."
docker rmi -f $image_namedocker build -t $image_name - <<Dockerfile
FROM centos:centos7.6.1810RUN mkdir /tools
WORKDIR /tools#Install default services
RUN yum install -y wget git gcc bzip2 fontconfig python3 java-1.8.0-openjdk-devel.x86_64
RUN ln -sf /usr/bin/python3 /usr/bin/pythonENV JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk/
ENV PATH $JAVA_HOME/bin:$PATHRUN wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificate
RUN tar xfz apache-maven-3.9.4-bin.tar.gz
RUN ln -sf /tools/apache-maven-3.9.4 /tools/mavenENV PATH /tools/maven/bin:$PATH
ENV MAVEN_OPTS "-Xmx2048m -XX:MaxPermSize=512m"RUN mkdir -p /scripts
RUN echo "#!/bin/bash" > /scripts/mvn.sh
RUN echo 'set -x; exec "\$@" ' >> /scripts/mvn.shRUN chmod -R 777 /scripts
RUN chmod -R 777 /toolsENTRYPOINT ["/scripts/mvn.sh"]
Dockerfilefi
src_folder=`pwd`
LOCAL_M2="$HOME/.m2"
mkdir -p $LOCAL_M2
set -xdocker run --rm -v "${src_folder}:/ranger" -w "/ranger" -v "${LOCAL_M2}:${remote_home}/.m2" $container_name $image_name $params
说明：
考虑实验运行环境为centos7.6，且国内有部分外国源访问不到，所以做了一定修改和裁剪，本次实验中，使用root用户运行该脚本，容器内使用root用户进行编译操作。
1、原脚本的基础镜像为centos:lastest，其对应Centos 8.1，现修改为centos:centos7.6.1810；
2、原脚本安装jkd8时，使用AWS s3's docker-assets里的jdk-8u101-linux-x64.rpm，现修改为使用centos自带的openjdk1.8，即java-1.8.0-openjdk-devel.x86_64；
3、原脚本未安装python3，最终编译时会找不到python3包而报错，现增加安装python3，同时设置默认使用python3，即RUN ln -sf /usr/bin/python3 /usr/bin/python
4、原脚本安装maven时，使用ADD来获取apache-maven-3.6.3-bin.tar.gz并校验包，现修改为使用wget获得最新的apache-maven-3.9.4-bin.tar.gz，且不做额外的包正确性校验，即wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificatewget
5、原脚本的启动脚本创建并使用了非root用户builder，但会与后面运行容器时映射本地卷组有权限限制，考虑到只是临时编译使用，剔除所有builder用户的内容，包含gosu安装、用户创建、用户判断等，只保留echo 'set -x; exec "\$@" ' >> /scripts/mvn.sh
6、原脚本中${remote_home}为空值，会将运行该脚本的用户Home下的.m2映射到容器内根目录的.m2，现修改为容器内工作用户的Home目录，即remote_home="$HOME"
2、运行脚本编译
chmod +x build_ranger_using_docker.sh
./build_ranger_using_docker.sh
说明：
参照脚本使用说明
#Usage: [sudo] ./build_ranger_using_docker.sh [-build_image] mvn <build params>
#Example 1 (default no param): (mvn -Pall -DskipTests=true clean compile package install)
#Example 2 (Regular build): ./build_ranger_using_docker.sh mvn -Pall clean install -DskipTests=true
#Example 3 (Recreate Docker image): ./build_ranger_using_docker.sh mvn -Pall -build_image clean install -DskipTests=true
3、编译检查
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for ranger 2.4.0:
[INFO]
[INFO] ranger ............................................. SUCCESS [ 12.567 s]
[INFO] Jdbc SQL Connector ................................. SUCCESS [ 13.553 s]
[INFO] Credential Support ................................. SUCCESS [ 14.914 s]
[INFO] Audit Component .................................... SUCCESS [01:09 min]
[INFO] ranger-plugin-classloader .......................... SUCCESS [ 9.662 s]
[INFO] Common library for Plugins ......................... SUCCESS [02:03 min]
[INFO] ranger-intg ........................................ SUCCESS [ 40.185 s]
[INFO] Installer Support Component ........................ SUCCESS [ 8.196 s]
[INFO] Credential Builder ................................. SUCCESS [ 12.157 s]
[INFO] Embedded Web Server Invoker ........................ SUCCESS [ 33.355 s]
[INFO] Key Management Service ............................. SUCCESS [01:40 min]
[INFO] HBase Security Plugin Shim ......................... SUCCESS [ 52.109 s]
[INFO] HBase Security Plugin .............................. SUCCESS [01:25 min]
[INFO] Hdfs Security Plugin ............................... SUCCESS [ 36.159 s]
[INFO] Hive Security Plugin ............................... SUCCESS [ 41.491 s]
[INFO] Knox Security Plugin Shim .......................... SUCCESS [ 9.255 s]
[INFO] Knox Security Plugin ............................... SUCCESS [ 21.750 s]
[INFO] Storm Security Plugin .............................. SUCCESS [ 16.017 s]
[INFO] YARN Security Plugin ............................... SUCCESS [ 13.554 s]
[INFO] Ozone Security Plugin .............................. SUCCESS [ 12.752 s]
[INFO] Ranger Util ........................................ SUCCESS [ 11.776 s]
[INFO] Unix Authentication Client ......................... SUCCESS [ 11.990 s]
[INFO] User Group Synchronizer Util ....................... SUCCESS [ 6.909 s]
[INFO] Security Admin Web Application ..................... SUCCESS [08:54 min]
[INFO] KAFKA Security Plugin .............................. SUCCESS [01:17 min]
[INFO] SOLR Security Plugin ............................... SUCCESS [01:18 min]
[INFO] NestedStructure Security Plugin .................... SUCCESS [ 24.474 s]
[INFO] NiFi Security Plugin ............................... SUCCESS [ 12.265 s]
[INFO] NiFi Registry Security Plugin ...................... SUCCESS [ 11.211 s]
[INFO] Presto Security Plugin ............................. SUCCESS [ 24.201 s]
[INFO] Kudu Security Plugin ............................... SUCCESS [ 14.920 s]
[INFO] Unix User Group Synchronizer ....................... SUCCESS [02:08 min]
[INFO] Ldap Config Check Tool ............................. SUCCESS [ 11.640 s]
[INFO] Unix Authentication Service ........................ SUCCESS [ 11.348 s]
[INFO] KMS Security Plugin ................................ SUCCESS [01:13 min]
[INFO] Tag Synchronizer ................................... SUCCESS [ 45.784 s]
[INFO] Hdfs Security Plugin Shim .......................... SUCCESS [ 9.535 s]
[INFO] Hive Security Plugin Shim .......................... SUCCESS [01:23 min]
[INFO] YARN Security Plugin Shim .......................... SUCCESS [ 42.092 s]
[INFO] OZONE Security Plugin Shim ......................... SUCCESS [ 23.710 s]
[INFO] Storm Security Plugin shim ......................... SUCCESS [ 10.665 s]
[INFO] KAFKA Security Plugin Shim ......................... SUCCESS [ 10.838 s]
[INFO] SOLR Security Plugin Shim .......................... SUCCESS [ 22.091 s]
[INFO] Atlas Security Plugin Shim ......................... SUCCESS [ 28.752 s]
[INFO] KMS Security Plugin Shim ........................... SUCCESS [ 52.920 s]
[INFO] Presto Security Plugin Shim ........................ SUCCESS [ 26.065 s]
[INFO] ranger-examples .................................... SUCCESS [ 0.272 s]
[INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [ 11.692 s]
[INFO] Ranger Examples - SampleApp ........................ SUCCESS [ 5.863 s]
[INFO] Ranger Examples - Ranger Plugin for SampleApp ...... SUCCESS [ 10.167 s]
[INFO] sample-client ...................................... SUCCESS [ 11.777 s]
[INFO] Apache Ranger Examples Distribution ................ SUCCESS [ 6.742 s]
[INFO] Ranger Tools ....................................... SUCCESS [ 35.518 s]
[INFO] Atlas Security Plugin .............................. SUCCESS [ 41.615 s]
[INFO] SchemaRegistry Security Plugin ..................... SUCCESS [03:02 min]
[INFO] Sqoop Security Plugin .............................. SUCCESS [ 53.693 s]
[INFO] Sqoop Security Plugin Shim ......................... SUCCESS [ 14.680 s]
[INFO] Kylin Security Plugin .............................. SUCCESS [03:33 min]
[INFO] Kylin Security Plugin Shim ......................... SUCCESS [ 41.171 s]
[INFO] Elasticsearch Security Plugin Shim ................. SUCCESS [ 22.381 s]
[INFO] Elasticsearch Security Plugin ...................... SUCCESS [ 37.204 s]
[INFO] Apache Ranger Distribution ......................... SUCCESS [02:26 min]
[INFO] Unix Native Authenticator .......................... SUCCESS [ 4.438 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49:17 min
[INFO] Finished at: 2023-08-07T10:43:31Z
[INFO] ------------------------------------------------------------------------
在target目录可以看到生成的程序包：
-rw-r--r-- 1 root root 579387182 Aug 7 18:42 ranger-2.4.0-admin.tar.gz
-rw-r--r-- 1 root root 43729654 Aug 7 18:43 ranger-2.4.0-atlas-plugin.tar.gz
-rw-r--r-- 1 root root 34172214 Aug 7 18:43 ranger-2.4.0-elasticsearch-plugin.tar.gz
-rw-r--r-- 1 root root 39122941 Aug 7 18:42 ranger-2.4.0-hbase-plugin.tar.gz
-rw-r--r-- 1 root root 37684529 Aug 7 18:42 ranger-2.4.0-hdfs-plugin.tar.gz
-rw-r--r-- 1 root root 37478412 Aug 7 18:42 ranger-2.4.0-hive-plugin.tar.gz
-rw-r--r-- 1 root root 56846325 Aug 7 18:42 ranger-2.4.0-kafka-plugin.tar.gz
-rw-r--r-- 1 root root 195376717 Aug 7 18:43 ranger-2.4.0-kms.tar.gz
-rw-r--r-- 1 root root 51454934 Aug 7 18:42 ranger-2.4.0-knox-plugin.tar.gz
-rw-r--r-- 1 root root 36625366 Aug 7 18:43 ranger-2.4.0-kylin-plugin.tar.gz
-rw-r--r-- 1 root root 34201 Aug 7 18:43 ranger-2.4.0-migration-util.tar.gz
-rw-r--r-- 1 root root 43393403 Aug 7 18:42 ranger-2.4.0-ozone-plugin.tar.gz
-rw-r--r-- 1 root root 57425250 Aug 7 18:43 ranger-2.4.0-presto-plugin.tar.gz
-rw-r--r-- 1 root root 16563346 Aug 7 18:43 ranger-2.4.0-ranger-tools.tar.gz
-rw-r--r-- 1 root root 36915 Aug 7 18:42 ranger-2.4.0-solr_audit_conf.tar.gz
-rw-r--r-- 1 root root 38256335 Aug 7 18:42 ranger-2.4.0-solr-plugin.tar.gz
-rw-r--r-- 1 root root 36860763 Aug 7 18:43 ranger-2.4.0-sqoop-plugin.tar.gz
-rw-r--r-- 1 root root 6376456 Aug 7 18:43 ranger-2.4.0-src.tar.gz
-rw-r--r-- 1 root root 51760282 Aug 7 18:42 ranger-2.4.0-storm-plugin.tar.gz
-rw-r--r-- 1 root root 31046503 Aug 7 18:42 ranger-2.4.0-tagsync.tar.gz
-rw-r--r-- 1 root root 20128101 Aug 7 18:42 ranger-2.4.0-usersync.tar.gz
-rw-r--r-- 1 root root 35792990 Aug 7 18:42 ranger-2.4.0-yarn-plugin.tar.gz
参考文档：
Ranger Installation Guide - Ranger - Apache Software Foundation