当前位置： 首页 > news >正文

Hive 部署

news 来源：原创 2024/5/18 22:41:45

安装mysql ，并创建hive库及授权如下操作：

[root@oversea-stable mysql]# systemctl start mysqld
[root@oversea-stable mysql]# grep temporary /var/log/mysqld.log 
2018-06-15T06:51:27.814464Z 1 [Note] A temporary password is generated for root@localhost: rdhC?1g+NSH,
2018-06-15T06:51:31.666047Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
[root@oversea-stable mysql]# mysql -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.22

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> alter user 'root'@'localhost' identified by '********';
Query OK, 0 rows affected (0.00 sec)

mysql> create database hive character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges on hive.* to 'hive'@'localhost' identified by '********';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> grant all privileges on hive.* to 'hive'@'%' identified by '********'; 
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)

mysql> quit
Bye
[root@oversea-stable ~]# systemctl enable mysqld

下载hive 2.3.3版本，并解压如下所示:

[hadoop@bus-stable ~]$ wget http://mirrors.hust.edu.cn/apache/hive/stable-2/apache-hive-2.3.3-bin.tar.gz
--2018-06-15 15:00:41--  http://mirrors.hust.edu.cn/apache/hive/stable-2/apache-hive-2.3.3-bin.tar.gz
Resolving mirrors.hust.edu.cn (mirrors.hust.edu.cn)... 202.114.18.160
Connecting to mirrors.hust.edu.cn (mirrors.hust.edu.cn)|202.114.18.160|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 232229830 (221M) [application/octet-stream]
Saving to: ‘apache-hive-2.3.3-bin.tar.gz’

100%[==========================================================================>] 232,229,830  635KB/s   in 12m 2s 

2018-06-15 15:12:43 (314 KB/s) - ‘apache-hive-2.3.3-bin.tar.gz’ saved [232229830/232229830]

[hadoop@bus-stable ~]$ tar xfz apache-hive-2.3.3-bin.tar.gz  -C /opt/
[hadoop@bus-stable ~]$ cd /opt/
[hadoop@bus-stable opt]$ ln -s apache-hive-2.3.3-bin hive
[hadoop@bus-stable opt]$ cd hive/conf/
 默认没有提供hive-site.xml，仅提供了配置模板hive-default.xml.template，需要把它复制成hive-site.xml
[hadoop@bus-stable conf]$ cp hive-env.sh{.template,}
[hadoop@bus-stable conf]$ cp hive-default.xml.template hive-site.xml
[hadoop@bus-stable conf]$ 
[hadoop@bus-stable conf]$ vim hive-env.sh
[hadoop@bus-stable conf]$ tail -8 hive-env.sh
# Set HADOOP_HOME to point to a specific hadoop install directory
HADOOP_HOME=/opt/hadoop

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/hive/conf

# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/opt/hive/lib
[hadoop@bus-stable conf]$

配置JDBC
复制JDBC的连接文件到 hive/lib 下，如下操作：

[hadoop@bus-stable hive]$ cd /tmp/
[hadoop@bus-stable tmp]$ tar xfz mysql-connector-java-5.1.46.tar.gz 
[hadoop@bus-stable tmp]$ cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /opt/hive/lib/
[hadoop@bus-stable tmp]$ 
[root@bus-stable ~]# vim /etc/profile
export HIVE_HOME=/opt/hive
PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin:$PATH

在HDFS上创建hive所用目录
因为在 hive-site.xml 中有默认如下配置：

  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>

所以进入 hadoop 安装目录 执行hadoop命令，在hdfs上新建/user/hive/warehouse目录，并授权用于存储文件，如下操作：

[hadoop@bus-stable opt]$ cd hadoop
[hadoop@bus-stable hadoop]$ bin/hadoop fs -mkdir -p /user/hive/warehouse  
[hadoop@bus-stable hadoop]$ bin/hadoop fs -mkdir -p /user/hive/tmp 
[hadoop@bus-stable hadoop]$ bin/hadoop fs -mkdir -p /user/hive/log  
[hadoop@bus-stable hadoop]$ bin/hadoop fs -chmod -R 777 /user/hive/warehouse  
[hadoop@bus-stable hadoop]$ bin/hadoop fs -chmod -R 777 /user/hive/tmp  
[hadoop@bus-stable hadoop]$ bin/hadoop fs -chmod -R 777 /user/hive/log
[hadoop@bus-stable hadoop]$

用以下命令检查目录是否创建成功
[hadoop@bus-stable hadoop]$ hadoop fs -ls /user/hive
Found 3 items
drwxrwxrwx   - hadoop supergroup          0 2018-06-15 16:13 /user/hive/log
drwxrwxrwx   - hadoop supergroup          0 2018-06-15 16:13 /user/hive/tmp
drwxrwxrwx   - hadoop supergroup          0 2018-06-15 16:13 /user/hive/warehouse
[hadoop@bus-stable hadoop]$

修改 hive-site.xml
(1) 搜索hive.exec.scratchdir,将该name对应的value修改为/user/hive/tmp

<property>  
    <name>hive.exec.scratchdir</name>  
    <value>/user/hive/tmp</value>  
</property>

(2) 搜索hive.querylog.location,将该name对应的value修改为/user/hive/log/hadoop

<property>
    <name>hive.querylog.location</name>
    <value>/user/hive/log/hadoop</value>
    <description>Location of Hive run time structured log file</description>
</property>

(3) 搜索javax.jdo.option.connectionURL,将该name对应的value修改为MySQL的地址

<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://oversea-stable:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>

(4) 搜索javax.jdo.option.ConnectionDriverName，将该name对应的value修改为MySQL驱动类路径

<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
</property>

(5) 搜索javax.jdo.option.ConnectionUserName，将对应的value修改为MySQL数据库登录名

<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>Username to use against metastore database</description>
</property>

(6) 搜索javax.jdo.option.ConnectionPassword，将对应的value修改为MySQL数据库的登录密码

<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive_db_password</value>
    <description>password to use against metastore database</description>
</property>

(7) 指定hive 运行的tmp目录
创建 tmp 目录
[hadoop@bus-stable hive]$ mkdir tmp
并在 hive-site.xml 中修改
把 ${system:java.io.tmpdir} 改成 /opt/hive/tmp
把 {system:user.name} 改成 {user.name}

初始化mysql 如下操作：

[hadoop@bus-stable hive]$ bin/schematool -initSchema -dbType mysql 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.9.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:mysql://oversea-stable:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:       hive
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed
[hadoop@bus-stable hive]$

运行hive，并测试 如下操作：

[hadoop@bus-stable hive]$ hive
which: no hbase in (/usr/java/latest/bin:/opt/hadoop/bin:/opt/hive/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.9.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/apache-hive-2.3.3-bin/lib/hive-common-2.3.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>  create database inspiry; 
OK
Time taken: 5.999 seconds
hive> use inspiry; 
OK
Time taken: 0.032 seconds
hive>  create table test (mykey string,myval string);
OK
Time taken: 0.657 seconds
hive> insert into test values("1","www.inspiry.cn");
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = hadoop_20180615164443_64b15698-8522-45d0-8867-a0271f2c88be
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1529036356241_0005, Tracking URL = http://oversea-stable:8088/proxy/application_1529036356241_0005/
Kill Command = /opt/hadoop/bin/hadoop job  -kill job_1529036356241_0005
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2018-06-15 16:45:08,423 Stage-1 map = 0%,  reduce = 0%
2018-06-15 16:45:17,910 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.58 sec
MapReduce Total cumulative CPU time: 2 seconds 580 msec
Ended Job = job_1529036356241_0005
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://inspiryhdfs/user/hive/warehouse/inspiry.db/test/.hive-staging_hive_2018-06-15_16-44-43_063_5287772895297314577-1/-ext-10000
Loading data to table inspiry.test
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1   Cumulative CPU: 2.58 sec   HDFS Read: 4081 HDFS Write: 85 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 580 msec
OK
Time taken: 37.688 seconds
hive>  select * from test;
OK
1       www.inspiry.cn
Time taken: 0.235 seconds, Fetched: 1 row(s)
hive> quit;
[hadoop@bus-stable hive]$

在浏览器中查看HDFS 下inspiry库test表 如下图所示：
Hive 部署

Hive 部署

转载于:https://blog.51cto.com/caiyuanji/2131255

相关文章：

  • Openstack 之 Prometheus 监控
  • python3 异步模块asyncio
  • 紫书 习题 11-10 UVa 12264 （二分答案+最大流）
  • 大数据经典学习路线（及供参考）
  • 一起来学SpringBoot | 第三篇：SpringBoot日志配置
  • MySQL数据库运维之数据恢复
  • 函数防抖和函数节流
  • 持续开放，腾讯TARS、TSeer助力Linux建设开源社区
  • 探索 JS 中的模块化
  • 跟我一起学docker(四)--容器的基本操作
  • Ubuntu安装jdk
  • python全栈开发 * 19 面向对象 知识点汇总 * 180701
  • replace 使用正则
  • LeetCode541. Reverse String II -- 按步长反转字符串
  • 三菱加工中心CNC编程G代码讲解
  • bootstrap创建登录注册页面
  • CSS 专业技巧
  •  D - 粉碎叛乱F - 其他起义
  • jquery cookie
  • Laravel 菜鸟晋级之路
  • mysql常用命令汇总
  • uni-app项目数字滚动
  • weex踩坑之旅第一弹 ~ 搭建具有入口文件的weex脚手架
  • 从tcpdump抓包看TCP/IP协议
  • 记一次用 NodeJs 实现模拟登录的思路
  • 精彩代码 vue.js
  • 强力优化Rancher k8s中国区的使用体验
  • 深入 Nginx 之配置篇
  • 十年未变！安全，谁之责？（下）
  • 听说你叫Java（二）–Servlet请求
  • 在GitHub多个账号上使用不同的SSH的配置方法
  • ​​​​​​​​​​​​​​Γ函数
  • ​LeetCode解法汇总2696. 删除子串后的字符串最小长度
  • ​ubuntu下安装kvm虚拟机
  • ​马来语翻译中文去哪比较好？
  • #{}和${}的区别？
  • (04)odoo视图操作
  • (2009.11版)《网络管理员考试 考前冲刺预测卷及考点解析》复习重点
  • （4）事件处理——（2）在页面加载的时候执行任务（Performing tasks on page load）...
  • （6）设计一个TimeMap
  • (阿里巴巴 dubbo,有数据库，可执行 )dubbo zookeeper spring demo
  • （编程语言界的丐帮 C#）.NET MD5 HASH 哈希 加密 与JAVA 互通
  • (层次遍历)104. 二叉树的最大深度
  • （二）Pytorch快速搭建神经网络模型实现气温预测回归(代码+详细注解)
  • (免费领源码）Python#MySQL图书馆管理系统071718-计算机毕业设计项目选题推荐
  • （四）Linux Shell编程——输入输出重定向
  • （原+转）Ubuntu16.04软件中心闪退及wifi消失
  • .dat文件写入byte类型数组_用Python从Abaqus导出txt、dat数据
  • .NET 8 编写 LiteDB vs SQLite 数据库 CRUD 接口性能测试（准备篇）
  • .NET MVC第五章、模型绑定获取表单数据
  • :=
  • @Import注解详解
  • [ 网络基础篇 ] MAP 迈普交换机常用命令详解
  • [100天算法】-每个元音包含偶数次的最长子字符串（day 53）
  • [BUUCTF NewStarCTF 2023 公开赛道] week3 crypto/pwn