問題描述
1、hadoop集群部署在騰訊云上，一個master，兩個slave
2、hadoop可以跑官方wordcount，能出結果
3、hadoop的版本是3.1.2，hive的版本是3.1.1
4、hive在跑select count(1) from t_test時卡死，事實上t_test表中只有兩條資料
5、hive卡死的現象如下：
hive> set mapreduce.job.reduces=2;
hive> set hive.compute.query.using.stats=false;
hive> select count(1) from t_test;
Query ID = root_20191130113749_ff62498c-ee7b-41a5-a4e4-bc3653b074d1
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
6、在/tmp/host/hive.log中發現時有mr的job創建的，如下：
2019-11-30T11:37:55,318 WARN [614f0786-de9d-4388-8fd4-28ecbf967e14 main] mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-11-30T11:37:55,399 INFO [614f0786-de9d-4388-8fd4-28ecbf967e14 main] mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1575042073347_0015
7、但是在http://134.175.61.114:8088/cluster/里面卻找不見這個job
uj5u.com熱心網友回復：
補充一下：昨天晚上掛起來跑了一夜，早上發現在http://134.175.61.114:8088/cluster/發現來job，資訊如下：
User: root
Name: select count(1) from t_test (Stage-1)
Application Type: MAPREDUCE
Application Tags:
Application Priority: 0 (Higher Integer value indicates higher priority)
YarnApplicationState: FAILED
Queue: default
FinalStatus Reported by AM: FAILED
Started: Sat Nov 30 02:13:22 +0800 2019
Elapsed: 24mins, 30sec
Tracking URL: History
Log Aggregation Status: DISABLED
Application Timeout (Remaining Time): Unlimited
Diagnostics: Application application_1575042073347_0009 failed 2 times due to ApplicationMaster for attempt appattempt_1575042073347_0009_000002 timed out. Failing the application.
Unmanaged Application: false
Application Node Label expression: <Not set>
AM container Node Label expression: <DEFAULT_PARTITION>
uj5u.com熱心網友回復：
如果感興趣，我可以提供系統賬號，查看更豐富詳細的日志uj5u.com熱心網友回復：
幾個小時過去來，終于看到錯誤資訊了：Hive Session ID = d802bdae-f12c-4074-be9e-9c425bffed34
hive> set mapreduce.job.reduces=2;
hive> set hive.compute.query.using.stats=false;
hive> select count(1) from t_test;
Query ID = root_20191130113749_ff62498c-ee7b-41a5-a4e4-bc3653b074d1
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1575042073347_0015, Tracking URL = http://master:8088/proxy/application_1575042073347_0015/
Kill Command = /opt/hadoop/bin/mapred job -kill job_1575042073347_0015
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
2019-11-30 12:41:14,331 Stage-1 map = 0%, reduce = 0%
Ended Job = job_1575042073347_0015 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive>
Socket error Event: 32 Error: 10053.
Connection closing...Socket close.
uj5u.com熱心網友回復：
你不加前面兩條，直接執行 select count(1) from t_test; 會怎么樣？uj5u.com熱心網友回復：
很快就回傳了，不過是0，因為讀的統計資訊，不走MR
uj5u.com熱心網友回復：
什么叫讀的是統計資訊？HQL默認就會轉換為mr去執行的，本身就不用設定
uj5u.com熱心網友回復：
hive默認設定 hive.compute.query.using.stats=true //改為false 不從元資料取值，可計數，但性能不高，不推薦
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/131151.html
標籤：分布式計算/Hadoop
上一篇：靜態路由配置
下一篇：機械行業以后的發展怎么樣呀？