hadoop source code reading #20

1. Upgrading
from HDP2.1 to HDP2.4
2016/03/24
@wyukawa
HadoopSCR #hadoopreading

2. About 1 year ago
hFp://www.slideshare.net/tagomoris/upgrading-from-hdp-21-to-hdp-22

3. 2014/12-
MRv2/HDFS
Hive
Analysis3(HDP2.1)
Input data
fluentd-plugin-webhdfs
shib
Execute queries
over hiveserver2 on MR
hive batch on MR

4. HDP2.1.5.0
• Install over Ansible, without Ambari
• Hadoop 2.4.0
• YARN ResourceManager HA
• NameNode HA
• Hive 0.13

5. HDP2.3.4
• Install over Ambari but use Ansible
• Hadoop 2.7.1
– YARN ResourceManager REST API(include kill)
– shib use yarn command to kill on HDP2.1.5.0
• Hive 1.2.1
– backport Hive-12084 Hive queries with ORDER BY
and large LIMIT fails with OutOfMemoryError Java
heap space.
• Tez 0.7.0

6. change Hadoop/Hive sebng by ambari
• dfs.datanode.failed.volumes.tolerated=1
• hive.server2.authendcadon=NOSASL
• hive.server2.enable.doAs=false
• hive.auto.convert.join=false
• change memory size because of presto and fluentd
– increase hadoop/datanode/resourcemanager/
hiveserver2 heap
– decrease yarn.nodemanager.resource.memory-
mb and yarn.scheduler.maximum-allocadon-mb

7. blue green deployment
MRv2/HDFS
Hive
Analysis3(HDP2.1)
Input data
fluentd-plugin-webhdfs
shib
Execute queries
over hiveserver2 on MR
MRv2/Tez/HDFS
Hive
Analysis4(HDP2.3.4)
hive batch on Tez hive batch on MR

8. OOME occured at Hive on Tez
• report as HIVE-13238
– increase hive.tez.container.size
• report as HIVE-13239
Switch to hive batch on MR!
I have the reason to use MR because of shib status feature...

9. Why do I use shib on MR?
• can execute Hive on Tez
– set hive.execudon.engine=tez
• can’t monitor query status
• can’t kill query
• shib use mapreduce.job.name to idendfy
submiFed query

10. OMME occured at Hive on MR
• report as HIVE-13273
– hiveserver2 memory leak

11. need to upgrade HDP2.4
hFps://community.hortonworks.com/quesdons/20116/
logfdscacheflushdmer-thread-increase.html

12. HDP2.4.0
• almost same as HDP2.3.4 except for Kala,
Spark
• backport HIVE-12766 !!!!!!!!!!!!!!!!!!!!!!!!!
• AMBARI-14389

13. upgrading
from HDP2.3.4 to HDP2.4.0
• report as AMBARI-15418
I want hortonworks to fix this issue.
But, anyway, switch to hive batch on Tez!

14. hiveserver2 is stable!!!

15. 2016/3-
MRv2/HDFS
Hive
Analysis3(HDP2.1)
Input data
fluentd-plugin-webhdfs
shib
Execute queries
over hiveserver2 on MR
MRv2/Tez/HDFS
Hive
Analysis4(HDP2.4.0)
hive batch on Tez hive batch on MR

16. somedmes corrupt/missing block...
monitoring with grafana, prometheus, hadoop_exporter

17. Conclusion
• Don’t use HDP2.3.4 if you use hiveserver2
• use HDP2.4.0
• see Apache patch informadon
• I want to see HDP source repository
• somedmes need to change query(HIVE-9390)
• I want Hive commiFer