SAP HANA setup and configuration for scaleout scenario with High-Availability and associated HANA Monitoring

1. SAP HANA Distributed System
Scale out and HA
This is a compilation of notes from installation/configuration of SAP HANA
multiple-host system, including automatic failover testing, and monitoring.
By OZSoft Consulting for ITConductor.com
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 1

2. Introduction
Overview of system architecture
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 2
• HANA scaleout requires data persistence layer to be on either shared storage
(such as NFS, NAS), SAN with clustered filesystem, or non-shared storage using
HAN storage connector API
In our scenario OZHANANFS – prerequisite shared file system (NFS) to contain
• Installation path (sapmnt) /hana/shared
• Data volume /hana/data/<SID>
• Log volume /hana/log/<SID>

3. References
Reference:
• SAP HANA Administration Guide
• SAP HANA Server Installation and Update Guide - SAP Help
Portal
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 3

4. Installation - overview
In this document, there are two installation steps
1. Install the server database in the master host (ozhanaitc) and at
the same time add host ozhdbnode2
2. Add another host ozhdbnode3
Some of the prerequisites
- All hosts in a multi-host system must have the same sapsys group
ID
- Although not strictly required, for convenience all hosts should
have the same root password, or can be overwritten during
installation. Installation is handled by hdblcm with root privileges
- Shared mount /hana accessible to all hosts
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 4

5. Master host + additional host Installation (1)
The installation of a SAP HANA multiple-host system uses the same hdblcm tool
used in installing single-host system, with additional prompts
• To add another host and its role
• Certificate to use in the additional host, for internal communications (between
hosts as well as between processes in a single-host)
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 5

6. Master host + additional host Installation (2)
After the installation, the two hosts and all the services will be
visible in the HANA studio. The ‘IGNORE’ in Host Status
column represents the STANDBY host
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 6

7. Adding a host
Reference: Add Hosts Using the Command-Line Interface
Command line to add worker host ozhdbnode3 to master host ozhanaitc
Display – hosts after the addition in HANA Studio
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 7
cd /hana/shared/HDB/HDB00/hdblcm
./hdlcm --addhosts=ozhdbnode3:role=worker
--certificates_hostmap=ozhdbnode3=ozhanaitc
--root_password=XXXXXXX
--remote_execution=saphostagent
--use_http=yes

8. Volume mount points after installation
After the installation, only the volumes for the SYSTEMDB will be created, e.g.
/hana/data/HDB/mnt00001/hdb00001
/hana/log/HDB/mnt00001/hdb00001
The digit at the end of mnt00001 refers to the host sequence.
mnt00001 is for the master host (in this example, ozhanaitc)
mnt00002 is for the next host that will have data
Only after creating a tenant database which implicitly contains an indexserver that
corresponding volumes will be created.
For example
‘create database 4H at location ozhanaitc system user password xxxxx’
This will be located still in /mnt00001 (since its on the same host with ozhanaitc)
‘create database ND3 at location ozhdbnode3 system user password yyy’
This will create /mnt00002
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 8

9. Redistribution of tables in Distributed System
A feature available with a HANA distributed system is ‘table distribution’:
• In a distributed system, tables and table partitions are assigned to an index
server on a particular host at the time of their creation, but this assignment can
be changed – the process known as “redistribution operations”
• Reference: 2081591 - FAQ: SAP HANA Table Distribution
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 9

10. Redistribution (2)
Use redistribution for the following needs:
• Redistribute data before removing a host from a system
• Redistribute data after adding a new host to the system, it can be done by
adding an index server to a tenant DB in another host.
• Optimize current table distribution
• Optimize table partitioning, commonly for SAP BW usage scenario
For example, the following tenants exist
• Tenant DB A4H in ozhanaitc
• Tenant DB ND3 in ozhdbnode3
• Alter database A4H add ‘indexserver’ at location ‘ozhdbnode3:30049’
• Alter database ND3 add ‘indexserver’ at location ‘ozhanaitc:30061’
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 10
OZHANAITC
Tenant DB A4H
OZHDBNODE3
Tenant DB ND3

11. Redistribution (3)
Table/view(s) related to table distribution
• REORG_PLAN – This view contains the last table redistribution plan
generated with this database connection. The contents of the session is
temporarily stored and will be deleted when the connection is closed.
• REORG_STEPS This view contains the executed or to be executed table
redistribution plan items
To easily display any possible ‘movement’,
Select * from reorg_steps
Where new_host is not null;
To display tables where run-time host_name is different from the original host
location of the tenant DB,
Select * from m_cs_tables
Where schema_name = ‘<schema_name>’
And host <> ‘<orginal_host>’
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 11

12. HA Failover – Testing (1)
Testing automatic failover to standby, when a worker host goes down
1. Normal/original configured roles
2. STOP worker ozhdbnode3
• Standby ozhdbnode2 will have ‘HOST_STATUS=PARTIAL’
• WORKER ozhdbnode3 will have ‘HOST_STATUS=NO’, and
‘FAILOVER_STATUS=FAILOVER to OZHDBNODE2’
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 12

13. HA Failover – Testing (2)
3. After the takeover is complete
• ozhdbnode2 is now a ‘WORKER’ host
• ozhdbnode3 is now the ‘STANDBY’ host
• the host roles are now switched (i.e. host_config_roles <> host_actual_roles)
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 13
ozhdbnode3
ozhdbnode2

14. HA Failover – Testing (3)
4. Restarting ozhdbnode3 (originally configured as SLAVE)
• Both ozhdbnode2 and ozhdbnode2 will have
HOST_STATUS=INFO
• By design, HANA will not automatically fail back to the
original configured worker system – even assuming it’s fixed
and restarted, therefore after the first failover, its’ Actual
role will be different than the Configured role
• There is no ‘automatic’ takeover, the status of ozhdbnode3
sysem will remain as STANDBY
• In order to restore the actual roles=configured roles, a
manual process must be invoked to stop ozhdbnode2, or
restart the entire HANA cluster
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 14

15. HA Failover – Testing (4)
5. Now stop ozhdbnode2 so ozhdbnode3 can takeover its
original role (i.e. SLAVE Worker)
a) While stopping of ozhdbnode2 system is in progress, ozhdbnode2 status will be
‘STOPPING/WARNING’
b) ozhdbnode3 will have status ‘starting/warning’, indicating it is taking over
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 15

16. HA Failover – Testing (5)
6. When ozhdbnode3 has fully taken over, its
host_actual_role will revert to its original role (SLAVE)
Finally, the host roles are now the same as it was originally configured
Note that while ozhdbnode2 is IGNORED, the OVERVIEW status in HANA
Studio will be red, since the services of ozhdbnode2 are stopped.
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 16

17. Monitoring/Reporting (1)
SQL queries that can be run from SYSTEMDB to monitor multiple-host systems.
Display any inactive services, including tenant DB services
Select *
From “SYS_DATABASES”.”M.SERVICES”
Where active_status = ‘NO’
List host that is not active or where the configured role is not the same as its active role
(i.e. after a failover has occurred)
Select *
from m_landscape_host_configuration
where
(
(host_active = 'NO')
OR
(host_config_roles <> host_active_roles)
);
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 17

18. Monitoring/Reporting (2)
Other views worth reporting from
M_LANDSCAPE_HOST_CONFIGURATION
• Detail information for each host (master, worker, standby, dynamic
tiering), - failover group, configuration and actual roles roles of
nameserver, indexserver and host.
M_DATA_VOLUMES
- Volume/filename for each database
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 18

19. Enterprise Monitoring
• Many configuration, status, and alerts can occur in distributed
HANA environment which require continuous monitoring.
Solutions can be automated via the OZSOFT HANA Management
Pack (HANA MP) for Microsoft SCOM and/or IT-Conductor
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 19

20. What’s next
Topics for future blogs:
- HANA DR with System Replication
- Dynamic Tiering – Data provision installation, extended
storage maintenance, and management
Author: Terry Kempis
Editor: Linh Nguyen
ITConductor.com 20

21. • Any Questions?
• Contact Info:
Linh Nguyen linh.nguyen@itconductor.com
Terry Kempis terry.kempis@itconductor.com
Let
itconductor.com
@itconductor