Vision of our problems Searching for solutions Practical software Some more ideas Analysis of results Literature Questions and answers

1. Security Walls in Linux
Environment: Practice,
Experience, and Results
Mykola Perehinets
I&O, IS Application Administrator
SoftServe Inc., 11/02/2016
System-Part1

2. Agenda
 Vision of our problems
 Searching for solutions
 Practical software
 Some more ideas
 Analysis of results
 Literature
 Questions and answers

3. Vendors Vision of Situation
GNU/Linux distribution for ALL people

4. Cruel Reality and Other Issues
Dirty COW (CVE-2016-5195) is a privilege
escalation vulnerability in the Linux Kernel
Why is it called the Dirty COW bug?
"A race condition was found in the way the Linux kernel's memory
subsystem handled the copy-on-write (COW) breakage of private read-only
memory mappings. An unprivileged local user could use this flaw to gain
write access to otherwise read-only memory mappings and thus increase
their privileges on the system."
Y2007 Y2016

5. Cruel Reality and Other Issues
Dirty COW (CVE-2016-5195) is a privilege
escalation vulnerability in the Linux Kernel
Why is it called the Dirty COW bug?
"A race condition was found in the way the Linux kernel's memory
subsystem handled the copy-on-write (COW) breakage of private read-only
memory mappings. An unprivileged local user could use this flaw to gain
write access to otherwise read-only memory mappings and thus increase
their privileges on the system."
Y2007 Y2016
Hackers
Vulnerability
Rootkits
Trojans
Human factors

6. Our Vision of Situation Y2076

7. Our Vision of Situation Y2016
Distribution for YOUR PRODUCTION!!!

8. 8. And conspired all of them together to come and to fight
against Jerusalem, and to hinder it.
14. And I looked, and rose up, and said unto the nobles, and to
the rulers, and to the rest of the people, Be not ye afraid of
them: remember the Lord, which is great and terrible, and fight
for your brethren, your sons, and your daughters, your wives,
and your houses.
17. They which builded on the wall, and they that bare burdens,
with those that laded, every one with one of his hands wrought
in the work, and with the other hand held a weapon.
18. For the builders, every one had his sword girded by his side,
and so builded. And he that sounded the trumpet was by me.
Nehemiah 4:8-18
Your Vision of Situation

9. Our Vision of Situation Y2016
Distribution for MY PRODUCTION!!!

10. Your Vision of Situation

11. Real Way for Us

12. We Really Need Solutions

13. We Really Need Solutions

14. Real Way for Us

15. Practices of Security
Internal
Audit
Protection of
Communications
Protection of File
Systems and Data
Protection of
Configuration
Files

16. Practices of Security
Protection
of Kernel
Internal
Audit
Protection of
Communications
Protection of File
Systems and Data
Protection of
Configuration
Files

17. Practices of Security (Software)
1. Etckeeper - is a revision
control system for your /etc
directory using bzr, git, hf, or
darcs as a back-end.
https://github.com/joeyh/etc
keeper
2. AIDE - (Advanced Intrusion Detection Environment - Host
Based IDS) is a file and directory integrity checker. It creates a
database from the regular expression rules that it finds from
the config file(s). Once this database is initialized it can be
used to verify the integrity of the files.
http://aide.sourceforge.net/
3. Tripwire Software - can help to ensure the integrity of critical
system files and directories by identifying all changes made to
them. http://www.tripwire.com/
Protection of
Configuration
Files

18. Practices of Security
4. Spacewalk is an open source
Linux systems management
solution that allows you to:
manage and deploy
configuration files to your
systems, distribute content
across multiple geographical sites in an efficient manner,
inventory your systems. http://spacewalk.redhat.com/
https://fedorahosted.org/spacewalk/wiki/HowToInstall#Settingup
Spacewalkrepo
5. Setup a Local Mail Server and Create Server Mail Group.
[root@ua /]# cat /etc/aliases
root: SecurityOperators@softserveinc.com
6. Use LogWatch is a log parsing program that analyzes and
generates daily reports on your system’s log activity.
Protection of
Configuration
Files

19. Practices of Security (Software)
1. Chkrootkit - locally checks for
signs of a rootkit.
http://www.chkrootkit.org/
2. Rkhunter - scanner tool for
Linux systems (+need update).
3. ClamAV - antivirus engine for detecting trojans, viruses,
malware & other malicious threats. http://www.clamav.net/
4. Available Repositories Provided by CentOS - these repositories
have varying levels of stability, support and cooperation
within the CentOS community. Please Verify Your Repo List!
https://wiki.centos.org/AdditionalResources/Repositories
5. Install additional plugin yum-cron - The package that allows us
to do automatic updates via yum (auto-update mechanism).
Please Always Update Your Systems!
Protection of
File Systems
and Data

20. Practices of Security
6. Spacewalk - is a systems
management solution allows
you to: install and update
software on your systems,
collect and distribute your
custom software packages.
7. Bacula/Bareos - is a set of Open Source, computer programs
that permit you (or the system administrator) to manage
backup, recovery, and verification of computer data across a
network of computers of different kinds. Bacula is relatively
easy to use and very efficient, while offering many advanced
storage management features that make it easy to find and
recover lost or damaged files. Please Backup Your Systems!
http://blog.bacula.org/source-download-center/
http://download.bareos.org/bareos/release/latest/
Protection of
File Systems
and Data

21. Practices of Security
8. Bacula File Integrity Check is a
feature can be used for
detecting changes to critical
system files similar to what a
file integrity checker like
Tripwire does.
9. OSSEC - is a Open Source HIDS SECurity. OSSEC watches it all,
actively monitoring all aspects of Unix system activity with file
integrity monitoring, log monitoring, rootcheck, and process
monitoring, writing scripts that take actions in response to
security alerts. http://ossec.github.io/
https://atomicorp.com/ , http://wazuh.com/
https://www.alienvault.com/
Protection of
File Systems
and Data

22. Practices of Security
10. Secure Partition Mount Options
please use in /etc/fstab:
noatime,nosuid,noexec,nodev
11. Use Secure Disk Partitioning
use for your server:
“/boot”, “/”, “/home”, “/var”,
“/tmp”, “/usr”, “/opt”
Protection of
File Systems
and Data
12. Prevent Mounting USB Storage in your servers
echo "install usb-storage /bin/false" > /etc/modprobe.d/usb-
storage.conf
13. Mount “/boot” partition in ‘read-only’ mode
use for this in /etc/fstab next options for “/boot”:
defaults,nosuid,nodev,ro
(manually re-mount as ‘read-write’ for system update)

23. [root@ua /]# df -Th
Ф. система Тип Розм Вик Дост Вик% змонтований на
devtmpfs evtmpfs 16G 0 16G 0% /dev
tmpfs tmpfs 16G 84K 16G 1% /dev/shm
tmpfs tmpfs 16G 410M 16G 3% /run
tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/system--lvm-root xfs 60G 657M 59G 2% /
/dev/mapper/system--lvm-usr xfs 60G 6,9G 53G 12% /usr
tmpfs tmpfs 16G 4,0M 16G 1% /tmp
/dev/sda1 xfs 1014M 402M 613M 40% /boot
/dev/mapper/system--lvm-var xfs 30G 5,8G 25G 20% /var
/dev/mapper/system--lvm-RW xfs 334G 58G 277G 18% /RW
/dev/mapper/system--lvm-home xfs 15G 48M 15G 1% /home
tmpfs tmpfs 3,2G 16K 3,2G 1% /run/user/42
tmpfs tmpfs 3,2G 0 3,2G 0% /run/user/0
[root@ua /]#

24. Practices of Security
Protection
of Kernel
Internal
Audit
Protection of
Communications
Protection of File
Systems and Data
Protection of
Configuration
Files

25. Practices of Security (Software)
1. Edit sysctl.conf - is an tweaking feature that reads and modifies
the attributes of the system kernel such as its version number,
maximum limits, and security settings.
2. Use nscd - is a daemon that provides a cache for the most
common name service requests.
4. NTP Client (Chrony) to synchronize the time of your local Linux
client machine with NTP server, edit the /etc/ntp.conf file on
the client side. Comparison of NTP implementations.
5. Configure Rsyslog with Any Log File Forwarding to other server!
Protection
of Kernel
3. Configure DNS Client -
to configure Linux as
DNS client you need
to edit or modify
/etc/resolv.conf file.

26. [root@ua /]# cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
…
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
…
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.arp_filter = 1
…
# Log Martian Packets
net.ipv4.conf.all.log_martians = 1
vm.swappiness = 0
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_low_latency=1
…

27. Practices of Security
6. Security-Enhanced Linux (SELinux) - is an implementation of a
Mandatory Access Control mechanism in the Linux kernel,
checking for allowed operations after standard discretionary
access controls are checked. SELinux can enforce rules on files
and processes in a Linux system, and on their actions, based on
defined policies.
7. Applications optimization – Java:Huge Pages, Lan:Multipathing.
8. ELRepo - is a community repository for Enterprise Linux
distributions. ELrepo-kernel channel provides the latest Stable
Mainline Kernels. http://elrepo.org/tiki/kernel-ml
Protection
of Kernel
SELinux is enabled by
default in Red Hat
Enterprise Linux.
Please use option
enforcing or permissive!

28. Practices of Security
9. Write Custom System Audit Rules (in SELinux) - by default, the
audit system records only a few events in the logs such as users
logging in, users using sudo, and SELinux-related messages. It
uses audit rules to monitor for specific events and create
related log entries. It is possible to create personal audit rules!
Protection
of Kernel
[root@ua rules.d]# cat /etc/audit/rules.d/audit.rules
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

29. [root@ua rules.d]# cat /etc/audit/rules.d/audit.rules
…
-w /etc/localtime -p wa -k time-change
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -S stime -k time-change
-a always,exit -F arch=b64 -S clock_settime -k time-change
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
…
-a always,exit -F path=/usr/bin/mount -F perm=x -F auid>=500 -F auid!=4294967295 -k
privileged
-a always,exit -F path=/usr/bin/su -F perm=x -F auid>=500 -F auid!=4294967295 -k
privileged
-a always,exit -F path=/usr/bin/umount -F perm=x -F auid>=500 -F auid!=4294967295 -k
privileged
-a always,exit -F path=/usr/libexec/abrt-action-install-debuginfo-to-abrt-cache -F perm=x
-F auid>=500 -F auid!=4294967295 -k privileged
…
-w /etc/sudoers -p wa -k scope
-w /var/log/sudo.log -p wa -k actions
-w /sbin/rmmod -p x -k modules
-w /sbin/modprobe -p x -k modules
-a always,exit -F arch=b64 -S init_module -S delete_module -k modules
-e 2

30. Practices of Security (Software)
5. On-Line System Monitoring -
for SSH sessions – use Glances
is a cross-platform curses-based
system monitoring tool written
in Python.
https://github.com/nicolargo/glances
1. Service Management - Systemd is an init system and system
manager that is widely becoming the new standard for Linux
machines. Verify your services and DISABLE UNNEEDED!
2. Enable Firewall - Firewalld provides a dynamically managed
firewall with support for network/firewall zones to define the
trust level of network connections or interfaces.
3. How do I disable IPv6? (Daniel Walsh not recommends)
4. Use Multiple IP Network Interfaces/cards for prevent network
performance bottlenecks and improved security.
Protection of
Communications

31. Practices of Security

32. Practices of Security
For WEB sessions - real-time performance monitoring, done
right! This is the default dashboard of NetData: real-time, per
second updates, snappy refreshes! 300+ charts out of the box,
2000+ metrics monitored! zero configuration, zero maintenance,
zero dependencies! https://github.com/firehol/netdata
https://github.com/firehol/netdata/wiki/Installation
Protection of
Communications
For FULL TIME monitoring – use
monitoring with Collectd, InfluxDB
& Grafana or The InfluxData
Platform is the first purpose-built,
end-to-end solution for collecting,
storing, visualizing and alerting on
time-series data at scale.

33. Practices of Security
Protection of
Communications

34. Practices of Security
Protection of
Communications
https://influxdata.com/get-
started/sending-data-to-influxdb-
with-telegraf/

35. Practices of Security
This Platform Based on the TICK stack, all of the components of
the platform are designed to work together seamlessly.
http://www.vishalbiyani.com/graphing-performance-with-
collectd-influxdb-grafana/
http://grafana.org/
https://dbiers.me/setup-grafana-influxdb-collectd-centos-7-x/
https://influxdata.com/get-started/what-is-the-tick-stack/
https://influxdata.com/get-started/download-and-install-
influxdb/
Check_MK is comprehensive
IT monitoring solution in the
tradition of Nagios.
http://mathias-kettner.com/check_mk.html
Protection of
Communications

36. Practices of Security
9. Suricata Engine is an Open
Source, high performance
Network IDS, IPS and Network
Security Monitoring engine.
https://oisf.net/suricata/
6. Protect with Fail2Ban(+setup) - this solution scans log files
(e.g. /var/log/error_log) and bans IPs that show the malicious
signs - too many password failures, seeking for exploits, etc.
http://www.fail2ban.org/wiki/index.php/Main_Page
7. ‘Hang’ all Production Services/Demons to the Separate
Network Adapters and/or Ports. (+setup Your Firewall Rules)
8. Certbot, is an easy-to-use automatic client that fetches and
deploys SSL/TLS certificates for your webserver to USE HTTPS!
https://certbot.eff.org/about/
Protection of
Communications

37. [root@ua /]# netstat -ntulp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program
name
tcp 0 0 192.168.xxx.xxx:8000 0.0.0.0:* LISTEN 26610/python
tcp 0 0 0.0.0.0:4545 0.0.0.0:* LISTEN 3359/rhnmd
tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 4518/varnishd
tcp 0 0 192.168.xxx.xxx:6789 0.0.0.0:* LISTEN 26446/ceph-mon
tcp 0 0 0.0.0.0:9102 0.0.0.0:* LISTEN 3291/bacula-fd
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 2474673/netdata
tcp 0 0 192.168.xxx.xxx:80 0.0.0.0:* LISTEN 4518/varnishd
tcp 0 0 0.0.0.0:8083 0.0.0.0:* LISTEN 3277/influxd
tcp 0 0 0.0.0.0:8086 0.0.0.0:* LISTEN 3277/influxd
tcp 0 0 192.168.xxx.xxx:22 0.0.0.0:* LISTEN 3296/sshd
tcp 0 0 192.168.xxx.xxx:3000 0.0.0.0:* LISTEN 3279/grafana-server
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 3277/influxd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 4523/master
tcp 0 0 0.0.0.0:6556 0.0.0.0:* LISTEN 3292/xinetd
udp 0 0 0.0.0.0:8096 0.0.0.0:* 3277/influxd
udp 0 0 172.xxx.xxx.xxx:123 0.0.0.0:* 2481262/ntpd
udp 0 0 192.168.xxx.xxx:123 0.0.0.0:* 2481262/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 2481262/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 2481262/ntpd

38. Practices of Security
Protection
of Kernel
Internal
Audit
Protection of
Communications
Protection of File
Systems and Data
Protection of
Configuration
Files

39. Some More Ideas for Us
Sending alerts to administrators:
[root@ua /]# cat /etc/profile
…
echo “ALERT on `hostname`: Shell access to your server! Detail
information: incident time - '`date` `who`'.” | mail -s "ALERT
from `hostname`: Access to your server from IP: `who | cut -
d"(" -f2 | cut -d")" -f1`! Please verify this issue and approve (if
need)!" SecurityOperators@softserveinc.com
…
Improve SSH protocol security:
[root@ua /]# cat /etc/ssh/sshd_config
…
# Specifies the ciphers allowed for protocol version 2
Ciphers aes128-ctr, aes192-ctr, aes256-ctr, arcfour256,
arcfour128, arcfour

40. Some More Ideas for Us
# Specifies the MAC (message authentication code) algorithms
MACs hmac-sha1, umac-64@openssh.com, hmac-ripemd160,
hmac-sha2-256, hmac-sha2-512
…
Disable reboot using ‘CTRL+ALT+DELETE’ keys:
[root@ua /]# systemctl mask ctrl-alt-del.target
The CIS-CAT Benchmark Assessment Tool:
CIS-CAT is a host-based configuration assessment tool. A Java-
based tool that compares the configuration of target IT systems
to CIS Benchmarks and reports conformance scores
on a scale of 0-100.
https://benchmarks.cisecurity.org/downloads/audit-tools/
The OpenSCAP Family Tools:
https://www.open-scap.org/tools/

41. Some More Ideas for Us
Monitoring users activity using ‘psacct’ or ‘acct’ tools:
If you have lot of users who access your servers frequently in
your company and if you wanna to keep an eye on what data
they are accessing, what commands they are issuing, how long
they have been accessing servers and how much system
resources are consumed by them, then psacct or acct are the
tools that you should have (starting psacct or acct as service)!
Display Statistics of Users Day-wise:
[root@ua /]# ac -d
Display Time Totals for each User:
[root@ua /]# ac -p
Print All Account Activity Information:
[root@ua /]# sa
Use iPerf - The ultimate speed test tool for TCP, UDP and SCTP.

42. Practices of Security
Protection
of Kernel
Internal
Audit
Protection of
Communications
Protection of File
Systems and Data
Protection of
Configuration
Files

43. Analysis of Results (Software)
4. Security Content Automation
Protocol (SCAP) Validation
Program is designed to test the
ability of products to use the
features and functionality.
https://scap.nist.gov/
https://www.open-scap.org/
1. Nmap - ("Network Mapper") is a free and open source
(license) utility for network discovery and security auditing.
https://nmap.org/
2. Wireshark - is the world’s foremost and widely-used network
protocol analyzer. https://www.wireshark.org/
3. Nessus(+plugins) - prevents network attacks by identifying the
vulnerabilities and configuration issues that hackers use to
penetrate your network. http://www.tenable.com/
Internal
Audit

44. Analysis of Results
5. Tcpdump - dump traffic on a network.
http://www.tcpdump.org/
http://www.winpcap.org/windump/
6. Elastic Stack (Beats, Logstash, Elasticsearch, Kibana, X-Pack) -
Elastic's open source solutions solve a growing list of search,
log analysis, and analytics challenges across virtually every
industry. https://www.elastic.co/
https://www.elastic.co/downloads/x-pack
Internal
Audit
7. Logscape - is a big data
analytics tool, which allows
you to turn your data into
knowledge.
http://logscape.github.io/
http://logscape.com/

45. Analysis of Results
10. Splunk (+plugins) makes it simple
to collect, analyze and act upon
the untapped value of the big
data generated by your
technology infrastructure,
security systems and applications.
https://www.splunk.com/
8. Lynis - is an open source security auditing tool. It runs on the
host itself, so it performs more extensive security scans than
vulnerability scanners. https://cisofy.com/lynis/
9. OSSIM - AlienVault’s Open Source Security Information and
Event Management (SIEM) product, provides you with a
feature-rich open source SIEM complete with event collection,
normalization and correlation.
https://www.alienvault.com/products/ossim
Internal
Audit

46. Analysis of Results
11. HTM Studio - Find Real-Time Anomalies in your Streaming
Data. HTM Studio allows you to test whether our Hierarchical
Temporal Memory (HTM) algorithms will find anomalies in
your data. With just one click, you can uncover anomalies
other techniques cannot find in your numeric, time-series
data, in minutes. http://numenta.com/htm-studio/
Internal
Audit

47. Analysis of Results
The Center for Internet Security (CIS) is a organization
dedicated to enhancing the cybersecurity readiness and response
among public and private sector entities.
The CIS Security Benchmarks program provides vendor-
agnostic, consensus-based best practices to help organizations
assess and improve their security. Resources include:
• secure configuration benchmarks
• automated configuration assessment tools and content
• security metrics
• security software product certifications
The Security Benchmarks program is an independent authority
that helps both public and private industry experts collaborate and
find consensus on practical cybersecurity solutions. Our resources
are used by organizations worldwide to help meet compliance
requirements for FISMA, PCI, HIPAA and more.

48. Analysis of Results (Example)
Overview This document, CIS CentOS Linux 7 Benchmark,
provides prescriptive guidance for establishing a secure
configuration posture for CentOS version 7.0 running on x86 and
x64 platforms. To obtain the latest version of this guide, please
visit http://benchmarks.cisecurity.org.

49. Analysis of Results (Example)

50. Analysis of Results (Example)
1.1.1 Create Separate Partition for /tmp (Scored)
Profile Applicability:
Level 1
Description:
The /tmp directory is a world-writable directory used for temporary
storage by all users and some applications.
Rationale:
Since the /tmp directory is intended to be world-writable, there is a
risk of resource exhaustion if it is not bound to a separate partition.
In addition, making /tmp its own file system allows an
administrator to set the noexec option on the mount, making /tmp
useless for an attacker to install executable code. It would also
prevent an attacker from establishing a hardlink to a system setuid

51. Analysis of Results (Example)
program and wait for it to be updated. Once the program was
updated, the hardlink would be broken and the attacker would
have his own copy of the program. If the program happened to
have a security vulnerability, the attacker could continue to exploit
the known flaw.
Audit:
Verify that there is a /tmp file partition in the /etc/fstab file.
# grep "[[:space:]]/tmp[[:space:]]" /etc/fstab
Remediation:
For new installations, check the box to "Review and modify
partitioning" and create a separate partition for /tmp.
For systems that were previously installed, use the Logical Volume
Manager (LVM) to create partitions.
References:
AJ Lewis, "LVM HOWTO", http://tldp.org/HOWTO/LVM-HOWTO/

52. Analysis of Results (Example)

53. Real Way for Us

54. Security Walls in Linux Environment
Protection
of Kernel
Internal
Audit
Protection of
Communications
Protection of File
Systems and Data
Protection of
Configuration
Files

55. Literature
1) CIS CentOS Linux 7 Benchmark
2) Kernel sysctl configuration file for Linux
3) SELinux User's and Administrator's Guide
4) Multipathing
5) How To Use Systemctl to Manage Systemd
Services and Units
6) FirewallD
7) Security Harden CentOS 7
8) System Settings in Linux Server
9) Hacker Tools Top Ten Y2016
10)Defining Persistent Audit Rules and
Controls

56. Literature
11)Bossie Awards 2016: The best open source
networking and security software
12)Host Based IDS
13)Open Source Host-based Intrusion
Detection System (OSSEC)
14)How to Install Splunk on CentOS 7
15)Penetration Testing Framework

57. Questions and Answers
Thank you!
Mykola Perehinets
I&O, IS Application Administrator
Skype: mykola.perehinets
Cell: +380 67 772 6910