Securing & Hardening Linux Web Server (Part II – Practice)

This paper will try to cover the most important steps to properly securing and hardening your Linux web server.

  1. Part I   - Theory
  2. Part II – Practice

So, let’s assume that we bought a new dedicated server and we just installed from the vendor’s control panel a fresh Linux distribution with the following details:

Server (Dedicated)
Operating System (Ubuntu 11.04)
Web Server (Apache 2.2.x)
Database Server (MySQL 5.1.x)
Name Server (Bind 9.7.x)
SSH Server (OpenSSH 5.8.x)
Firewall (IPTables 1.4.x)

The following security solutions will most probably work with the different services’ versions and different Linux distributions. You will just need to find some more details for your system such as different paths for configuration files etc.

Operating System [Ubuntu]

Software and Services

- Review installed packages and remove unnecessary

1. dpkg -l			// list of installed packages or dpkg --get-selections
2. dpkg -s			// to get information abou the package
3. apt-get remove	// uninstall unnecessary packages

- Review running services and remove/disable unnecessary

1. service --status-all			// list services
2. service <name> [start|stop|status]	// start/stop temporary
3. update-rc.d <name> defaults		// enable permanently
4. update-rc.d -f <name> remove		// disable permanently

Avoid using unencrypted services such as FTP, TELNET, HTTP, etc… use instead SFTP/FTPS/SCP/RSYNC, SSH, HTTPS

Tools:  sysv-rc-conf, chkconfig, rcconf

- Keep your system up to date

1. apt-get update         // update package list
2. apt-get upgrade        // upgrade all packages

You should ensure that new packages are compatible with the other installed packages before you continue for the full upgrade.
However, the best security practice would be an automatic safe upgrade for security issues.  EG:-

1. paste the following data to /etc/cron.weekly/apt-security-updates    [new file]

echo "**************" >> /var/log/apt-security-updates
date >> /var/log/apt-security-updates
aptitude update >> /var/log/apt-security-updates
aptitude safe-upgrade -o Aptitude::Delete-Unused=false --assume-yes --target-release `lsb_release -cs`-security >> /var/log/apt-security-updates
echo "Security updates (if any) installed"

2. chmod +x /etc/cron.weekly/apt-security-updates
3. paste the following data /etc/logrotate.d/apt-security-updates    [new file]

/var/log/apt-security-updates {
rotate 2
weekly
size 250k
compress
notifempty
}

With the above commands we schedule the /etc/cron.weekly/apt-security-updates script to run every week, install the new security updates and keep the logs from the installation at /var/log/apt-security-updates.

We should also watch security forums, mailing lists and sites for new threats and patching our system as soon as possible.

User Accounts and Password Policy

We will need to remove any unnecessary user accounts and apply a strong password policy (allow only strong passwords, force users to change their passwords every 40 days and dont allow them to have similar passwords with their previous passwords)

- Remove unnecessary user accounts
1. userdel <username>            // delete user account

Don’t forget to delete temporary accounts and don’t allow accounts without a password. Review /etc/passwd, /etc/shadow files.

- Force users to use strong passwords using [pam_passwdqc]
1. apt-get install libpam-passwdqc
2. paste the following data to /etc/pam.d/common-password

password requisite pam_passwdqc.so min=disabled,24,12,8,7 max=40 passphrase=3 match=4 similar=deny random=47 enforce=everyone retry=3

This line will force users to use password with at least 7 chars with all 4 different char classes, 8 chars with 3 different char classes, 12 chars for a pass-phrase or 24 chars with 2 different classes. The max password can be 40 chars, it will not allow similar passwords with the previous one. See man pam_passwdc for more details.

- Force users to use strong passwords using [pam_cracklib]
1. paste the following data to /etc/pam.d/common-password

password required pam_cracklib.so retry=3 minlen=8 difok=4
password required pam_unix.so md5 remember=10 use_authtok

This will force users to use at least 8 chars password and it must be different from the last 10 passwords.

- Use Password Aging [new users]
1. add/edit the following lines to /etc/login.defs

PASS_MAX_DAYS=40
PASS_MIN_DAYS=0
PASS_WARN_AGE-5

2. add/edit the following line to /etc/default/useradd

INACTIVE=5

- Use Password Aging [existing users]
1. chage -m 0 -M 40 -I 5 -W 5 <username>

With the above password policy we will enforce our users to change their password every 40 days, they will be informed 5 days before and they will lock out 5 days later if they don’t change their password.

Kernel Parameters and Security Patches

- Review and configure properly the Kernel Parameters within /etc/sysctl.conf file
1. sysctl -a     // list parameters
2. sysctl -p    // load /etc/sysctl.conf parameters
3. paste the following data to /etc/sysctl.conf

# Prevent Stack Overflow Exploits
kernel.randomize_va_space=1
# Prevent SUID core dumps
fs.suid_dumpable=0
# Prevent packet forwarding
net.ipv4.ip_forward = 0
# Prevent source routing
net.ipv4.conf.all.accept_source_route=0
# Prevent IP spoofing
net.ipv4.conf.all.rp_filter=1
# Prevent SYN Flood
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_syn_backlog = 2048
# Prevent ICMP broadcasts/redirects
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.conf.all.accept_redirects = 0

To protect properly your system against exploits, (Stack/Heap) buffer overflows, format string and shell code attacks it is highly recommended to install TOMOYO with GrSecurity and PAX kernel patch. See also, SELinux and AppArmor. Ubuntu comes by default with AppArmor which is a good start.

- Securing kernel with AppArmor

1. apparmor_status            // Current status
2. apt-get install apparmor-profiles  // Update profiles' list
3. aa-genprof <application eg. /usr/sbin/apache2> // Add new profile
4. aa-complain <application>   // Placing in Complain mode for testing
5. aa-enforce <application>    // Placing in Enforce mode
6. aa-logprof   // Test apparmor profiles configuration and fix problems

Tools: AppArmor, GrSecurity+PAX, TOMOYO, SELinux, Chroot

File System

- Review your file system and permissions

1. find / \( -perm -4000 -o -perm -2000 \) –print   // Find SUID/SGID files
2. find / -perm -2 ! -type l -ls   // Find all wordl-writable files

- File Integrity [Tripwire]

1. apt-get install tripwire
// install tripwire and apply local/site passphrase
2. tripwire --init --cfgfile /etc/tripwire/tw.cfg --polfile /etc/tripwire/tw.pol --site-keyfile /etc/tripwire/site.key --local-keyfile /etc/tripwire/{YOUR_HOSTNAME}-local.key
// Initialize tripwire database
3. tripwire --check | tripwire -m c
// check your system for changes
4. cp /etc/tripwire/twcfg.txt /etc/tripwire/twcfg.txt.orig
// keep backup of configuration file
5. cp /etc/tripwire/twpol.txt /etc/tripwire/twpol.txt.orig
// keep backup of policy file
6. tripwire -m p -Z low /etc/tripwire/twpol.txt
// Edit policy rules to your needs and update them

It is recommended to add tripwire in cron and send the results with an email, eg:
7. add the following data to /etc/crontab

25 6 * * * /usr/sbin/tripwire -m c | /usr/bin/mail root -s "Tripwire Results" 2>&1

- File Integrity / Host-Based Intrusion Detection System (HIDS) [Samhain]

1. wget http://www.la-samhna.de/samhain/samhain-current.tar.gz
2. gunzip samhain-current.tar.gz
3. tar -xf samhain-current.tar
4. gunzip samhain-2.8.5a.tar.gz
5. tar -xf samhain-2.8.5a.tar
6. cd samhain-2.8.5a
7. ./configure  // run --help (enable kernel, suid checks etc)
8. make
9. make install
10. review /etc/samhainrc configuration file
11. samhain -t check --foreground  // check your system for changes

Tools: Tripwire , AIDE , OSSEC , Samhain , CFEngine, Afick

Backups

- File system backup [Incremental Backup with Tar]
1. Backup [1st day]

tar -zpcf backup-`date +%y%m%d`.tar.gz -g /var/log/backup_incremental.log /var/www /home (configuration files, logs, etc)

2. Backup [2nd day]

tar -zpcf backup-`date +%y%m%d`.tar.gz -g /var/log/backup_incremental.log /var/www /home

3. Restore [till 1st day]

tar -zxf date_1st_day.tar.gz -g /dev/null

4. Restore [till 2nd day]

tar -zxf date_1st_day.tar.gz -g /dev/null
tar -zxf date_2nd_day.tar.gz -g /dev/null

- MySQL database backup

1. Backup All Databases

mysqldump -u root -h localhost -p<password> --all-databases| gzip -9 > mysql-`date +%y%m%d`.sql.gz

2. Backup Specific Databases

mysqldump -u root -h localhost -p<password> --databases DB1 DB2 DB3 | gzip -9 > mysql-`date +%y%m%d`.sql.gz

3. Backup Specific Tables from DB1

mysqldump -u root -h localhost -p<password> DB1 TB1 TB2 | gzip -9 > mysql-`date +%y%m%d`.sql.gz

4. Restore All Databases backup

gzip < all_dbs.gz | mysql -u root -h localhost -p<password>

5. Restore Specific Database

gzip < db1.gz | mysql -u root -h localhost -p<password> DB1

- Full disk image backup (NOT TESTED!)

1. Backup the full image of your Linux hard drive (close services and applications)

tar -zpcf /this_backup.tar.gz --exclude=/this_backup.tar.gz --exclude /etc/fstab --exclude /etc/mtab --exclude /etc/inittab --exclude=/lost+found --exclude=/mnt --exclude=/proc  --exclude=/sys --exclude /tmp /

2. Restore full image backup (booting to rescue mode should be safer)

tar xvpfz /this_backup.tar.gz -C /

Consider creating shell script with cron to automate the backup method and sending the backup files safely remotely using encrypted protocols such SFTP, SCP etc. You should consider also the possibility that if your server get owned then the attacker will have also access to your remote backup server. Restrict access to the remote server from deleting backups and dont use root accounts.

Tools: tar, scp, rsync, Baracula

Secure Shell [OpenSSH]

1. add/edit the following data to /etc/ssh/sshd_config

	Port 35823 					// Change default listen port
	Protocol 2						// Use only Protocol 2
	PrintLastLog yes				// Print last success logon
	PermitRootLogin no				// Disable root login
	banner /etc/issue.net				// Use warning banners
	IgnoreRhosts yes				// Ignore .rhost/.shosts files
	HostbasedAuthentication no			// Disable host based authentication
	AllowUsers/DenyUsers user1 user2	// Allow specific users to login

2. Use warning banners

mv /etc/motd /etc/motd.bak
echo "Unauthorized access..." > /etc/issue.net
cp /etc/issue.net /etc/motd

3. Consider using Public Key Authentication and then disabling Password Authentication
4. Consider using port-knocking with IPTables

Tools: fail2ban

Name Server [Bind]

1. add/edit the following data to /etc/bind/named.conf.options

allow-transfer {none;};			// Disable zone transfers
recursion no;					// Disable recursions
fetch-glue no;					// Disable glue fetching
version "Microsoft DNS";			// Change version banner

Database [MySQL]

1. Run mysql with less privileges (edit /etc/mysql/my.cnf)

	user=mysql

2. Restrict permissions to mysql data

	chown -R mysql /var/lib/mysql/
	chmod go-rwx /var/lib/mysql/

3. Delete unneccessary user accounts / sample databases

	mysql -u root -p
	mysql> use mysql;
	mysql> SELECT * FROM user;			// Review user accounts
	mysql> DELETE FROM user WHERE User=";// Delete unnecessary accounts
	mysql> FLUSH PRIVILEGES;
	mysql> DROP DATABASE sample_test;	// Delete unnecessary/sample databases

5. Change default root username

	mysql -u root -p
	mysql> UPDATE user SET user = "dbadmin" WHERE user = "root";
	mysql> FLUSH PRIVILEGES;

6. Use different user accounts for different applications, eg:

	mysql> CREATE DATABASE blog;
	mysql> CREATE USER 'blog_user'@'localhost' IDENTIFIED BY 'blog_password';
	mysql> GRANT ALL PRIVILEGES ON blog.* TO 'blog_user'@'localhost' WITH GRANT OPTION;
	mysql> FLUSH PRIVILEGES;

Tools: GreenSQL, securich plugin

Web Server [PHP]

1. add/edit the following data to /etc/php5/apache2/php.ini

	register_globals = Off
	allow_url_fopen=off			// Prevent (remote) file inclusions
	allow_url_include=off			// Prevent (remote) file inclusions
	open_basedir = /var/www/  // Prevent access to file/dirs (session.save_path should be change too)
	expose_php = Off			// Prevent version exposure
	display_errors = Off			// Prevent verbose error messages
	display_startup_errors = Off	// Prevent verbose error messages
	// disable dangerous function and classes
	disable_classes = splfileobject
	disable_functions = apache_get_modules, apache_get_version, apache_getenv, apache_note, apache_setenv, curl_exec, curl_multi_exec, disk_free_space, diskfreespace, dl, escapeshellcmd, exec, file_get_contents, highlight_file, ini_alter, ini_restore, mail, openlog, parse_ini_file, passthru, pcntl_exec, phpinfo, popen, proc_open, proc_nice, proc_close, proc_get_status, proc_terminate, putenv, shell_exec, show_source, symlink, system, virtual,  chgrp, chmod, chown, copy, link, mkdir, rename, rmdir, touch, unlink, syslog

Review PHP logs (error.log) from time to time to see if your web application need some of the above disabled functions.

Tools: PHP Security Audit

Web Server [Apache]

1. Review available modules and enable/disable necessary/unnecessary

ls -al /etc/apache2/mods-available/	// All available modules
ls -al /etc/apache2/mods-enabled/	// Enabled modules (.load + .conf)
apache2ctl -M			// View loaded modules
a2dismod <module>		// Disable module
a2enmod <module>		// Enable module
a2dismod autoindex		// Prevent directory listing

2. add/edit the following data to /etc/apache2/conf.d/security

# Prevent Server Header exposure
ServerTokens Prod
# Prevent Signatures exposure
ServerSignature off
# Prevent XSS attacks
TraceEnable off

3. Prevent DOS Attacks with mod_qos (such Slowloris)

3.1. apt-get install libapache2-mod-qos
3.2. paste the following data to /etc/apache2/mods-available/qos.load

		LoadModule qos_module /usr/lib/apache2/modules/mod_qos.so

3.2. paste the following data to /etc/apache2/mods-available/qos.conf

#Review below values based on your traffic
<IfModule mod_qos.c>
MaxClients 70
Timeout 20
KeepAlive on
MaxKeepAliveRequests 60
KeepAliveTimeout 3

QS_ClientEntries 10000
QS_SrvMaxConnPerIP 15
QS_SrvMaxConnClose 80%
QS_SrvMinDataRate 120 1500 400
</IfModule>

3.3. a2enmod qos

Modules: mod_limitipconn, mod qos, mod_evasive, mod_security, mod_noloris, mod_antiloris

4. Prevent Web attacks with Mod-Security2 with OWASP CRS Ruleset

4.1. Install OWASP CRS Ruleset (http://sourceforge.net/projects/mod-security/files/modsecurity-crs/0-CURRENT/)
4.2. wget <modsecurity-crs link .tar.gz>
4.3. cd modsecurity-crs_<version>
4.4. cp modsecurity_crs_10_config.conf.example modsecurity_crs_10_config.conf
4.5. cd ..
4.6. mov modsecurity-crs_<version> /etc/apache2/mods-available/crs

4.7. Install Mod-Security2
4.8. apt-get install libapache2-mod-security2
4.9. mkdir /var/log/modsecurity/
4.10. mkdir /var/log/modsecurity/data/
4.11. chown -R www-data /var/log/modsecurity/
4.12. paste the following data to /etc/apache2/mods-available/mod_security2.load

			LoadModule security2_module /usr/lib/apache2/modules/mod_security2.so

4.13. paste the following data to /etc/apache2/mods-available/mod_security2.conf

<IfModule security2_module>

# Enable the engine
SecRuleEngine On

SecAuditEngine RelevantOnly
SecAuditLogType Serial

# Set log files
SecAuditLog /var/log/modsecurity/modsec_audit.log
SecDebugLog /var/log/modsecurity/modsec_debug.log
SecUploadDir /var/log/modsecurity/data
SecAuditLogStorageDir /var/log/modsecurity/data
SecDataDir /var/log/modsecurity/data
SecTmpDir /var/log/modsecurity/data

# Change Server: Signature
SecServerSignature "Secure"

# Include OWASP CRS Ruleset
Include mods-available/crs/*.conf
Include mods-available/crs/base_rules/*.conf
</IfModule>

4.14. a2enmod mod_security2;a2enmod unique_id

Review your mod_security logs (/var/log/modsecurity/modsec_audit.log) from time to time to see if you drop valid request and you can remove specific security rules like this:

<LocationMatch "/admin/trust_script.php">
SecRuleRemoveById <rule_id1>
SecRuleRemoveById <rule_id2>
...
</LocationMatch>

Firewall [IPTables]

1. Seperate IP Tables log files
11. Create /etc/rsyslog.d/40-iptables.conf with the following data

:msg, contains, "FRW-" -/var/log/iptables.log
&~

This will redirect all log messages containing FRW- to /var/log/iptables.log file

2. Create firewall On / Off scripts to control your firewall
2.1. Firewall Off (fw_off.sh)

iptables -F
iptables -X
iptables -Z
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

2.2. Firewall On (fw_on.sh)

- Default policy DROP ALL
- Allow TRUSTED IP ADDRESSES
- INVALID IP ADDRESSES
- INVALID PORTS (PORT SCAN IDENTIFICATION)
- INVALID TCP FLAGS
- Denial of Service / Brute force / Syn flood / Web Attacks
- Allow access to external running services
- Allow internet access from your system

Review your IP Tables rules and configure it based on your traffic and running services

3. Create Ban / UnBan Scripts
3.1. Ban Script (fw_ban_ip.sh)

- Add IP in ban list
- Ban IP with IP Tables
- Keep logs

3.2 UnBan Script (fw_unban_ip.sh)

- Remove IP form ban list
- Unban IP with IP Tables
- Keep logs

4. Create firewall IPS script to automatically ban on specific attacks (fw_ips.sh)

- BAN on port scanning (X ports within Y seconds)
- BAN on X IP Tables records within Y seconds
- BAN on X Mod Security records within Y seconds
- BAN on Z (X1 IP Tables + X2 Mod Security) records within Y seconds

You could create custom more specific ban rules for specific attacks, by changing the initial script, such as:
- Ban on 1 IP Tables ‘DOS/BF_SSH’ record with 3600 seconds
- Ban on 2 Mod Security ‘SQL Injection’ records within 300 seconds

Download Firewall Scripts
fw_off.sh
fw_on.sh
fw_ban_ip.sh
fw_unban_ip.sh
fw_ips.sh

Tools: iptables, snort, apf, ufw

Monitoring

Finally, some commands to monitor your processes/services/accounts:

1. ps -auxw / pstree -Alph / top
2. netstat -apn / ss -an / lsof -i -U
3. w / last /lastlog

tools: tcpdump, iptraf, cacti

Related

Securing & Hardening Linux Web Server (Part I – Theory)

Share

Leave a Reply

Photo

root

July 7th


Papers


line

© 2017 SuRGeoNix | Security Blog