Security Blanket Technical Blog

MAC versus DAC in SELinux

2011-09-23T12:58:00.000-04:00

Some system administrators do not understand Mandatory Access Control (MAC) and how it interacts with Discretionary Access Control (DAC) in Linux®. In a previous post, I stated Security-Enhanced Linux (SELinux) employs MAC rules to facilitate fine-grained security. I also discussed some of the collection of rules which form standard SELinux policies such as Targeted and Strict.

I received some emails from readers which said they weren't clear on how MAC and DAC work together in the operating system. Since I am a hands-on guy, I love to see real-world examples, so that's what I am going to show you.

I have configured a standard CentOS 5.6 Linux system with default Apache web server packages installed. The sestatus(8) command reports the operating system is enforcing the default SELinux Targeted policy.

# /usr/sbin/sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 21
Policy from config file:        targeted

Using the -Z argument to the ps(1) command shows the Apache daemon (httpd) is executing in the (httpd_t) domain:

# /bin/ps -aefZ |/usr/bin/grep httpd
root:system_r:httpd_t           root     24855     1  0 10:23 ?        00:00:00 /usr/sbin/httpd
root:system_r:httpd_t           apache   24856 24855  0 10:23 ?        00:00:00 /usr/sbin/httpd
root:system_r:httpd_t           apache   24857 24855  0 10:23 ?        00:00:00 /usr/sbin/httpd
root:system_r:httpd_t           apache   24858 24855  0 10:23 ?        00:00:00 /usr/sbin/httpd
root:system_r:httpd_t           apache   24859 24855  0 10:23 ?        00:00:00 /usr/sbin/httpd
root:system_r:httpd_t           apache   24860 24855  0 10:23 ?        00:00:00 /usr/sbin/httpd
root:system_r:httpd_t           apache   24862 24855  0 10:23 ?        00:00:00 /usr/sbin/httpd
root:system_r:httpd_t           apache   24863 24855  0 10:23 ?        00:00:00 /usr/sbin/httpd
root:system_r:httpd_t           apache   24864 24855  0 10:23 ?        00:00:00 /usr/sbin/httpd

I will use the curl(1) command to retrieve a URL from the Apache web server — this is equivalent to a web browser accessing the URL. By default, the provided Apache packages establishes the document root as /var/www/html. So, we will create a test.html file in the document root with a permissive DAC and a MAC which the httpd_t domain is not allowed to access.

# cd /var/www/html
# /bin/touch test.html
# /usr/bin/chcon system_u:object_r:shadow_t test.html 
# /bin/ls -lZ test.html
-rw-r--r--  root root system_u:object_r:shadow_t         test.html

I have temporarily set the SELinux context of this file to be of shadow_t type using the chcon(1) command. This restrictive type is set on the /etc/shadow system file – a web server would have no reason to access a file of this type. Using the -Z option to the ls(1) command you can see the type is now shadow_t in the SELinux context.

Note the DAC mode (-rw-r--r--) of the file permits the owner to read and write (rw), the group to read (r), and everybody else [other] is able to read (r) the file. However, when I try to retrieve the file the MAC rule in the SELinux policy FORBIDS the process running in the httpd_t domain from accessing files of type shadow_t — REGARDLESS of the permissive DAC mode:

# /usr/bin/curl -I http://localhost/test.html
HTTP/1.1 403 Forbidden
Date: Fri, 23 Sep 2011 14:48:51 GMT
Server: Apache/2.2.3 (CentOS)
Connection: close
Content-Type: text/html; charset=iso-8859-1

The SELinux policy also defines default SELinux contexts for directories and file patterns. When the system is rebooted or the restorecon(8) utility is called, file system objects are labeled according to the policy. I will use the restorecon utility to restore (set) the SELinux context on our test file appropriately:

# /sbin/restorecon test.html
# /bin/ls -Z test.html
-rw-r--r--  root root system_u:object_r:httpd_sys_content_t test.html

In order for the Apache web server running in the httpd_t domain to read files in its document root (/var/www/html), the files should be of http_sys_content_t type.

Now, I am able to access the file:

# /usr/bin/curl -I http://localhost/test.html
HTTP/1.1 200 OK
Date: Fri, 23 Sep 2011 14:50:16 GMT
Server: Apache/2.2.3 (CentOS)
Last-Modified: Fri, 23 Sep 2011 14:40:32 GMT
ETag: "15aac8-6-ca935800"
Accept-Ranges: bytes
Content-Length: 6
Connection: close
Content-Type: text/html; charset=UTF-8

This simple real-world example demonstrates how MAC rules supersede DAC settings. I encourage you to read the system documentation and experiment on lab systems. Too often system administrators become frustrated by “AVC Denial” messages and resort to disabling this enhanced security.

Security Blanket Supports SELinux

2011-09-21T12:54:00.002-04:00

Security-Enhanced Linux (SELinux) is an enhancement to the standard Linux® kernel that provides fine-grained security by employing Mandatory Access Control (MAC) rules. Security Blanket® v4.0.7 now supports Red Hat® Enterprise Linux 4, 5, and 6 enforcing the default Targeted SELinux policy – as well as Fedora™ 10 through 13.

The aim of the Targeted policy is to provide additional security to some of the more commonly used daemons such as httpd, dhcpd, mailman, named, portmap, nscd, ntpd, portmap, mysqld, postgres, squid, syslogd, winbind, and ypbind by employing MAC rules.

For example, the Apache Web Server (httpd) daemon executes in its own domain httpd_t. Other daemons on the system which do not have policy written specifically for them run in the domain unconfined_t.

Daemons and system processes running in the unconfined_t domain only use the standard Linux Discretionary Access Control (DAC) method of access control. In SELinux, access is granted to processes on a per-domain basis; each domain has a set of operations it may perform on each type of file, directory, or other resource.

For security reasons, the Security Blanket team preferred not to execute in the unconfined_t domain. Therefore, a specific policy module was written to augment the Targeted policy, which separated Security Blanket's Console, Dispatcher, and Core Engine components into their own domains. For more details and exceptions, please see the Security Blanket Administration Guide.

Processes and files are labeled with an SELinux Context that contains additional information, such as an SELinux user, role, type, and, optionally, a security level. When running SELinux, all of this information is used to make access control decisions. In Red Hat Enterprise Linux, SELinux provides a combination of Role-Based Access Control (RBAC), Type Enforcement® (TE), and, optionally, Multi-Level Security (MLS).

The above image is the output from the ls(1) command using the -Z argument , which displays the SELinux Context assigned to a file object.

In previous releases of Security Blanket, SELinux was not supported because the SELinux Context on file system objects could be destroyed and could only be restored by relabeling the object. Each file system object is referenced by its information node (inode) and the SELinux context is stored as an extended attribute.

Some Security Blanket modules created new files or worked with temporary copies of configuration files — then subsequently copied it to their final location. In these situations, a new information node was assigned. Several modifications to the Security Blanket Core Engine and associated modules were made to restore the SELinux context on such file system objects.

In future releases of Security Blanket, we may provide support for the Strict policy with MLS. The goal of MLS policy is to allow a Linux operating system to get EAL4+/LSPP certification. In developing this policy, the fourth field of the security context, the security or sensitivity level has been turned on — this facilitates the handling of labeled files.

Furthermore, the MLS policy contains rules that not only govern what security types are able to do, but also what they can do when running at a particular security level. In MLS there are two components of the Security Level, the sensitivity level, which can go from s0–s15, and the capabilities, which can go from c0–c255. The Multi Category System (MCS) policy was also added to the Targeted and Strict policies, which confines the sensitivity level to s0 but permits user defined capabilities.

The Security Blanket team is also watching the National Security Agency's (NSA) Certifiable Linux Integration Platform (CLIP) project. This project defines a specific configuration of SELinux designed to provide the foundation for hosting secure applications.

The Security Blanket team is excited to offer SELinux Targeted policy support and we are anxious to hear from our customers. Finally, if you are interested in using Security Blanket on systems enforcing the Strict policy or you have plans to use configurations defined by CLIP, send us an email at SecurityBlanket@TrustedCS.com.

For more information see: Taking advantage of SELinux in Red Hat® Enterprise Linux®, by Faye Coker and Russell Coker

Red Hat 5 STIG: Kernel Modules

2011-08-25T13:10:00.009-04:00

In the last few years, I've seen organizations extending the concept of least privilege to least installed or running. I have written about minimizing a system's attack surface by removing unnecessary software and turning off as many unused features as possible.

The draft release of the U.S. Defense Information Systems Agency's (DISA) “Red Hat 5 STIG” is no exception. I love it when security guidelines make these suggestions but it can be frustrating when assessment scanners report false-positives. For example, some scanners will report a failure if it can't find a setting in a configuration file for software which isn't even installed on the system.

The new draft STIG requires entries in a configuration file to prevent the kernel from loading modules – even if the modules aren't installed on the system. Nonetheless, I have compiled a list of the required settings which must be set in your modprobe.conf configuration file.

First of all, the STIG requires the loading and removing of kernel modules be recorded by the auditing subsystem (GEN002825). The system calls (-S) init_module and delete_module are tracked and watches (-w) are placed on some command line utilities. Add the following rules to your audit.rules file:

-a always,exit -S init_module -S delete_module -k modules 
-w /sbin/insmod -p x -k modules 
-w /sbin/modprobe -p x -k modules 
-w /sbin/rmmod -p x -k modules

Unless your system has a specific need for the following networking related modules, they shouldn't be loaded into the kernel. These modules aren't installed in a default Red Hat Enterprise Linux installation but you'll still need to implicitly add entries in modprobe.conf file:

Network Bridging (GEN003619)
Stream Control Transmission Protocol (SCTP) (GEN007020)
Datagram Congestion Control Protocol (DCCP) (GEN007080)
Reliable Datagram Sockets (RDS) (GEN007480). Some implementations of Oracle Real Application Clusters (RAC) uses RDS as an interconnect.

If you don't use IPv6, it is recommended you disable support for it (GEN007700) and you should configure the system to prevent dynamic loading of the IPv6 protocol handler (GEN007720).

Additionally, the guideline recommends disabling support for Bluetooth, USB storage devices, and Firewire:

Transparent Inter-Process Communication (TIPC) (GEN007540). TIPC was specially designed for intra-cluster communication and products like Wind River's VxWorks use it.
Bluetooth (GEN007660)
USB Mass Storage (GEN008480)
IEEE 1394 (Firewire) (GEN008500)

To prevent the loading of the aforementioned modules and network modules, add the following to your /etc/modprobe.conf file:

# Network related
install bridge /bin/false
install sctp /bin/true
install dccp /bin/true
install dccp_ipv4 /bin/true
install dccp_ipv6 /bin/true 
install rds /bin/true

install tipc /bin/true
install bluetooth /bin/true

# IPv6
alias net-pf-10 off 
alias ipv6 off 
install ipv6 /bin/true

# USB and Firewire
install usb-storage /bin/true
install ieee1394 /bin/true

Section 2.2.2.5 of the National Security Agency's “Guide to the Secure Configuration of Red Hat Enterprise Linux 5” also recommends not loading kernel modules for uncommon filesystem types:

install cramfs /bin/true
install freevxfs /bin/true
install jffs2 /bin/true
install hfs /bin/true
install hfsplus /bin/true
install squashfs /bin/true
install udf /bin/true

My suggestion is always remove unused software, however many auditors will probably still want to see the implicit setting in the modprobe.conf file.

Red Hat 5 STIG: Network Settings

2011-08-22T11:31:00.024-04:00

The draft release of the U.S. Defense Information Systems Agency's (DISA) “Red Hat 5 STIG” earlier this year has a few system administrators panicking. For Red Hat® Enterprise Linux® 5 administrators, this Security Technical Implementation Guide (STIG) has supplanted the generic UNIX STIG.

The generic UNIX STIG had the single potential discrepancy indicator (PDI) “GEN003600 - Network Security Settings.” The checklist document required you to check four network settings in the running kernel. The new Red Hat 5 STIG, however, has many more settings and provides better explanations.

I would caution administrators from rushing to add all of these settings to your systems because most of these settings are defaults. Nonetheless, the settings must be implicitly set in the sysctl.conf configuration file. My recommendation is to review the entire STIG in order to define a complete sysctl.conf file, so that it can be deployed and tested all at once.

To help get you started with the new Red Hat 5 STIG, I have compiled a list of settings from the guideline as well as a few from the National Security Agency's “Guide to the Secure Configuration of Red Hat Enterprise Linux 5.”

First of all, the ability to configure network interfaces should be limited to privileged users (GEN003581). This is achieved by setting “USERCTL” to “no” in the /etc/sysconfig/network-scripts/ifcfg* files.

The NSA configuration guide recommends disabling zero configuration networking (zeroconf); which is a set of techniques that automatically creates a usable Internet Protocol (IP) network without manual operator intervention or special configuration servers. To disable zeroconf, set “NOZEROCONF” to “yes” in the /etc/sysconfig/network file.

As before, IP forwarding [ip_forward] should be disabled (GEN005600) and the new guideline recommends not forwarding (GEN003600) or accepting (GEN003607) source-routed packets [accept_source_route] either.

The system must not respond to broadcast Internet Control Message Protocol (ICMP) echoes (GEN003603) or timestamp requests (GEN003604) [icmp_echo_ignore_braodcasts].

Furthermore, the system must ignore ICMP redirect messages (GEN003609) [accept_redirects] as well as not send ICMP redirects (GEN003610) [send_redirects]. Other guidelines recommend not accepting secure ICMP redirects [secure_redirects]. The secure redirect message is sent by a gateway that appears in the host's default gateway list.

To provide some mitigation to TCP denial of service attacks the guideline (GEN003601) recommends adjusting the TCP backlog queue size [tcp_max_syn_backlog]. Additionally (GEN003612), your system should be configured to send out requests [tcp_syncookies] to remote hosts if they are flooding your system’s backlog queue with SYN packets. These requests check whether or not the inbound SYN packets are legitimate.

Enabling TCP syncookies option on a system under normal load is useful. If your system is under high load it will make new connections but without advanced features such as explicit congestion notification (ECN) or selective acknowledgment (SACK).

To help mitigate the leaking of addressing information between attached network segments, the guideline (GEN003608) recommends disabling proxy Address Resolution Protocol (ARP) [proxy_arp].

The guideline (GEN003613) recommends the system perform source validation by reversed path [rp_filter]. When you enable reverse path source validation, inbound packets are dropped if the IP address from where the packets were received is not reachable (i.e., asymmetrical route).

It should be noted, however, that enabling this may cause problems in complex networks running a slow and unreliable protocol, using static routes, or where asymmetric routes are present. Asymmetric routes are not common, but may be necessary in certain cases. By default, Linux drops packets in which asymmetric routes are used because of the security risk.^[1]

It is also recommended (GEN003619) network bridging be disabled. This is usually a kernel module which can be checked with lsmod(8) command utility. To prevent it from loading, add the appropriate line for that module in the modprobe.conf file.

Finally, the guideline (GEN003611) recommends the kernel log all martian packets [log_martians]. Martian packets are packets which contain addresses known by the system to be invalid.

As I said earlier, many of these settings are kernel defaults. However, it is best to add the following settings into your /etc/sysctl.conf file:

net.ipv4.ip_forward = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.tcp_max_syn_backlog = 1280
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

Now, you can execute /sbin/sysctl -p as root or reboot the system.

[1] Benvenuti, Christian. Understanding Linux Network Internals, Chapter 31. Sebastopol, CA: O'Reilly Media, Inc., 2006.

Getting started with the new Red Hat 5 STIG

2011-08-17T09:13:00.024-04:00

I am pleased the U.S. Defense Information Systems Agency (DISA) released a draft of “Red Hat 5 STIG” earlier this year. As expected, there are some specific configuration guidelines for Red Hat® Enterprise Linux® 5 which were missing from the generic UNIX Security Technical Implementation Guide (STIG).

The generic UNIX STIG supported numerous UNIX® and Linux® distributions but never addressed Red Hat Enterprise Linux 5. For many years, this lack of support was a source of frustration for system administrators. In February 2011, I discussed the new guidelines in the article “DISA UNIX STIG for Red Hat Enterprise Linux 5 and 6.”

I am reviewing the “Red Hat 5 STIG” to develop a plan to add support for it to our Security Blanket® product. I've completed my initial review and I want to share some of my findings along with some tips.

First of all, the “Red Hat 5 STIG” is not to be used in conjunction with the generic UNIX STIG. Download the ZIP file, extract it, and transform the XML document to a more usable format. I used the XSLT processor which comes with Red Hat as follows:

/usr/bin/xsltproc STIG_unclass.xsl U_RedHat_5_V1R0_STIG_Manual-xccdf.xml > stig.html

The previous command will transform the document from XML format to HTML, so you can open it with your web browser. I like to have a delimited text list of the guideline, so I created the following simple XSLT:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:cdf="http://checklists.nist.gov/xccdf/1.1">
 <xsl:output method="text" />
 <xsl:template match="/">
  <xsl:for-each select="cdf:Benchmark/cdf:Group/cdf:Rule">
      <xsl:apply-templates select="." />
  </xsl:for-each>
 </xsl:template>

 <xsl:template match="cdf:Rule">
  <xsl:value-of select="@id" />
  <xsl:text>|</xsl:text>
  <xsl:value-of select="@severity" />
  <xsl:text>|</xsl:text>
  <xsl:value-of select="cdf:version" />
  <xsl:text>|</xsl:text>
  <xsl:value-of select="cdf:title" />
  <xsl:text>&#x0A;</xsl:text>
 </xsl:template>
</xsl:stylesheet>

Save the above as a file and use it instead of the “STIG_unclass.xsl” file.

This STIG, like the generic UNIX one, requires you to have a Disaster Recovery Plan or some form of Business Continuity Plan (BCP). The plan must contain detailed procedures for performing backups and recovery.

Wherever possible, try to have on-line, off-line, and off-site backups. On-line could include storage mirroring technologies and off-line could be some sort of media which isn't connected to equipment. If at all possible, perform encrypted backups and then store them in a media safe on-site and off-site. The off-site backups could be as simple as taking the off-line media outside of your data center and preferably not in the same building.

Account management and personnel security practices must be clearly documented. This includes documenting how access is granted to personnel. Procedures on removing accounts or revoking rights must also be documented. This also means taking a serious look at shared application and administrative accounts. Stop logging directly into these accounts (especially root!) — personnel should be logging into their own personal account then switching to the application/administrative account.

The new “Red Hat 5 STIG” has 596 potential discrepancy indicators (PDI) – sometimes called “STIG items.” Of these, 62 of them are related to extended access control lists (ACL), 8 to IPv6, and 14 to FIPS 140-2.

In the generic UNIX STIG, many STIG items restrict standard user-group-other permissions. These Discretionary Access Controls (DAC) are a fundamental security mechanism and POSIX.1e Access Control Lists (ACL) provide administrators the ability to grant access to additional users and groups without having to grant access to other (a.k.a, the world).

Nonetheless, there are 62 new STIG items which recommend stripping ACLs from files and directories. By default, file systems such as ext2 and newer support POSIX.1e ACL. If none of your applications require POSIX ACLs, I recommend mounting file systems with the 'noacl' option to disable support for them. This is an easy way to address all 62 items.

As for IPv6, my opinion is that if you are not using IPv6 simply disable support for it. First edit the /etc/sysconfig/network and set “NETWORKING_IPV6” to “no”. Next, add the following two lines to /etc/modprobe.conf:

alias net-pf-10 off
alias ipv6 off

Lastly, stop the IPv6 firewall service and configure it to not start when the system is rebooted:

# /sbin/service ip6tables stop
# /sbin/chkconfig ip6tables off

Fourteen of the new items require the “...[use of] a FIPS 140-2 validated cryptographic module (operating in FIPS mode).” These items, however, don't give the detailed check procedures as many of the other items do. Instead, DISA states, “The NIST CVMP web site provides a list of validated modules and the required security policies for the compliant use of such modules. Verify that the module is on this list and configured in accordance with the validated security policy.”

To make Red Hat Enterprise Linux 5 compliant with the Federal Information Processing Standard (FIPS) Publication 140-2 you need to make several changes to ensure that accredited cryptographic modules are used. For detailed instructions, download DOC-3923 from the Red Hat Knowledgebase.

Ensure the following accredited packages are available:

        kernel  2.6.18-164.2.1.el5
     libgcrypt	1.4.4-5.el5
       openssl  0.9.8e-12.el5
      openswan	2.6.21-5.el5_4.3
           nss  3.12.6-2.el5_4
selinux-policy	2.4.6-255.el5_4.2
 fipscheck-lib	1.2.0-1.el5

Next, disable prelinking by editing /etc/sysconfig/prelink and setting “PRELINKING” to “no”. Previous prelinks should be undone with:

# /usr/sbin/prelink -u -a

You'll need to recreate the initial RAM disk for x86_64 based platforms as follows:

# /sbin/mkinitrd --with-fips -f /boot/initrd-$(uname -r).img $(uname -r)

and for IA64 based platforms as follows:

# /sbin/mkinitrd --with-fips -f /boot/efi/efi/redhat/initrd-$(uname -r).img $(uname -r)

Finally, add “fips=1” as a kernel boot parameter. It's best to do this by editing your grub.conf file.

Once the operating system is operating in “FIPS” mode, you can proceed to address items such as GEN005490 – which requires the Secure Shell Daemon be configured to use the accredited modules. I will cover these specific items in some later posts.

Overall, I am impressed with the “Red Hat 5 STIG” and will be sharing tidbits over the next few months. The Security Blanket development team is working hard to automate the assessment and configuration of your systems to these new guidelines.

Sysadmin Basics: Account Management Is Critical

2011-07-28T15:47:00.008-04:00

These days I believe many inexperienced system administrators don't focus as much as they should on the fundamentals—such as user account management. Many system administrators are distracted by the urge to patch their systems to address vulnerabilities they aren't susceptible to or to configure their systems with nifty tricks they found on a “tips-and-tricks” or “how-to” website. This urge, I believe, is rooted in their belief they are a warrior combating the forces of evil in this cyber war.

When a young, inexperienced administrator reports they've “plugged” a major hole in their defenses to their boss, the administrator feels like an important asset. If they quote details from the vulnerability report including the CVSS base score, impact subscore, and exploitability subscore their boss must think they are a genius.

In Peter Bright's article: “Anonymous speaks: the inside story of the HBGary hack”, he states:

The Anonymous hack was not exceptional: the hackers used standard, widely known techniques to break into systems, find as much information as possible, and use that information to compromise further systems. They didn't have to, for example, use any non-public vulnerabilities or perform any carefully targeted social engineering.

I would never discourage organizations from patching their systems—when it is necessary. Often times, organizations waste time patching unused services which should be removed from the system altogether. Organizations may patch software to address a vulnerability in a feature which isn't even being used.

Several years ago, I had a young, enthusiastic, inexperienced system administrator working for me. He loved reading these vulnerability reports and he found the inner workings of the system fascinating. His curiosity and his affinity for the hacker subculture was a distraction from his daily responsibilities.

I remember him passionately arguing on two occasions that we must patch our systems immediately. The first was a flaw in OpenSSL's SSL/TLS handshaking code when using Kerberos ciphersuites and the second, a vulnerability in the Apache module which provides Web-based Distributed Authoring and Versioning (WebDAV).

He calmed down after I explained to him that none of our applications use Kerberos ciphersuites and the Apache module isn't statically linked to the daemon nor is it loaded as a dynamic shared object. I urged him to slow down and think about what he was doing. I did not, however, want to discourage him from learning new things but I needed him to focus on the fundamentals.

Account management is a discipline often overlooked because it is mundane and boring. Neglected, unused accounts are a front door for attackers and often used to escalate privileges or worse, extend the attacker's reach into your enterprise.

I am not a fan of the “Top X Tips to Y” articles but many people prefer this abridged list format due to their busy schedules. Nonetheless, here are some key aspects of account management which I feel are overlooked:

Types of accounts: Are they system or application accounts? What authentication realm do they belong? Are they internal to an application, database, centralized authentication, or simply local accounts managed by the operating system?
Account ownership: Who owns the account? If it is an application account responsible for providing a service, who is the primary point of contact? Can anyone directly login to the account?

When was the last time the account was accessed? Is it obsolete? If it is a user account, when was the last time it was accessed. In my opinion, when in doubt about a user account—lock it. If it is important enough and they need it, they will contact you.
Password policy: Have a good password policy, implement account aging mechanisms, and change ALL default passwords. In Linux, try to avoid setting a user account's password as root without forcing the user to change it at their next login.

A common mistake made by system administrators is to login as root, issue the command to change a user account's password but allow the user to enter a new password. When you set a password on an account as root, on most Linux systems the password complexity rules are bypassed allowing the user to choose a weak password.

I only covered the basics. It is critical that system administrators not neglect these not-so-glamorous activities. This is part of a system administrator's daily grind. Don't be afraid to question why someone has a user account on your system—you're responsible.

What is a kernel level audit trail?

2011-07-11T12:39:00.022-04:00

Many security experts believe that audit trails and logs are the key to identifying an ongoing cyber attack as well as providing forensic data following an attack.

However, few people understand how audit records are generated or the difference between a kernel level audit trail and an application event log.

It is critical to configure auditing and logging mechanisms to capture the right data in an optimal manner and to safeguard the data to prevent it from being modified.

Like most operating systems, the main component of Linux® is its kernel. It acts as a bridge between software applications and hardware and as such is responsible for managing the system's resources. Generically speaking, these resources include the central processing units (CPU), memory, and any Input/Output (I/O) devices present in the computer.^{[Wikipedia:Kernel]}

Running software applications are considered processes by the kernel. Generally, a run-time library sits between applications and the operating system such as libc and glibc. The application calls functions in these libraries — which act as wrappers to system calls. Ultimately, a system call (syscall) is how a process requests a service from the kernel that it does not normally have permission to run.

When we talk about kernel level audit trails, we are referring to records of system calls and file system watches. The kernel has the ability to intercept every system call (see linux/kernel/auditsc.c) and optionally record them to a file system using an auditing daemon (auditd). This auditing daemon can be configured with a set of rules on specifically what to record.

There are plenty of examples on the Internet of how to implement auditing rules but I want to show you a simple example:

# /sbin/auditctl -w /etc/hosts -p rwa 
# /sbin/auditctl -l
LIST_RULES: exit,always watch=/etc/hosts perm=rwa 


# /bin/grep test /etc/hosts
# /sbin/ausearch -f /etc/hosts

time->Mon Jul 11 09:53:28 2011

type=PATH msg=audit(1310392408.506:36): item=0 name="/etc/hosts" 
inode=4197780 dev=08:02 mode=0100644 ouid=0 ogid=0 rdev=00:00

type=CWD msg=audit(1310392408.506:36):  cwd="/var/log/audit"

type=SYSCALL msg=audit(1310392408.506:36): arch=c000003e 
syscall=2 success=yes exit=3 a0=7fff2ce9471d a1=0 a2=61f768 
a3=7fff2ce92a20 items=1 ppid=20478 pid=21013 auid=1000 uid=0 
gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 
ses=1 comm="grep" exe="/bin/grep"

In the above example, I set an audit rule to watch the /etc/hosts file. Then I used the grep utility to search the contents of the file for the word “test.” Finally, I searched the audit trail for entries pertaining to /etc/hosts. As you can see, it creates quite a detailed audit record.

Many security guidelines recommend that you establish watches on administration utilities. Two such command line utilities, /bin/chmod and /bin/chown are used to change a file's mode and ownership respectively. I find it more beneficial to establish a rule to monitor the underlying system calls: chmod and chown. Keeping in mind, the aforementioned utilities use the system calls: chmod and chown. Since other utilities and programs can make similar calls, it is imperative we capture the system calls themselves rather than just the utilities.

Some guidelines go overboard in my opinion. They want you to establish a rule to record every successful file opening. Here is an example rule:

/sbin/auditctl -a exit,always -F arch=b64 -S open -F success=0

The problem I have with this rule is that the audit daemon now records every file that is opened including every shared library than an application is dynamically linked to.

When it comes to application event logs, they tend to be much more useful. If programmed correctly, applications can produce meaningful event logs which tend to be mapped to business process flow.

For example, if a Java-based web application is running on a Apache Tomcat servlet container, the process is usually owned by a non-privileged user account. Thus kernel level audit trails of the Tomcat process accessing report files can be ambiguous. On the other hand, Java application's event logs are far more useful if they record the identification and authentication of an internal user, the user's request to view the report, and then the report delivered to the user.

In summary, there are a couple of things to remember when configuring system auditing:

First and foremost, you must balance what you need to collect with your ability to collect it. In other words, ensure you have the system resources (storage and CPU power) to generate the required audit trail without adversely impacting application performance.
Consolidate your rules where possible. The system call rules are loaded into a matching engine that intercepts each syscall that all programs on the system makes. Therefore it is very important to only use syscall rules when you have to since these affect performance. The more rules, the bigger the performance hit. You can help the performance, though, by combining syscalls into one rule whenever possible.^{[auditctl(8)]}
Ensure applications are configured to record the necessary events. Additionally, be sure that in-house developed applications provide plenty of details in their event logs.
Protect audit trails and event logs. It is imperative that audit records are not sent over unsecure channels, unauthorized users are denied access, and audits can not be modified.

Protecting Linux Against DoS/DDoS Attacks

2011-06-27T10:55:00.025-04:00

When I first heard ridiculous-sounding terms like smurf attack, fraggle attack, Tribal Flood Network (TFN), Trinoo, TFN2K, and stacheldraht, I didn't take them too seriously for a couple of reasons — I worked mainly on non-Internet facing systems and I was never a victim. I thought it was primarily a network or application administrator's problem.

I am not too proud to admit that I was completely wrong. The truth is that I only had a grasp of the impact of such attacks but I didn't know anything about the methods and the things that can and should be done at the operating system level.

I have been neck-deep in completing documentation for our product's Common Criteria for Information Technology Security Evaluation (ISO/IEC 15408) submission so I hadn't had much time to think about topics for a blog post. Besides, how can I compete with all of these eye-catching, dramatic headlines about LulzSec, Anonymous, and Ryan Cleary?

A co-worker asked me how our Security Blanket® operating system lock down tool could help against denial-of-service (DoS) attacks. So began my research and I quickly had the epiphany that I barely knew anything about DoS attacks.

Of course this topic is far too broad and complex to cover in one blog post but I am going to highlight some of my findings. First of all, I strongly recommend visiting the SANS Institute InfoSec Reading Room and reading “A Summary of DoS/DDoS Prevention, Monitoring and Mitigation Techniques in a Service Provider Environment.” Secondly, read the W3C's “The World Wide Web Security FAQ - Securing against Denial of Service attacks.”

In “HACKING the art of exploitation”¹, Erikson describes two general forms of DoS attacks: those that crash services and those that flood services. Wikipedia goes on to describe five basic types of attacks:

Consumption of computational resources, such as bandwidth, disk space, or processor time.
Disruption of configuration information, such as routing information.
Disruption of state information, such as unsolicited resetting of TCP sessions.
Disruption of physical network components.
Obstructing the communication media between the intended users and the victim so that they can no longer communicate adequately.

The W3C defines DoS as “an attack designed to render a computer or network incapable of providing normal services. The most common DoS attacks will target the computer's network bandwidth or connectivity. Bandwidth attacks flood the network with such a high volume of traffic, that all available network resources are consumed and legitimate user requests can not get through. Connectivity attacks flood a computer with such a high volume of connection requests, that all available operating system resources are consumed, and the computer can no longer process legitimate user requests.”

The W3C differentiates a DoS attack from a Distributed Denial of Service (DDoS) attack. The DDoS “attack uses many computers to launch a coordinated DoS attack against one or more targets. Using client/server technology, the perpetrator is able to multiply the effectiveness of the Denial of Service significantly by harnessing the resources of multiple unwitting accomplice computers which serve as attack platforms.”

In the case of smurf and fraggle attacks, one method of prevention is to configure the router to block broadcast packets that did not originate from that network. On Linux systems, you can configure the kernel to disregard ICMP ECHO and TIMESTAMP requests that were sent to broadcast or multicast addresses by setting the kernel parameter net.ipv4.icmp_echo_ignore_broadcasts to one.

When it comes to “SYN flood” DoS form of attacks, you can configure Linux to send out requests (syncookies) to remote hosts if they are flooding your system’s backlog queue with SYN packets; to enable this set the kernel parameter net.ipv4.tcp_syncookies to one.

These requests (syncookies) check whether or not the inbound SYN packets are legitimate. In cases where these inbound SYN packets are not legitimate, your system might be experiencing a “SYN flood” DoS attack. Enabling this option on a system under normal load is useful. If your system is under high load it will make new connections but without advanced features such as explicit congestion notification (ECN) or selective acknowledgment (SACK).

All of the normal hardening procedures for the operating system will of course help. Namely, it will help reduce the likelihood your system will become compromised and become the platform for which attacks will be launched.

Additionally, it is critical to know what software is present on your system. One technique to monitor this is to baseline (or fingerprint) your system to include the use of cryptographic hashes where possible. Then periodically, perform another baseline and compare it to the previous one.

The use of host-based firewalls (i.e., iptables) is strongly encouraged as well as disabling of unnecessary server services. System minimization has been a topic in many of my posts before and I believe it is one of the easiest but most effective techniques because it reduces your “attack surface.” The W3C FAQ also says, “assume a service should be turned off, unless it is absolutely required.” And I would take it one step further by removing the software packages associated with those unused services.

Safeguarding and monitoring operating systems against DoS and DDoS are areas which I continue to learn about and develop techniques. Please, share your knowledge and techniques so we all might learn.

[1] Erikson, Jon (2008). HACKING the art of exploitation (2nd edition ed.). San Francisco: No Starch Press. p. 251. ISBN 1-59327-144-1.

Security Blanket SELinux Policy Module

2011-06-08T17:26:00.009-04:00

The next release of Security Blanket will support Security-Enhanced Linux (SELinux). Previous releases of Security Blanket would only work if SELinux was enabled but was operating in permissive mode. SELinux has been growing in popularity for many reasons and is no longer used by just government organizations requiring mandatory access control (MAC) and multilevel security (MLS) systems.

Among other things, domain-types and data confidentiality help protect a system's integrity. In particular, Red Hat® (and its clones) have been delivering the “targeted” policy. According to the documentation, this policy is defined as follows:

Under the targeted policy, every subject and object runs in the unconfined_t domain except for the specific targeted daemons. The objects on the system that are in the unconfined_t domain are allowed by SELinux to have no restrictions and fall back to using standard Linux security, that is, DAC. This policy is flexible enough to fit into enterprise infrastructures. The daemons that are part of the targeted policy run in their own domains and are restricted in every operation they perform on the system. This way daemons that are broken or exploited are limited in the damage they can do.

The targeted policy has grown to include domain-types such as httpd_t and samba_t to isolate Apache web services and Samba file sharing services respectively.

Earlier versions of Security Blanket just ran in the unconfined_t domain. However, if Security Blanket creates new configuration files for the operating system, in addition to setting the appropriate discretionary access controls (DAC), Security Blanket must ensure the security context is set. In addition to writing the appropriate rules in the policy module to supplement the “targeted” policy, we have to ensure that all security contexts are maintained. We certainly didn't want to just tell our customers, “Just reboot and relabel.”

In addition to code modifications, we have defined three specific domain-types for the following Security Blanket components: the console, the dispatcher, and the core engine. These independent domains will offer better isolation as well as tighten access controls on auditing and reporting data produced by Security Blanket.

If you have questions regarding Security Blanket's SELinux support, send us an email at SecurityBlanket@TrustedCS.com.

Do Software Engineers Encourage Bad Security Practices?

2011-04-26T08:14:00.006-04:00

Software engineers sometimes enable or even encourage bad security practices. You might be saying, “How dare he say that!” It's not the technology that we've chosen or developed but rather those convenient features which continue to enable system administrators to do bad things.

For example, electronic mail (email) is probably the primary means of notifying system administrators of problems or job completion. These emails might be notifying the system administrator that a backup job is complete, a system is down, or a system must be patched.

Many vendor services such as a subscription to the Red Hat® Network will help system administrators manage their systems by sending a list of systems and the packages which need to be patched due to vulnerabilities.

As a software engineer on a product which assesses and hardens the operating system, my customers ask me for an email notification capability. I reluctantly tell them that we are working on a “secure solution” but they wonder why it is such a big deal.

When I explain to them that sending security assessment results of their systems via email isn't a smart thing to do, they can appreciate my concern but they still want the notification capability.

The thought of listing how their systems are vulnerable over an unsecured email channel just makes me cringe. By the way, it isn't just the system administrators — their own security officers are requesting such notifications. Oh, the horror.

Nonetheless, my customers want it. I've proposed implementing technology which might mitigate the risk such as encrypting the emails. However, some people think that it is just too confusing and inconvenient for the end-user to configure.

I have even suggested just sending a hyperlink to a secure console connection where they'll need to authenticate in order to see the assessment report. In the end, perhaps I can convince the team to just deliver a pass or fail status of the client machines and provide a hyperlink.

Many organizations are so scared of email attachments and phishing attacks they've set up elaborate complex mechanisms severely limiting email capabilities. When communicating with many of my customers, they request that I send emails with attachments (e.g., zip files) to their Hotmail™ or Gmail™ account because their work email won't accept such emails.

When I ask them if I can place the file on our secure public website (https), send them a hyperlink and a SHA-1 fingerprint of the file via email, some respond with “Oh, I don't have access to download stuff from some websites.” Then I wonder why their employer permits them to go to Gmail and Hotmail.

Some have even admitted that their system administrator related notifications are sent to their free Internet email accounts. Given all of the email security problems, such as hijacked accounts, all I can say again is “Oh, the horror!”

Internal restrictions on email which result in personnel using outside resources is similar to the problem with excessively long and complicated password requirements. Users just end up writing the password down on a Post-it® note which is stuck to their monitor.

As software engineers, we want to deliver the right solutions but when it comes to commercial products, the customers drive the features. Do we simply submit to these demands in order to make a buck or do we take a stand as leaders in information security and encourage system administrators to treat vulnerability data with more care?

Detecting Vulnerable Software Versions Using SCAP/OVAL

2011-04-07T09:09:00.039-04:00

In a few of my previous posts, I discussed misleading results some scanners might produce with regard to the versions of software present on your system. In this post, I will demonstrate how to use an SCAP capable scanner using vendor-maintained OVAL patch definitions.

In order to assess my system, I need an OVAL-capable scanner and data it can use to assess my system against. Let's start by downloading the source code to the OpenSCAP scanner. Once downloaded, I'll build the scanner on my openSUSE 11.1 system:

# /bin/tar xzvf openscap-0.7.0.tar.gz
# cd openscap-0.7.0/ 
# ./configure --prefix=/usr
# /usr/bin/make
# /usr/bin/make install

Next I need some appropriate content the scanner can use to assess my system. Since I am running openSUSE 11.1, I need to get my specific OVAL patch definitions from Novell. I simply use curl to get the exact file:

# /usr/bin/curl -O http://support.novell.com/security/oval/opensuse.11.1.xml

Now, use the OpenSCAP command utility to assess the system and write the results to a file:

#  /usr/bin/oscap oval eval --result-file ./myresults.xml ./opensuse.11.1.xml
…
…
…
Evalutated definition oval:org.opensuse.security:def:20070048: false 
Evalutated definition oval:org.opensuse.security:def:20070045: false 
Evalutated definition oval:org.opensuse.security:def:20054881: false 
Evaluation: All done. 
====== RESULTS ====== 
TRUE:            36 
FALSE:           1162 
ERROR:           0 
UNKNOWN:         0 
NOT EVALUATED:   0 
NOT APPLICABLE:  0

The last step is to create an HTML-formatted report from the results file:

# /usr/bin/oscap oval generate report \
--output ./howamidoing.html ./myresults.xml

The report could do a better of job of correlating the reference identification to the actual vulnerable software. According to OpenSCAP's website, this is on their “to-do” list.

The OVAL patch definitions comprise a definition which identifies the tests to be performed. The tests determine if an object is in a particular state. Let's consider a real world example and then we'll walk through the OVAL definition. A port scanner detects an Apache web server running:

# /usr/bin/nmap -sV -p 80 localhost

Starting Nmap 4.75 ( http://nmap.org ) at 2011-04-06 08:32 EDT
Interesting ports on localhost (127.0.0.1):

PORT   STATE SERVICE VERSION
80/tcp open  http    Apache httpd 2.2.10 ((Linux/SUSE))

Service detection performed. Please report any incorrect 
results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 6.17 seconds

From this the security officer believes it is vulnerable to CVE-2009-1195. Within minutes, they communicate the issue and demand the web server be shut down because it is not at least version 2.2.12! The Nessus® vulnerability scanner also reports version 2.2.10 however, its plug-in (#39521) Backported Security Patch Detection (WWW) reports a low-severity problem. Better than a false-positive but it still means that you'll need to investigate your installed version to see if it has been patched correctly.

At this point, the system administrator verifies the package installed is version 2.2.10-2.9.1:

# /bin/rpm -q apache2
apache2-2.2.10-2.9.1

However, the package's change log and a scan performed using the vendor-maintained OVAL definitions reports it is okay. This is effective because the vendor knows when the problem was patched (backported security fixes) and it is defined in the OVAL data file. Here is Novell's definition:

<definition id="oval:org.opensuse.security:def:20091195"
      version="0" class="vulnerability">
 <metadata>
 <title>CVE-2009-1195</title>
   <affected family="unix">
     <platform>openSUSE 11.1</platform>
   </affected>
   <reference ref_id="CVE-2009-1195"
       ref_url="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1195"
       source="CVE"/>
   <description>
      The Apache HTTP Server 2.2.11 and earlier 2.2 versions does not properly
      handle Options=IncludesNOEXEC in the AllowOverride directive, which allows
      local users to gain privileges by configuring (1) Options Includes, (2)
      Options +Includes, or (3) Options +IncludesNOEXEC in a .htaccess file, and
      then inserting an exec element in a .shtml file.
   </description>
 </metadata>
<criteria operator="AND">
   <criterion test_ref="oval:org.opensuse.security:tst:2009044425"
         comment="suse111 is installed"/>
   <criteria operator="OR">
      <criterion test_ref="oval:org.opensuse.security:tst:2009055738"
             comment="apache2-devel less than 2.2.10-2.8.1"/>
      <criterion test_ref="oval:org.opensuse.security:tst:2009055739"
             comment="apache2-doc less than 2.2.10-2.8.1"/>
      <criterion test_ref="oval:org.opensuse.security:tst:2009055740"
             comment="apache2-example-pages less than 2.2.10-2.8.1"/>
      <criterion test_ref="oval:org.opensuse.security:tst:2009055741"
             comment="apache2-prefork less than 2.2.10-2.8.1"/>
      <criterion test_ref="oval:org.opensuse.security:tst:2009055742"
             comment="apache2-utils less than 2.2.10-2.8.1"/>
      <criterion test_ref="oval:org.opensuse.security:tst:2009055743"
             comment="apache2-worker less than 2.2.10-2.8.1"/>

      <criterion test_ref="oval:org.opensuse.security:tst:2009055744"
             comment="apache2 less than 2.2.10-2.8.1"/>

      </criteria>
</criteria>
</definition>

The definition's criteria includes a test (...:tst:2009055744) to determine if the Apache package is less than 2.2.10-2.8.1. The test itself evaluates the object (...:obj:2009030668) to determine if it is installed and in a specific state (...:ste:2009037265):

<rpminfo_test id="oval:org.opensuse.security:tst:2009055744" version="0"
       comment="apache2 is <2.2.10-2.8.1"
       check="at least one" 
       xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5#linux">
   <object object_ref="oval:org.opensuse.security:obj:2009030668"/>
   <state state_ref="oval:org.opensuse.security:ste:2009037265"/>
</rpminfo_test>

The object is an RPM package object and the name of the package is “apache2”:

<rpminfo_object id="oval:org.opensuse.security:obj:2009030668"
         version="0"
         xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5#linux">
  <name>apache2</name>
</rpminfo_object>

Finally, if the package object's epoch-version-string (EVR) is less than 0:2.2.10-2.8.1 then the test is true and the installed software is vulnerable.

<rpminfo_state id="oval:org.opensuse.security:ste:2009037265"
       version="0"
       xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5#linux">
   <evr datatype="evr_string" operation="less than">0:2.2.10-2.8.1</evr>
</rpminfo_state>

However, our system is not vulnerable because we have 2.2.10-2.9.1 installed on the system. Too often system administrators believe they must build and maintain their own version of the Apache web server because they think the vendor's package is out-of-date and vulnerable. Unfortunately, in some cases they just want to avoid the heartburn and frustration which comes with trying to explain to auditors and security officers that they are not vulnerable.

I have found that OVAL patch definitions are useful and accurate. However, if vendors don't maintain them, you'll have to do some more digging. And of course, if the software isn't installed as a package, you'll have to dig even more. Overall, I am impressed with OpenSCAP and am excited about the growing OVAL repositories.

NOTE: OVAL patch definition repositories for Debian, Red Hat, and others (including Microsoft) can be found here.

How Accurate is your Software Vulnerability Scanner?

2011-04-04T16:01:00.029-04:00

When it comes to determining if the version of software present on your system is vulnerable, it isn't always as straightforward as you might think. First, you must identify all of the software on your system and then determine if it is a vulnerable version. At this point, you must decide if the risk is worth mitigating by applying a patch.

My job at Raytheon Trusted Computer Solutions is to architect and develop software which focuses on securely configuring operating systems and software components. Patch management tools and software version scanners are only concerned with the version of software present on your system. For example, our Security Blanket® product would ensure that the Samba file sharing software is configured to disallow anonymous guest connections while a software version scanner is only concerned with the version of Samba.

We've been asked by some of our customers if we'd consider including software version checking in Security Blanket. I thought about the myriad of scanners and their methods and I've come to a realization…… there really isn't a complete, accurate solution available.

Before everyone jumps to the defense of their favorite product or tool, let me explain. First and foremost, you must be able to determine what software is on your system. Generally, the two most widely used methods are:

Query the package database to see what has been “installed.”
Remotely scan open network ports. Many of these tools have proprietary algorithms which utilize service signatures to aid in the determination of which versions of software are running.

These both seem straightforward. However, what about all of the software which has been copied onto systems in non-package form, such as Apache Tomcat™ servers and the numerous JAR files used? This information isn't in the package database because it wasn't “installed” — just copied.

Maybe the port scan will detect and identify the Tomcat instances — provided they are running and are accessible via the network being scanned. Many of these port scanners first check to see if the service is advertising its version but these days most system administrators suppress the version information. The port scanner must then resort to its service signatures which isn't 100% accurate.

Additionally, what about all of those plug-ins and components Tomcat may be using? What about software in non-package form and not exposed via the network? Let's say a sysadmin decided they need multiple Java Runtime Environments (JRE) and Java Development Kits (JDK) so, they copy them onto the system in non-package form.

Some scanners might inventory all “notable” executable files on the file systems such as locating all files named “java” or “javac”. If you do locate any, how do you determine their version? You certainly don't want to simply execute it with “java -version” because it may be a Trojan horse! This is what got the DISA FSO in trouble in September 2009 (see CVE-2009-4211). This is a time consuming and ineffective technique, and not to mention, possibly dangerous.

Once you've identified the software present on your system, you need to determine if it is in fact a vulnerable version. This has frustrated many of my customers in the past. Let's consider CVE-2010-4180 which identifies a vulnerability in “OpenSSL before 0.9.8q, and 1.0.x before 1.0.0c.”

The diligent sysadmin checks their installed package version and then verifies the command file (/usr/bin/openssl) before executing it to discover that 0.9.8h is present.

# /bin/rpm -q openssl
openssl-0.9.8h-28.20.1

# /bin/rpm -qf /usr/bin/openssl
openssl-0.9.8h-28.20.1

# /bin/rpm -Vv openssl |/bin/egrep /usr/bin/openssl
........    /usr/bin/openssl


# /usr/bin/openssl version
OpenSSL 0.9.8h 28 May 2008

At this point the logical conclusion is that this software must be patched. It is deceiving because the package version is actually 0.9.8h and vendor release 28.20.1. If you examine the package's change log, you'll see that the package has been patched to fix the CVE-2010-4180 even though the vendor didn't change the major version number (0.9.8h).

# /bin/rpm -q openssl --changelog

* Tue Dec 07 2010 gjhe@novell.com
- fix bug [bnc#657663]
  CVE-2010-4180
  for CVE-2010-4252,no patch is added(for the J-PAKE
  implementaion is not compiled in by default).

Scanners which use vendor-maintained OVAL patch definitions would know that this package is okay. Unfortunately, not all vendors or open-source projects provide this level of detail.

The bottom line is that system administrators must take into account all methods in which software may get copied (or installed) onto their systems. A strong change management program and strict access to systems is required. System administrators must be aware of any and all changes to their systems. Software patching is inevitable. Unfortunately, I have yet to experience an all-encompassing software version scanner and patch management tool.

U.S. Government Configuration Baseline for Red Hat Enterprise Linux 5

2011-03-01T07:02:00.015-05:00

On February 28, 2011, the U.S. Government Configuration Baseline (USGCB) for Red Hat Enterprise Linux 5 was released. The long awaited Security Content Automation Protocol (SCAP) content is the next phase in supplanting the legacy Bourne shell scripts collectively known as the System Readiness Review (SRR) scripts.

In 2010, the USGCB replaced the Federal Desktop Core Configuration (FDCC) which has always been associated with Microsoft® software. The USGCB initiative is to create security configuration baselines for Information Technology products widely deployed across the federal agencies.

In my previous post, “DISA UNIX STIG for Red Hat Enterprise Linux 5 and 6” I discussed the release of the “OS SRG (UNIX), Version 1.1” on February 2, 2011. The download included only benchmark documents in the XCCDF format.

On February 28, 2011, in an email to Red Hat’s gov-sec mailing list, Steve Grubb announced the availability of the alpha release of the USGCB content for Red Hat Enterprise Linux 5. He also had this to say about the project:

“The project took a long time, required getting involved in standards committees to update OVAL to understand modern Linux security mechanisms, plus lots of work from people that do content authoring and system testing. The project is looking for feedback via the official NIST channels (not this email list). Somewhere in the downloads should be some info on that.”

http://usgcb.nist.gov/usgcb/rhel_content.html

This release has only been tested on Red Hat Enterprise Linux Desktop 5 so, if you’ve got the time, test some Red Hat installations and provide some feedback. I want to commend the committee and the contributors because I know it was a long and laborious process. There are still lots of challenges ahead so community involvement will certainly help mature the baselines much quicker.

I, for one plan on downloading the latest version of OpenSCAP and performing some tests. I will be sharing my procedures, experiences, and test results in an upcoming blog post.

DISA UNIX STIG for Red Hat Enterprise Linux 5 and 6

2011-02-09T13:06:00.040-05:00

Organizations which use Red Hat Enterprise Linux 5 and must adhere to the U.S. Defense Information Systems Agency's (DISA) UNIX Security Technical Implementation Guide (STIG) have been stuck with documentation and assessment tools which only support up to Red Hat Enterprise Linux 4. This frustrates many system administrators because they must deal with false positives produced by the System Readiness Review (SRR) scripts and a checklist with incorrect procedures.

The only hint of future support is seen on the DISA Field Security Office (FSO) website under their frequently asked questions (FAQ). The question, “When will there be a RedHat 5 STIG and SRR scripts?” was answered as follows:

“FSO is currently working with the vendor and the DoD Consensus team to develop the security requirements for RedHat5. FSO has no plans to develop UNIX scripts which support RedHat5 since they are moving towards utilizing the Security Content Automation Protocols and HBSS (Host Based Security System) Policy Auditor for future assessments. There is no estimated completion date at this time for the HBSS Policy Auditor RedHat5 benchmark. FSO expects to have the STIG completed by late FY10 or early FY11.”

In January 2011, Red Hat Enterprise Linux 6 was released adding to the frustration. In a January 27, 2011 email, Steve Grubb (Red Hat) writes to the gov-sec mailing list that Red Hat Enterprise Linux 6 includes an OpenSCAP scanner.

Before I go into too much detail about OpenSCAP, I should give a little background. The STIGs are published by DISA FSO. There are three pieces:

UNIX STIG V5R1 (April 2006) — This is the main document and it gives plenty of explanations, rationale, and general information.
UNIX Security Checklist (published quarterly) — details the requirements by line item to include what to look for in configuration files as well as commands to be executed.
System Readiness Review (SRR) Scripts (published quarterly) — scan the operating system for compliancy. To download, you must have DoD PKI access.

The checklists and associated SRR scripts are the components most people have heartburn about. In May 2010, the DISA FSO gave a presentation titled “STIGs, SCAP, and Data Metrics.” In this presentation, they identified some “Maintenance Challenges”:

High demand from the user community for new and updated security guidance
Rapid pace of new technology
Limited resources to develop guidance and tools to evaluate compliance
Development and maintenance of varying tools/techniques for supporting compliance checks:

Gold Disk (Windows)
Security Readiness Review Scripts (Unix, some DB)

The SRR scripts have always been written in Bourne shell script for portability reasons. However, the difficulty in maintaining the collection of scripts, limited reporting capabilities, and lack of interoperability with other tools has pushed DISA in a new direction.

The Information Security Automation Program (ISAP) is a U.S. government multi-agency initiative to enable automation and standardization of technical security operations. ISAP’s technical specifications are contained in the related Security Content Automation Protocol (SCAP).

ISAP’s security automation content is either contained within, or referenced by, the National Vulnerability Database (NVD) maintained by NIST. Current SCAP protocols include CVE, CCE, CPE, CVSS, OVAL, and XCCDF.

In the aforementioned presentation, DISA went on to say that they were “Developing Security Requirements Guides (SRGs) that address overarching requirements...” in order to “promote structure.”

On February 2, 2011, DISA FSO posted “OS SRG (UNIX), Version 1.1” which was completed in November 2010. The downloaded archive contained several files but no Microsoft Word formatted Checklist document (I can live with that). Instead, you get two XML documents:

Operating System Security Requirements Guide (UNIX Version)
Operating System Policy Security Requirements Guide (UNIX Version)

The documents are in what is referred to as XCCDF. According to the provided “STIG Transformation to XCCDF FAQ” document.

The move to an eXtensible Configuration Checklist Description Format (XCCDF) formatted STIG provides the ability for the consumption of the STIGs by the various automated assessment tools, such as Host Based Security System (HBSS).

According to the FAQ, “The STIG is still used the same way; it is just that the data is in a different format.” They do provide an XSL to transform the XCCDF/XML documents to XHTML so you can read them in your web browser.

As an engineer, I love the idea of promoting structure so this makes sense to me. Now that I found my checklists, how can I perform a scan like I did with the SRR scripts?

According to the email announcement from DISA FSO, the “automated assessments can then be accomplished using SCAP compliant tools through the use of DISA generated benchmarks based on the STIG requirements. This will replace the UNIX SRR scripts.”

A benchmark is the version of the STIG that contains the Open Vulnerability and Assessment Language (OVAL) code. This code allows an OVAL-compliant security tool to perform an automated assessment of the system. This is similar to the quarterly released checklist and associated SRR scripts which are used today.

Like the SRR scripts, the benchmark does not contain all the checks necessary to meet the STIG requirements. For example, GEN00080 requires that “all server system equipment must be located in a controlled area.” There is no way an automated benchmark could check for this so you would have to “manually” verify this yourself.

Based on the information above, I need a benchmark (OVAL content) in order to assess a system. The FAQ document lists the following:

STIG-OVAL.xml – This file contains the detailed OVAL check code. This will only be provided if OVAL exists for the technology.
STIG-CPE-OVAL.xml - This is OVAL code that will provide information to the tool on how to check to see if the product being evaluated exists on the system.
STIG-CPE-DICTIONARY.xml – This is the file that contains the CPE information about the product.

However, I could not find any of those files. The only XML files found were the checklist documents. According to the DISA FSO email announcement, this content won't be available until late 2011.

Out of curiosity, I went to the NVD website and searched through all of the operating system checklists. Specifically, I needed a checklist with the associated OVAL content in order to assess the system.

I found 164 checklists, most of them are for Microsoft products to support the U.S. Government Configuration Baseline (USGCB), formerly known as the Federal Desktop Core Configuration (FDCC). The only UNIX/Linux related content I found were:

zOS ACF2 STIG Checklist (Version 6, Release 4) (XCCDF only)
zOS TSS STIG Checklist (Version 6, Release 4) (XCCDF only)
zOS RACF STIG Checklist (Version 6, Release 4) (XCCDF only)
DoD Consensus Security Configuration Checklist for Red Hat Enterprise Linux 5 (XCCDF and OVAL Content, yeah!)

However, there is no OVAL content to support the UNIX STIG. The Red Hat content listed above had both the checklist and the OVAL content but it was only for Red Hat Enterprise Linux 5 and not for the UNIX STIG.

The SCAP website emphasizes the importance of community involvement. It appears that DISA's strategy is to focus on developing the Security Requirement Guides (SRGs) and associated checklists (XCCDF) while the “community” develops the OVAL content. This alleviates them from having to write and maintain the SRR scripts.

Presently, the primary vendor contributing to content is Red Hat. As a matter of fact, they are sponsoring the OpenSCAP project, hence the delivery of the OpenSCAP scanner in Fedora 14 and Red Hat Enterprise Linux 6.

So the bottom line is Red Hat Enterprise Linux 5 and 6 are still not supported by the DISA UNIX STIG. It is expected that vendors (and the SCAP community) will deliver the appropriate SCAP content to assess systems in late 2011. Until then, system administrators can continue to use the SRR scripts and deal with false positives on the newer Red Hat distributions.

In my next blog post, I will discuss Security Blanket's use of SCAP enumeration and mapping specifications as well as our possible role as a “producer.”

Common Criteria (ISO 15408) Evaluation Assurance Level

2011-01-28T11:52:00.032-05:00

I develop security products for Linux® and many of my customers are only permitted to implement technology with a specific level of assurance.

Most of these customers must also adhere to the U.S. Defense Information Systems Agency (DISA) UNIX Security Technical Implementation Guide (STIG).

I have been asked, “Since Red Hat Enterprise Linux 4 and 5 are Common Criteria EAL4+ certified, do they still need to be tested against the DISA UNIX STIG?” The answer is YES, they do.

First of all, I don't work for DISA or Red Hat but I do have years of experience in this area so, I will try to shed some light on this topic.

The Common Criteria for Information Technology Security Evaluation (abbreviated as Common Criteria or CC) is an international standard (ISO/IEC 15408) for computer security certification. It originated from three standards: ITSEC, CTCPEC, and TCSEC.

Common Criteria is a framework in which computer system users can specify their security functional and assurance requirements. Vendors can then implement and/or make claims about the security attributes of their products, and testing laboratories can evaluate the products to determine if they actually meet the claims.

To obtain a certification, organizations can go through the National Information Assurance Partnership's (NIAP) Common Criteria Evaluation and Validation Scheme (CCEVS). The NIAP is a U.S. government initiative to meet the security testing needs of both information technology consumers and producers and is operated by the National Security Agency (NSA).

The Evaluation Assurance Level (EAL1 through EAL7) of an IT product or system is a numerical grade assigned following the completion of a Common Criteria security evaluation.

To achieve a particular EAL, the computer system must meet specific assurance requirements. Most of these requirements involve design documentation, design analysis, functional testing, and/or penetration testing. The EAL number assigned to a certified system indicates that the system completed all requirements for that level.

In some cases, the evaluation may be augmented to include assurance requirements beyond the minimum required for a particular EAL. Officially this is indicated by following the EAL number with the word augmented and usually with a list of codes to indicate the additional requirements. As shorthand, vendors will often simply add a "plus" sign (as in EAL4+) to indicate the augmented requirements.

One particular augmentation is the evaluation against the Labeled Security Protection Profile (LSPP). The LSPP requirements are derived from the B1 class of the U.S. Department of Defense security standard called Trusted Computer System Evaluation Criteria (TCSEC) which was originally published in 1985.

The Multilevel Security (MLS) component of Security-Enhanced Linux (SELinux) helps a Linux operating system successfully pass an evaluation against the LSPP. SELinux is predominantly available in Red Hat-based systems but is also available in Debian as of the etch release, Ubuntu as of 8.04 Hardy Heron, Hardened Gentoo, Yellow Dog Linux, and openSUSE 11.1.

When I think of an EAL certification, I think of Underwriters Laboratory (UL) certifying the safety of an appliance like a toaster. It says that the toaster is safe as designed but if you use it in an incorrect manner—like sticking a metal knife in it while it is plugged in—then all bets are off. The certified operating system has the capabilities and features to meet the tested security level but in many cases, you must still configure and enable those controls. Hence, the implementation of security guidelines such as DISA UNIX STIG and Center for Internet Security Benchmarks.

Of particular interest to my customer base are the Linux operating systems. Here are a few:

Red Hat Enterprise Linux 3 is EAL3+ CAPP
Red Hat Enterprise Linux 4 is EAL3+ CAPP and EAL4+ CAPP
Red Hat Enterprise Linux 5 is EAL4+ CAPP/RBACPP/LSPP
Red Hat Enterprise Linux 6 (Pursuing EAL4+) (Read the announcement here)
Novell SUSE Linux Enterprise Server 10 SP1 is EAL4+
Oracle Enterprise Linux 4.4 and 4.5 is EAL4+ (See here under the operating system section)
Oracle Enterprise LInux 5.1 is EAL4+ LSPP (See here)

It is my understanding that even though CentOS Linux is built from publicly available Red Hat sources, it is NOT EAL4+ certified.

The associated cost with certifying an operating system or application has been a deterrent for many open source development communities and a source of frustration for technologists wanting to implement different solutions. Also, these certifications can take months, even years, to achieve. This time line can significantly delay the adoption in organizations with strict requirements for these validations, such as the military and government agencies.

I hope this information was helpful in explaining the Command Criteria (ISO/IEC 15408) Evaluation Assurance Level. Watch for my next article in which I will discuss the availability of the DISA UNIX STIG for Linux operating system versions as well as SCAP.

Tips for Taking Charge of a Sysadmin Team

2011-01-14T15:06:00.002-05:00

Over the years I have worked in various roles but the most challenging has been when I assumed control of an already established system administration team. Several times, a former colleague moved to a new company and inherited a system administration organization that needed some improvements. I am proud to say that those colleagues convinced me to work for them again to overhaul the system administration operations.

Each of us has our own leadership style and everyone's approach may differ from organization to organization; nonetheless, I wanted write about my approach. First of all, I have a handful of generic questions I ask:

What am I responsible for?
Does the architecture make sense? Are there any overlaps of responsibility?
Who has access to the assets I am responsible for?
What are we doing to maintain system availability? Backups?
What is our configuration management process? How fast do we turn up systems and introduce new applications?
What are the strengths and weaknesses of my team?
What are we missing?

When I enter the organization, I don't immediately request access to the systems but rather I begin going over architectural diagrams, operational procedures, and just peering over the shoulders of the system administrators. If the aforementioned documents are not present, then we have a problem.

It is imperative that there is a clear understanding of the system components within the architecture. I would immediately have the team begin compiling diagrams and work flows so that we can understand the system's architecture.

This would include high-level diagrams as well as a detailed asset management inventory of EVERY host. I want to see every host, its operating system version, and respective application versions (e.g., Tomcat, Oracle, Apache).

I would also require networking diagrams and a mapping of each system component to a particular organizational group. For example, a particular database contains billing information and it is used by group XYZ. Who maintains that database schema and the software which manages the data? I would begin mapping those groups to system components so my team has a clear understanding what organizations they are supporting.

Next, I would set out to change the password of every privileged account in the system. This is where some people become upset, but remind yourself it is for the best. As we make changes to the system to stabilize it or improve performance, we need to know exactly what changes were made. This password changing step is imperative in order to clearly identify who has access to the system.

I would ensure that every system is logging and auditing accordingly so you can see who is attempting and gaining access to privileged accounts. Furthermore, I would no longer allow any privileged account (e.g., root and oracle) to be logged into directly.

Secondly, I would start changing the root password on every system and only give it out to a select few senior system administrators. We can sort out sudo access for junior administrators as we move forward. Next, I would work with the lead database administrator and have the appropriate account passwords changed.

As people start complaining they no longer have access, we will evaluate each individual's role and determine if they truly need access. Too often system developers have root access to production systems. If developers need access to production systems, then, in my opinion, the application isn't ready for production.

Of course, while reviewing these critical system and application accounts the system administrator accounts should also be reviewed. Sometimes, you will find accounts for individuals who are no longer employed so it should be removed immediately. I would also set password aging on system administrator accounts so that unused accounts are locked. This will help identify dormant accounts.

Once I have narrowed down the systems I am responsible for and who has access to them, I will closely examine the architecture and processes to ensure business continuity. Are we doing backups? Do we have redundant or mirrored storage solutions? How often do we test fail over and recovery procedures?

By this time, you should already have a good understanding of the existing change management processes. If it is insufficient, then lobby to get it fixed! Do you have a high provisioning rate? In other words, are new systems routinely being inserted into production with little-to-no testing?

Understanding the team is critical. You're always going to find an eclectic blend of personalities and talent in a group of system administrators. It's your job as a leader to determine who really “knows their stuff” and who has everyone bamboozled. Find out who the information hoarders are and those who have been stuck in a role they aren't happy in.

Lastly, you might stumble across some strange component in the architecture that is either antiquated or is just completely different from everything else. For example, the architecture is comprised of 99% Red Hat Linux and you have one HP-UX box running one small application and none of the system administrators knows anything about it.

My first question is how did it get into the architecture? What's the long term plan to maintain and support it? Will the team get any training on it? Or is this one of those situations where someone outside of the operations group has been granted exclusive root access to the system? [Cringe]

In the end, it is really your experience and leadership which can help to improve a system administration team. It has been my experience that having a clear picture of the environment you are expected to build and maintain is a critical first step to ensuring the success of the team.

Tips for Deploying Secure Shell in Linux and UNIX

2011-01-10T12:13:00.011-05:00

There is no doubt that the de facto technology to gain remote command line access is Secure Shell on Linux and UNIX systems. While there are numerous security guidelines suggesting many different configuration settings, I think there are some fundamentals which are missed.

Secure Shell is, in my opinion, the best method for remote access due to its flexibility and security. It makes it attractive for system administrators as well as system developers and architects. The ability to easily execute commands on remote systems and retrieve files over “secure” channels is seductive. Especially, since there is little-to-no programming required to facilitate this. However, in my opinion, this should be done with caution.

First of all, Secure Shell (SSH) is a network protocol that allows data to be exchanged using a secure channel between two networked devices.¹ SSH was designed as a replacement for Telnet and other insecure remote access methods, which send information, notably passwords, in plain text, rendering them susceptible to packet analysis.

The most common server daemon and client software which employs the SSH Protocol is developed by the OpenSSH Project. Of course, there are several SSH capable clients available including the popular Windows client called PuTTY.

Besides having encrypted communications, the idea of being able to remotely log in or copy a file from one machine to another without being prompted for a password is attractive to many developers and architects. Through the use of Public Key Infrastructure (PKI) the machines can “trust” certain keys and allow remote access.

This is great and many would ask, “What's your beef with it Jamie? You're just a grumpy paranoid old man.” Yes, I am. My beef is that this technique often gets abused and is sometimes deployed in a sloppy manner.

If you're going to use hands-free logins, please consider a couple of things. First of all, restrict access to the OpenSSH daemon through the use of host-based (iptables) firewall and network-based firewalls.

Also, restrict who (which users) can login using configuration options such as DenyUsers, AllowUsers, DenyGroups, and finally AllowGroups. Some system administrators create a user group called “sshusers” and then set “AllowGroups sshusers” in the configuration file. Then assign the “sshusers” as the secondary group to user accounts to login via Secure Shell.

Administrative, shared accounts such as Oracle should NEVER be logged into directly. Furthermore, “PermitRootLogin” should be set to “no” to prevent direct login to the root account.

As an added layer of protection ensure the “StrictModes” value is set to “yes”. Sometimes users, and system administrators, get sloppy and relax the discretionary access controls on user home directories where the SSH credentials are stored.

Finally, I would ensure that the Secure Shell service is using the strongest encryption possible and is only configured to accept protocol version 2 connections. Set the following in /etc/ssh/sshd_config:

Protocol 2
Ciphers blowfish-cbc,aes256-cbc,aes256-ctr,aes192-cbc,aes192-ctr,aes128-cbc,aes128-ctr

I hope this article got you thinking about how you use Secure Shell. I encourage you to read the various security guidelines and other recommendations that are available. In the end, however, one of the best defenses is diligence by system administrators who review and remove inactive user accounts and are aware of how their architecture is configured.

1. Network Working Group of the IETF, January 2006, RFC 4252, The Secure Shell (SSH) Authentication Protocol

Tips for Securely Using Temporary Files in Linux Scripts

2010-12-06T15:03:00.012-05:00

Over the years, I've written hundreds, if not thousands, of shell scripts. With the ease at which you can redirect input and output within a shell script, many sysadmins store data in temporary files for processing purposes. In some situations scripts become essential to the day-to-day operations of a system and as such, may end up running on a regular basis via crontab – never to be looked at again.

Unfortunately, some sysadmins who write scripts might store sensitive data in temporary files, don't restrict access to temporary files, and might forget to remove them from the system when they are no longer needed. In many cases, they use them when it isn't even necessary. The beauty of Linux and UNIX is that there are hundreds of ways to accomplish the same task. I will keep my Bash examples simple so you can focus on grasping the general concepts.

Restrict access to temporary files

This is the most commonly forgotten step. If you are like most sysadmins who write temporary files to /tmp or /var/tmp, set your umask before file creation.

# cut -f1 -d: /etc/passwd > /tmp/test
# ls -l /tmp/test
-rw-r--r-- 1 root root 207 Dec  6 11:56 /tmp/test

# rm -f /tmp/test

Now, let's set the umask, create the file again, and check its access controls:

# umask 077
# cut -f1 -d: /etc/passwd > /tmp/test
# ls -l /tmp/test
-rw------- 1 root root 207 Dec  6 11:58 /tmp/test

# rm -f /tmp/test

As you can see, the more restrictive umask only grants the file owner read and write permission. Additionally, instead of writing temporary files to /tmp or /var/tmp, write the files to a dedicated, private area such as one under the user account's home directory. Limit access to this directory with permissions such as 0700.

Use a random string as the filename

To reduce the likelihood that someone knows the exact name of the temporary file your script creates, avoid using file name prefixes and use random characters as the filename. For example:

# umask 077
# tempfile=$(head -c 12 /dev/urandom |mimencode  |tr -d "/")
# echo $tempfile
wwiboOPRHbozVuce

# cut -f1 -d: /etc/passwd > /tmp/$tempfile
# ls -l /tmp/$tempfile
-rw------- 1 root root 207 Dec  6 12:10 /tmp/wwiboOPRHbozVuce

# rm -f /tmp/$tempfile

Don't use temporary files at all

Of course, the safest method is to do in-line processing using pipes, a subshell environment, or a variable. For example, if you wanted an alphabetical listing of user accounts simply use a pipe:

# cut -f1 -d: /etc/passwd | sort -u

or even:

# sort -u -t: -k1 /etc/passwd |cut -f1 -d:

If you wanted to perform an action on each account on the system, invoke a subshell on a for...loop such as:

for user in $(cut -f1 -d: /etc/passwd)
do
    printf "some action on %s\n" $user
done

Summary

In summary, take a look at your /var/tmp and /tmp directories. Do they have a bunch of strange files which are open to the world? Take inventory of all of the scripts running on your system especially those which are executed regularly via crontab. Make sure you clearly understand what they are doing and if they are creating any temporary files. If they do, try some of the aforementioned tips to help secure them.

Sandboxing: Understanding System Containment

2010-11-29T09:23:00.000-05:00

I recently attended the 24th Large Installation System Administration (LISA) Conference in San Jose, California. I met so many great people including Matt Olander, the Chief Technology Officer of iXsystems and a member of FreeBSD Project’s Marketing and Public Relations teams. Matt has been a long time BSD advocate and he was previously the Director of IT at BSDi.

His enthusiasm and in-depth knowledge of Berkeley Software Distribution (BSD) history was impressive. Moreover, his knowledge of its reliability aspects, code base management, security, and upcoming release information was equally impressive. He talked to me for a few minutes on camera which I will release in another post.

Later that evening, I reflected on the long history of BSD and its innovative contributions. In particular, I thought about the FreeBSD jail mechanism and its contributions to system survivability through the concept of containment.

I also thought about all of the incarnations of technologies aimed at providing some level of containment. As a foundation of explaining these technologies to my sales and marketing team, I felt it necessary to explain the concept of containment and how it relates to system survivability.

Because of my years of service in the United States Navy, I think of system survivability as being analogous to ship survivability.

Visualize a ship's internal layout as a honeycomb. Each cell, or compartment, can be sealed off from another in case of compromise. If one compartment begins to flood, the hatches (doors) can be closed to prevent adjacent compartments from flooding. In effect, the flood is contained and the ship can continue its mission.

In information technology, most people grasp the concept of network segregation through use of virtual local area networks (VLAN), routing, and firewalls but the containment idea seems to be lost when it is applied to a collection of operating systems running in virtual guests on a single platform. Even more confusing is the idea of containment within a single operating system by segregating processes or groups of processes to serve as a single application.

Consider a hardware- or software-based virtualized system which is running many virtualized guests. Each guest could be running a separate operating system. If you think of each guest as a compartment on the ship, you must think about how you can contain one guest if it is compromised during an attack or suffers a catastrophic failure.

Of course, these guests can't be completely isolated because they will need some shared resource (e.g., oxygen) or possibly a means to communicate with another compartment. Just like in a ship, each compartment serves a purpose but interacts with other compartments to serve the entire ship. Likewise, in a virtualized system architecture, the access control points between compartments and to the outside world must be managed appropriately for survivability.

Within each guest resides an operating system running one or more applications such as a database or a web service. Similarly, ship compartments serve various purposes such as navigation, fire control, or propulsion. In both situations, it makes sense to not “put all of your eggs in one basket” in case of catastrophic failure. So, you should at least have a redundant compartment if possible.

If a single guest operating system were running two web services, wouldn't it make sense to have them buffered from each other? For example, ensure that a compromise to one of them doesn't jeopardize the other. Likewise, if a ship's compartment had two pieces of equipment and one caught fire or had an electrical failure shouldn't it be insulated enough to protect the other equipment?

The value of containment should be obvious by now. Yet, I haven't mentioned the most important characteristics of containment technologies – that is to be simple in design and easily managed. Complex configurations can result in an inconsistent configuration which translates into improper containment and an uncertain security posture.

In the Navy, the term “Material Conditions” is defined by Naval Warfare Publication (NWP) 3-20.31 as a ship's configurations representing varying degrees of closure (hatches being opened or closed among other things). The material condition of readiness is set according to the degree of threat to the ship and the current operational directives.

Examples of operational directives would be day-to-day operations as opposed to being in a combat situation. These conditions establish the fighting integrity of the ship and maintain its survivability. In the simplest form, the aforementioned hatches provide access into the compartments and containment when necessary. Each door is adorned with a symbol such as a big black letter 'Y' enclosed in a black ring — referred to as “Circle Yoke.”

When a ship's readiness is set, all of the respective hatches marked with that level and below are secured. More importantly, many traditional hatches have eight dogs (levers) which must be secured in a specific order for the hatch to be completely sealed (locked down). A more simplified hatch such as the QAWTD has a single mechanism to secure all of its levers at once in order to simplify and expedite the establishment of the required readiness level.

If you think of the hatches (doors) as access control points in a system, such as discretionary access controls, mandatory access controls, host-based firewall rules, or the grouping of processes, the easier it is for an administrator to configure and understand them, the better. This may help when establishing the operational directive for a public facing web server that will likely be configured differently (tighter) than an internal web server due to the risk of attack involved.

The idea of containment within an operating system is sometimes referred to as sandboxing. Some of the more popular technologies specifically designed to accomplish this come to mind: FreeBSD jails, Solaris Containers (including Solaris Zones), and SUSE Linux's AppArmor.

Security-Enhanced Linux (SELinux) is full implementation of mandatory access controls (MAC) and is capable of Multilevel Security (MLS). Although it wasn't designed specifically for containment, it certainly can be used for this purpose and several policies delivered with Red Hat-based systems accomplish this out-of-the-box.

There are more technologies available but an organization should weigh a technology's capabilities against the mission requirements or “directives.” You should then ultimately consider the ease of deployment and manageability. In my opinion, the easiest technologies to implement are FreeBSD jails, Application Armor (AppArmor), and then Solaris Containers (including Solaris Zones).

LISA 2010: Sysadmins Discuss Virtual Mentorship

2010-11-23T08:07:00.016-05:00

In November of this year, I attended the 24th Large Installation System Administration (LISA) Conference in San Jose, California.

LISA is an annual technical conference sponsored by USENIX: the Advanced Computing Systems Association in cooperation with the League of Professional System Administrators (LOPSA) and the Storage Networking Industry Association (SNIA).

On Friday, November 12th, I led a “Guru Is In” session on “Security” where we had a lively exchange of views and information.

In attendance were junior and senior system administrators, seasoned administrators, and salty old-schoolers like myself. At one point, we discussed the idea of mentorship and the operational environment many younger system administrators find themselves.

For me, this was a subject of poignant interest and later that evening I reflected upon my early work environments compared to what system administrators face today.

First and foremost, the ratio of systems to system administrators is much different. Today, there seems to be fewer system administrators but each is responsible for far more systems.

In the early days, system administrators were required to have a diverse set of skills and knowledge. For example, we were required to have a deep knowledge of how file systems worked, networking concepts, and access control mechanisms.

Today, there is a greater dependency on automated tools in order to improve productivity. Our field also seems to have as many specialists as the medical community.

There are system administrators focused on different operating systems, cluster specialists, network administrators, database administrators, system virtualization managers, and even security administrators.

When I started, I was originally a system programmer but eventually assumed responsibilities as a system administrator. I had a mentor who assigned me specific jobs such as performing system backups, account management, and analyzing daily logs.

I continued to perform those jobs until my mentor determined I was competent in that role and then I moved on to the next role. This could last anywhere from weeks to months. I rotated through all of the on-site roles until I was capable of performing any task in the data center.

In those early days, vendor manuals and the few technical books available on the market were treasured. More importantly, the strong relationship with my mentors helped me gain the most out of my experiences.

A good mentor would not tell you “exactly” how to do something but rather encourage you to learn how and why things are done.

Of course, they would instruct you on the site's best practices but most importantly they taught you “how to learn.”

In other words, how to solve problems without jeopardizing an entire system. Of course, if I made a serious mistake or had a lapse in judgment... let's just say that some of us were motivated through public humiliation. But it was all in good fun.

Several years later when I became the mentor, I would immediately set precedence by asking them if they did any research on the problem before coming to me with the question.

If they came to me with a question they've asked before, I would ask them why they didn't write down the answer the first time. It is so important that system administrators become self-sufficient.

I believe having a mentor is extremely beneficial. These days, however, several factors hinder this type of relationship:

Fewer on-site administrators.
Higher turnover rate. Many administrators don't remain on a project as long as they did many years ago.
The Internet. The massive amount of online technical resources allows one to be self-taught.
Volumes of printed publications (e.g., O'Reilly books)

The unfortunate fact is that many junior system administrators can quickly get in over their heads.

One of the administrators at the LISA 2010 conference told me he started out being responsible for only a few machines but the number of machines quickly grew out of control.

Many of his generic configuration decisions did not scale well and it became very difficult to manage. This is a classic problem.

Many of the operational resources, such as software repositories for updates, a good back-up system, and sound configuration management, hadn't been considered during initial set up either.

These are the kinds of things experienced administrators can pass on to junior administrators, including documentation skills! Yes, this skill is one that must be mastered but is often brushed aside and considered trivial early in one's career.

Of course, there are plenty of training programs and certifications available. Personally, I've never been a fan of vendor specific certifications because it focuses on proprietary technology. It certainly benefits the immediate company which has hired the individual or has paid for the training.

However, I think it is far more critical for a system administrator to have a vendor neutral understanding of the fundamentals. This gives them a strong foundation which supports easily learning other technologies.

Finally, I strongly believe a mentor helps harvest those soft skills. After all, knowledge is far more powerful and useful when coupled with wisdom.

Because there are fewer on-site administrators and there seems to be higher turnover rate these days, it is critical that a virtual mentorship be in place. This may not be a one-on-one relationship but having a means to ask questions in almost real-time is critical.

Strong organizations and communities such as USENIX and LOPSA are invaluable. Membership dues must be reasonable because many junior administrators aren't willing to shell out the cash early on if they don't see great value. I strongly recommend checking out the The LOPSA Mentorship Program launched in August 2010.

In an effort to protect an organization's assets, many restrict or deny access to valuable on line tools and resources such as instant messaging, sites with “blog” in the URL, or user group websites.

This is unfortunate because these resources can help administrators quickly resolve problems. Additionally, most system administrators maintain a long term professional relationship with fellow system administrators and having the ability to ask questions via instant messaging is critical.

I think an organization that already entrusts their information technology assets to a system administrator should seriously consider trusting them not to abuse such Internet access. Providing the newer system administrators with access to the tools that can help them become successful, faster, will ultimately be beneficial to the organizations that they support.

LISA 2010: OS Security In The Cloud

2010-11-22T06:54:00.002-05:00

In November of this year, I attended the 24th Large Installation System Administration (LISA) Conference in San Jose, CA. LISA is an annual technical conference sponsored by USENIX: the Advanced Computing Systems Association.

On Friday, November 12th, I spoke at a “Guru Is In” session on “Security” where I was able to facilitate a discussion with all levels of system administrators (and an auditor) about compliancy versus security. We had a lively exchange of views and information.

I really like LISA and USENIX. For one thing, the atmosphere is engaging but relaxed. Even though it comprises some of the best minds in the industry, there is no pretentiousness but rather a willingness to share ideas and experiences. Outsiders would probably consider this eclectic blend of personalities and skill sets as strange whereas, the rest of us just consider it a comfortable environment which fosters an exchange of ideas.

There is no doubt that cloud computing is a hot topic these days. As an engineer, most of my career has been focused on operating systems and related security. I find it interesting that some people tend to forget the operating system security fundamentals when it comes to cloud computing, hardware virtualization, and software virtualization.

In the end, it is still an operating system. A SUSE® Linux® instance running under an IBM® Integrated Facility for Linux (IFL) on System z® mainframe deserves the same consideration when it comes to operating system security. Unused services should and must still be disabled, discretionary access controls tightened, and regular software updates applied. The same holds true when it comes to operating system images deployed in the cloud.

Given today's operational tempo, many organizations tend to quickly provision new operating system instances and virtual machines in order to handle increasing workloads. When storage technology was growing at an exponential rate, some system administrators would frivolously allocate huge amounts of space as if there was a bottomless pit of resources. By limiting the ports, services, and installed software on a virtual machine an image will ultimately conserve resources and reduce the attack surface.

Another side effect of rapidly provisioning new operating system images is a varying security state. Consider a standard image which may get deployed over and over. This image has been “locked down” in accordance with some set of security guidelines. However, if the security guidelines or requirements change, how do organizations consistently reconfigure previously deployed images with the new standards?

One situation I've seen quite often is when an administrator would clone or snapshot an image. Then they apply new security settings to the image. However, when a failure occurs they restore the image to a previous snapshot which does not include the recently applied security settings.

Technology facilitates today's operational pace however, without disciplined administrators willing to clearly document activities this situation can quickly get out of control. It is imperative that processes not be burdensome in order for the administrator to keep up the pace but still clearly understand their current security posture.

Of course, the right automated tools can be a great resource provided they are non-intrusive, easy to use, and perform consistently and reliably.

Which Linux or UNIX Version Am I Running?

2010-11-17T09:50:00.015-05:00

This may seem like a simple question but for a lot of system administrators who “inherit” systems or are unfamiliar with operating systems that have been forced upon them, it can be very confusing. Especially if you're coming from a proprietary UNIX® operating system such as Solaris™ or HP-UX to a Linux®-based distribution.

For most of us old-school UNIX people, the reliable “uname” utility is what we are most familiar with. Execute it with the -a option and you get something like:

SunOS sungod 5.10 Generic_137138-09 i86pc i386 i86pc

Cryptic to most but for a seasoned Solaris administrator it means that the host “sungod” is running Solaris 10 on an x86 (non-SPARC) system and the current kernel patch level is 137138-09. If you run the same command on a Linux system, you might see something like:

Linux greenlantern 2.6.27.48-0.3-default #1 SMP 2010-09-20 11:03:26 -0400 \
x86_64 x86_64 x86_64 GNU/Linux

At most, you can determine that the host “greenlantern” is in fact a Linux system running a default kernel version 2.26.47 and it is a 64-bit system because of the “x86_64” in the statement.

The “uname” utility was first introduced as part of the UNIX Programmer's Workbench (PWB) in 1973. Not only is “uname” a utility, it is a system call – uname() conforms to System Vr4 and POSIX.1-2001. It extracts information from the running kernel.

Linux distributions are built off of standard kernels but are packaged and bundled differently. Some distributions are Debian-based while others might be Red Hat-based. The collection of packages and how the packages were compiled and ultimately delivered are what make Linux distributions unique.

Most UNIX and Linux operating systems have some form of a release file detailing the operating system version and release information. This file, usually in the /etc directory, is a simple text file.

Some operating systems adhere to POSIX while others strive to be Linux Standard Based (LSB). Of course there are more standards and this fact reminds me of Andrew Tanenbaum's famous statement, “The nice thing about standards is that there are so many of them to choose from.”

For those systems which comply with LSB, you can use the lsb_release(8) utility. For example, running the lsb_release command on my openSUSE system reveals the following:

$ lsb_release -r -i -c -d
Distributor ID: SUSE LINUX
Description:    openSUSE 11.1 (x86_64)
Release:        11.1
Codename:       n/a

Much more informative than the “uname” utility. It should be noted that the utility just parses various configuration files such as those in /etc. Specifically, on SUSE systems it examines the following files:

$ ls -l /etc/SuSE-*
-rw-r--r-- 1 root root 24 Dec  3  2008 /etc/SuSE-brand
-rw-r--r-- 1 root root 38 Dec  4  2008 /etc/SuSE-release

Here is a list of some operating systems, related commands, and their release files which will help you determine the specific version and release of your operating system:

Operating System	Command or Configuration Files
AIX	uname -a oslevel -g
Fedora	`/etc/fedora-release`
FreeBSD	uname -a
HP-UX	uname -a
OpenSUSE and Novell SUSE	/etc/SuSE-brand /etc/SuSE-release /etc/lsb-release
Red Hat	`/etc/redhat-release`
Solaris	`/etc/release`
Ubuntu	`/etc/lsb-release`

Finally, many system administrators are confused when they apply all of the available updates to their system via their local software repositories but are still running the same minor revision. For example, if you are running openSUSE 11.1 and you perform a “zypper update” to install available software updates, this will not bring your system up to openSUSE 11.2. To do this, you must specifically issue the distribution upgrade command. (zypper dist-upgrade). This is because when you perform a normal update, it is only examining the repositories your current system has configured. For example, here is my list of repositories (zypper lr):

$ zypper lr

#  | Alias              | Name                       | Enabled | Refresh
---+--------------------+----------------------------+---------+--------
1  | NVIDIA             | NVIDIA                     | Yes     | Yes
2  | NVIDIA-11.1        | NVIDIA-11.1                | Yes     | No     
3  | Packman Repository | Packman Repository         | Yes     | Yes    
4  | adobe-linux-i386   | Adobe Systems Incorporated | Yes     | No     
5  | google             | Google - i386              | Yes     | No     
6  | google-chrome      | google-chrome              | Yes     | Yes    
7  | google-testing     | Google Testing - i386      | Yes     | No     
8  | openSUSE 11.1-0    | openSUSE 11.1-0            | No      | No     
9  | repo-debug         | openSUSE-11.1-Debug        | No      | Yes    
10 | repo-non-oss       | openSUSE-11.1-Non-Oss      | Yes     | Yes    
11 | repo-oss           | openSUSE-11.1-Oss          | Yes     | Yes    
12 | repo-source        | openSUSE-11.1-Source       | No      | Yes 
13 | repo-update        | openSUSE-11.1-Update       | Yes     | Yes

On the other hand, Red Hat distributions such as CentOS would be updated to the next minor revision (e.g., 5.4 to 5.5) because of the way the repositories are structured.

For system administrators maintaining patch levels and an accurate inventory of their systems, it is imperative they know how to determine the exact operating system version. Hopefully, this post has provided some guidance on clarifying how to find this important information.

Large Installation System Administration (LISA) 2010

2010-11-15T15:32:00.021-05:00

Today is my first day back in the office after spending last week at the 24th Large Installation System Administration (LISA) Conference in San Jose, CA. LISA is sponsored by USENIX in cooperation with the League of Professional System Administrators (LOPSA) and the Storage Networking Industry Association (SNIA). On Friday, November 12th, I spoke at a “Guru Is In” session on “Security” where I was able to facilitate a discussion with all levels of Sysadmins (and an Auditor) about compliancy versus security. We had a lively exchange of views and information.

I've never been a member of LOPSA or SNIA. Personally, I wasn't interested in storage technology at the conference although I've been a part of USENIX for many years. I really like these shows because it isn't just about vendors pushing their technology but rather an opportunity to learn about what's going on in the community. When I am talking with the system administrators, technologists, and fellow engineers I get an unfiltered data stream. It is so much more exciting to speak with fellow geeks instead of reading some marketing research paper.

I also had an opportunity to speak with John Arrasjid, a USENIX Director and Principal Architect at VMware. We talked about operating system security, cloud computing, and next year's LISA to be held in Boston, Massachusetts. He was even gracious enough to answer a couple of questions on camera. I will be sharing that video soon.

I met people from all over the world including North America, Taiwan, Germany, Norway, and England. Not only were the locations diverse, so were the industries which these system administrators supported. Raytheon Trusted Computer Solutions received great feedback on our Security Blanket product. I fielded lots of questions regarding security guidelines and learned about other guidelines that others are using in their organizations.

Of course, both Facebook and Google were present and were aggressively looking to hire more technical people. LISA is certainly a great place to find great talent! I also had an opportunity to speak with a Google engineer regarding their Google Cloud solutions. One thing is for certain, the Google team's energy and enthusiasm were unmatched. The RTCS booth was directly across from theirs so I was able to snag this picture with some of my new Google friends.

Over the next few weeks I will be sharing the great information I gathered while at the conference on my blog (http://tcs-security-blanket.blogspot.com/).

Lock Down Novell SUSE Linux on IBM System z

2010-11-01T08:25:00.010-04:00

Our current Security Blanket® development road map includes support for openSUSE® 10.3 and Novell SUSE® 10 SP3 scheduled for January 2011. In a previous release, we delivered support for the mainframe but in January, Security Blanket support will be expanded to include all of the following operating systems running on IBM® System z®:

Red Hat Enterprise Linux 5.2 and later
Novell SUSE 11 (including SP1 and SP2)
Novell SUSE 10 SP3

If you didn't know already, Security Blanket is the only tool that automatically configures your operating systems to meet industry standards and allows for the easy creation of customized security lock down profiles.

We have supported openSUSE 11 and Novell SUSE 11 in the last several releases however, we've had a lot of customers request support for release 10 — so, we are obliging.

In addition to the new SUSE support, we have also improved many existing modules and added some new ones to support the Center for Internet Security (CIS) Benchmark for “Solaris 10 11/06 through 10/09”, v5.0.0, July 2010.

I will be in San Jose, California, at the 24th Large Installation System Administration (LISA) Conference (November 7–12, 2010). I’ll be speaking at a “Guru Is In” session on Friday, November 12th at 2:00 pm PT. We're going to be in booth 101 all week so, please drop by if you have any questions or suggestions.

Stuxnet Worm Reveals Default Password Vulnerabilities

2010-09-26T08:43:00.044-04:00

Just put it into production... it will be okay

In the last two weeks, I've heard some things which made my blood boil. Such as the recent Stuxnet worm, continued cyber attacks against The Pentagon and NATO, and an article describing past U.S. electricity grid intrusions. Perhaps, I am oversimplifying the root causes but the information available to me makes me want to get up on the soap box to talk about security basics.

I have a love-hate relationship with the idea of computer appliances. On one hand, this pre-installed piece of hardware is ready-to-go. It has already been configured, tested, and you can pretty much guarantee it is going to work when you plug it in. This is a real operational cost savings. On the other hand, I have many security concerns which stem from the “default” nature of their configuration. After all, an appliance usually runs on top of a general-purpose operating system combined with commonly available software such as databases.

After reading an article which identified the primary attack vector as a default password on a programmable logic controller (PLC), I cringed:

“The [Stuxnet] worm was directed at a very popular process controller (Siemens Simatic Programmable Logic Controller) and exploited a zero-day vulnerability in the PLC's WINCC SQL database.

The exploit lay bare the disconnect between the IT and Industrial Control Systems (ICS) communities. This particular PLC (as well as many other ICSs) burned the default passwords in software. The hackers exploited this design to get access to the database.”[1]

If you're an organization which deploys appliances, does the vendor provide the ability to change default parameters such as a password?

When it comes to minimizing the attack surface and applying patches, I hear so many reasons not to remove software and not to apply patches. I've heard that the cost to install software later is more than if they just delivered it in the original installation – besides, there are very few services or packages one can leave off the system.

As Colonel Sherman T. Potter, my favorite character from the television series M*A*S*H, would say, “Mule Muffins.”

I ran the following shell command on two generic Linux server installations to determine how many services were not running and their associated packages:

chkconfig --list |egrep -v ":on" |awk \
'{printf "rpm -q --file /etc/init.d/%s\n", $1}' |sh

In Fedora 12, there were approximately 30 services not running and in openSUSE 11 there were about 40 services. My argument is if the system is performing its assigned tasks and these services aren't running, then remove them before they become inadvertently started or associated tools are exploited.

This is no reflection on an operating system itself; it simply means that operating system distributions typically include many services for maximum interoperability and ease of configuration. Nonetheless, you should take a serious look at what isn't used on your system and remove it.

Every good operating environment should have a digitally signed software repository where system administrators can pull authorized software and patches. This only takes a few seconds and the beautiful thing about Linux packaging is that it resolves dependencies.

So, if you needed to add a webserver (e.g., Apache), all of the associated packages could easily be pulled and installed in your operating system very quickly.

When I read about the state of NATO systems and their reported reluctance to apply system patches, I began to grind my teeth:

NATO's systems are behind the U.S.'s, said one person familiar with U.S. assessments of NATO's systems after a recent trip the deputy defense secretary made there. "The Chinese totally owned them," this person said, adding that NATO hadn't installed many of the basic network security patches, because it had decided some of its computers were too important to ever turn off.^[2]

NATO spokesman James Appathurai denied that the alliance's computers were regularly compromised. However, I didn't hear him dispute the fact that the systems were missing many of the basic security patches.

So, is it just a matter of time? Or have the systems already been comprised but NATO is unaware? Lastly, if the systems are so important, why isn't there any redundancy? Are there no load-balancing or fail-over systems?

How many applications have been deployed in your environment with default passwords? When was the last time they were patched? How many lingering, dormant services reside on your systems?

1. “Opinion: IT needs to help secure industrial control systems” by Joe Weiss of Computerworld (August 13, 2010)

2. “Cyber Attacks Test Pentagon, Allies and Foes” by Siobhan Gorman in Washington and Stephen Fidler in London, The Wall Street Journal (September 25, 2010)

Support for NSA Secure Configuration Guide

2010-09-07T10:01:00.002-04:00

Later this month, we will be releasing Security Blanket v4.0.3 which, among other things, will include support for the National Security Agency's “Guide to the Secure Configuration of RHEL5.”

I’ve been busy defining our development road map for the next six months, delivering webinars, developing presentations, working with customers, authoring white papers, and of course writing code. Needless to say, I’ve been neglecting my blog. As I was gathering information for our technical writer to update the release notes, it occurred to me that I should highlight key features of the next release on the blog. And for my readers who prefer more technical posts, I promise I will resume with more technical ones this fall.

The key areas we focused on for this release were:

Reliability and Performance
Additional Security Guidelines
Software Components - Updates and Dependencies
Baseline Feature Improvements
Reporting

Reliability and Performance

Reliability is one of the most critical operational characteristics of Security Blanket. Its ability to consistently detect, remedy, and reverse previous changes is the core of the product and is our first priority. This is especially challenging given the broad spectrum of operating system versions, subtle nuances between minor revisions of a distribution, and of course, non-standard changes system administrators may make to an operating system. Nonetheless, we’ve worked closely with customers, aggressively tested and retested modules, and performed countless applies and undos to ensure reliability.

In this next release, a greatly improved core engine and state handler will be delivered. For clarity's sake, the purpose of these components are as follows:

The core engine is responsible for loading modules, requesting actions from modules, and generating reports that are to be sent back to the console.
The state handler interacts with the core engine by persisting each module's last action to disk. It also stores and retrieves change records so that a module may reverse (undo) previous changes it has made.

Improvements include a more robust, strict programming interface between the modules and core engine, resulting in more reliable behavior. We also simplified the components by removing unnecessary actions which of course, improved performance.

Additional Security Guidelines

Besides reviewing and improving our modules to ensure compliancy with PCI DSS v1.2.1, we added support for the following three guidelines:

“NSA Guide to the Secure Configuration of RHEL5” (Rev.3 / Oct 2009)
CIS Benchmark for “Mozilla Firefox 3.5”, v1.0.0, January 7th, 2010
“Mozilla Firefox”, DISA STIG Version 4, Release 2, April 23, 2010

This support added another 30 or so security modules as well as a new standard profile called "NSA GUIDE". You can use this profile by itself or customize it to fit your needs. It should be noted that this new profile can be applied to any of our other supported operating systems, too; not just Red Hat 5. The NSA configuration guide has some great recommendations which we've extrapolated to other operating systems.

Software Components - Updates and Dependencies

First and foremost, the console no longer requires a Java Development Kit (JDK). It only requires a Java Runtime Environment (JRE) version 1.6 or later. An earlier version of a console subcomponent needed to dynamically compile Java Server Pages (JSP) hence, the previous JDK requirement. And for security reasons, the console's Apache Tomcat has been upgraded from 6.0.20 to 6.0.29.

As a result of the core engine's rework, the PyXML software package is no longer required. Profile processing no longer uses Document Object Model (DOM) — instead, it now uses XPath.

Baseline Feature Improvements

Security Blanket's baseline feature has been enjoyed by customers since its introduction in version 1.0. Security Blanket deployments continue to expand and diversify, we wanted to make it more robust and flexible.

Its deployment in virtualized environments and various types of hardware have forced us to probe for information in multiple ways. For example, not all virtualization frameworks may use DMI or may not report attached PCI or USB devices in the same manner.

The new baseline feature is now more flexible because you can pick-and-choose what information you want collected. For example, you may only want to inventory /usr/bin and no other system directory — or you may want to collect everything except for your host-based firewall rules. This modular design also allows us to deliver customized probes in future releases which could be tailored for specific types of platforms.

Reporting

Security Blanket has always had an impressive list of reports in various formats. Our Modules Guide details what the modules are looking for and what they will configure. Once you’ve allowed Security Blanket to apply (configure) changes to your system, you can easily look in the log to see exactly what was changed. However, many customers wanted a formal report to provide as evidence for their change management process. So, in this new release "Apply Reports" and "Undo Reports" will be available. And these reports will include "module messages" such as what file was modified and what parameters were added or modified. Take a look at the following excerpt from an Apply Report:

Of course, the logs will still contain more details if you need them. The existing Assessment Reports were enhanced to include some additional messages to help streamline your analysis process:

Conclusion

We are excited by the improvements and new features in v4.0.3. As our customer base grows and diversifies, we continue to grow and improve the product accordingly. If you’re interested in the new features, or have questions or suggestions for upcoming releases, please send us an email at SecurityBlanket@TrustedCS.com.

Configuring Red Hat to Meet PCI DSS

2010-08-27T07:25:00.043-04:00

Configuring the Red Hat® Enterprise Linux® operating system to meet the Payment Card Industry Data Security Standard (PCI DSS) can be a tedious, time consuming, recurring activity if you don't take the time to automate it.

Becoming compliant with PCI DSS involves a lot more than just configuring the operating system as I described in a previous post titled “Operational Security for Non-Techies”. PCI DSS requirement 2.2 is the bulk of a Linux® system administrator’s work as I explained in my “PCI DSS from a Linux Sysadmin’s Perspective” post.

Regardless if this is your first PCI DSS audit or not, in my opinion there are three distinct phases involved:

Develop a security profile.
This involves taking an inventory of all of the necessary technical controls that will be implemented on each operating system to meet the PCI DSS.
Implementing the profile.
This involves implementing the technical controls and TESTING applications to ensure they are still functioning properly. If they aren’t functioning properly, the applications should be modified or find an alternative way to mitigate the risk associated with the missing or inadequate security control.

One compliance manager of a large retailer reports that he mobilizes over 600 workforce personnel and spends millions of dollars every six months in order to be ready for a PCI DSS audit. Some organizations consider the auditing process a “fire-drill” and after passing the audit, will often undo all of the technical controls they put in place.^[1]
Maintaining the security posture.
This means maintaining the same security posture despite software updates and patches, new software, and new personnel.

Using Security Blanket®, several organizations have successfully become PCI DSS compliant and more importantly can regularly monitor and maintain this security posture with ease. Unlike scan-only tools which tell you what you need to manually configure, Security Blanket can automatically configure your operating system and reverse the setting if necessary. This is critical during the implementation and testing phase.

This was certainly the case when we recently teamed up with ezRez Software, Inc., a leading Software-as-a-Service (SaaS) provider for the online travel industry. ezRez Software provides customizable online travel agency functionality to some of the world’s largest airlines, hotels, travel agencies and banking loyalty programs, which require strict compliancy to both the PCI DSS and Center for Internet Security (CIS) benchmarks.

“We found that Security Blanket could assess the security posture of our Red Hat and CentOS servers, identify non-compliant areas and automatically fix them in half the time versus having to do it manually,” stated Jared Wright, Data Center Operations Manager, ezRez Software. “Our team is able to define custom profiles that combine PCI DSS and CIS standards, as well as harden our systems in less than two weeks. Security Blanket’s additional value is the reporting functionality as it validates the system’s security status. The product is now integrated into our standard server deployment process, and we will continue to use the “one click” assessment and automated configuration features to monitor our security posture regularly.” (Read Entire Press Release)

Jared Wright worked closely with Robert Sanders, a senior Security Blanket engineer as they progressed through the three phases. Now that their Security Blanket profiles are in place, maintaining this level of security throughout the year will be seamless. No more “fire-drills” prior to being audited.

Working with ezRez Software was rewarding for our team and we were impressed with their commitment to security. Wherever they could, they implemented security controls exceeding the PCI DSS and CIS. The flexibility of Security Blanket profiles enabled the addition or deletion of security modules from their profiles with just a few clicks.

ezRez Software also provided us with great feedback to streamline the work flow. These enhancements, new reports, and other features will be available in Security Blanket v4.0.3 to be released in September 2010.

If you are interested in automating your lock down process with Security Blanket to achieve your security goals, contact us at SecurityBlanket@TrustedCS.com

1. “For PCI, the Future is Now: How to comply with the global standard without breaking the bank”, Tripwire, Sandra Gitlen, January 2010

Linux Clusters: Keeping nodes consistently and securely configured

2010-08-23T12:59:00.044-04:00

Today, organizations are utilizing technology more and more to meet their business objectives. As such, the survivability of the employed technology as a “system” can directly impact the ability to meet their business objectives.

Survivability is defined as the capability of a system to fulfill its mission, in a timely manner, in the presence of attacks, failures, or accidents. As a discipline, survivability builds on related fields of study such as security, fault tolerance, reliability, and performance.^[1]

To improve system survivability many organizations are using “cluster” technology – which unifies resources to provide higher-availability and better performance. In a traditional, non-cluster architecture, if a single machine fails then the service it was providing is no longer available. To improve a machine's survivability, many vendors build their hardware with redundant power supplies and mirrored storage subsystems to eliminate a single point of failure. So, in the event of a single failure the system continues to operate.

Some organizations don't like the idea of idle resources such as those in a failover cluster. A load-balancing cluster addresses this by distributing the workload across all nodes in the cluster and will continue to operate even if one node fails. This also improves performance and scalability. If the load-balancing cluster's response time has degraded due to increased demands, organizations will typically add more CPU and memory to existing nodes in the cluster. When the node's maximum hardware capacity has been reached, the organization can add more nodes to the cluster thus scaling to meet the operational demands.

In both configurations, it becomes critical that each node's operating system be similarly configured. In a high-availability cluster, a critical piece of software may be required to be a specific version because the previous version had both security and performance flaws. Therefore, it is critical that each node be running the same version otherwise, in the event of a failure, a flawed version of the software could be running.

Likewise, in a load-balancing cluster each node should be similarly configured. In this scenario, users may have different experiences depending on which node serviced their request. This translates into an unreliable service which is very difficult to troubleshoot on a per user basis. Besides software version differences, consider the impact of different operating system parameters such as those which control TCP/IP communications.

Another family of clusters focuses on unifying resources for computational purposes, rather than handling IO-oriented operations such as web services or databases. These types of clusters are generally referred to as “compute clusters”. These clusters require a lot of resources to solve large, complex problems often involving mathematics – such computational simulations include analyzing architecture structures under duress and railway crashes.

Regardless if it is a compute cluster, high-availability cluster, or load-balancing cluster, their nodes must be consistently and securely configured without impeding their performance or reliability.

Security Blanket® Can Help

Security Blanket is the only tool that automatically configures the operating system to meet industry security standards and allows for the easy creation of customized security lock down profiles. Security Blanket is well suited for clusters for several reasons including the ability to track configuration drifts, has flexible reporting capabilities, and most importantly it is cost-effective.

Security Blanket has been successfully deployed in several cluster configurations including Linux-HA Clusters, Beowulf Clusters, and Rocks Clusters. Security Blanket's architecture consists of a single console and dispatchers installed on each node which executes the core engine to perform scans, apply configurations, undo previously applied configurations, and baselines. The dispatcher's virtual memory size is around 61 kilobytes and authenticates and encrypts communications with the console using Transport Layer Security (TLS).

The use of event-driven Simple Object Access Protocol (SOAP) messaging between the console and client is kept to a minimum lessening the impact to tightly-coupled clusters. The console never probes the network for new nodes and only performs actions on nodes when specifically requested. The console gives you the ability to schedule actions, which are then scheduled on each node and will be performed regardless if the console is available or not. Some products require the console to be alive before scheduled actions will be performed, but not Security Blanket.

Other technological safeguards include the ability to configure a node's dispatcher to not accept console requests during certain time frames. This prevents console users from inadvertently disrupting compute jobs or hindering a load-balancing or high-availability cluster during peak hours of operation. Furthermore, the dispatcher can also be configured to ignore console requests if the node's current load average is over a specific threshold. Again, protecting the node from inadvertent requests which could otherwise degrade the node's performance.

Another feature which Security Blanket customers appreciate is the ability to define an “exclusion list”. This is a list of directories which many Security Blanket modules will not traverse. In cluster configurations utilizing shared storage such as a storage area network (SAN), every node should not be required to analyze discretionary access controls on the same multi-terabyte file system. The use of the exclusion list prevents the same files from being analyzed over and over. One Security Blanket customer previously used a scan-only product but was required to remove the node from the grid, otherwise it would take a day or two to perform a scan. When they deployed Security Blanket, they could keep the node within the grid by using the exclusion list.

In addition to performing security lock downs, Security Blanket can be configured to collect information such as all installed software packages, network routing configuration, host-based firewall rules, attached hardware, and access controls and cryptographic hashes on key system files.

The baseline data itself is stored in a single, Extensible Markup Language (XML) structured text file. Through the console, many types of reports are available and in various formats including plain text, HTML, comma separated values for use in spreadsheet applications, and Adobe Portable Document Format (PDF). Moreover, Security Blanket ships with the XML schema definition (XSD) files to aid an organization's custom reporting requirements or to assist with feeding the information upstream.

The real power of the baseline feature is the ability to quickly compare a node's configuration at two points in time. Many organizations discover that their patch management process alters previously established access controls and settings within the operating system. Security Blanket can identify them and correct them immediately. Security Blanket reports not only provide evidence to the change management process but can assist forensic activities. Ironically, the rigid, structured nature of the XML-based reports makes it flexible enough to compare two different nodes. This is ideal for cluster configurations to ensure that all of the installed software packages and file permissions are the same on each node.

In addition to baseline reports, there are assessment reports detailing the node's security posture. A single group assessment report can also be generated in order to view all of the nodes in a cluster within in a single report. Like baseline reports, two assessment reports of the same node or different nodes can be compared to quickly identify changes or differences in the security posture.

The intuitive web-based console makes it easy for administrators to implement and maintain a consistent, securely configured cluster. The console's role-based access controls include pre-defined roles such as an administrator who can manage clients and modify security profiles to a manager who can only view reports. The easy-to-use, streamlined workflow reduces the time it takes for system administrators to learn the console and allows them to work on other mission critical and revenue generating tasks.

1. “Survivable Network Systems: An Emerging Discipline”, Carnegie Mellon University, Software Engineering Institute , 1999

Operational Security for Non-Techies

2010-08-09T09:45:00.046-04:00

There is an abundance of technologies which focus on security while other technologies have security features built into them. It can be overwhelming choosing the right technology to meet your security requirements. Yet, any technology would be rendered useless without operational security. Incorrect implementations, misconfiguration, poor procedures, lack of contingency plans, untrained or undisciplined personnel all contribute to poor operational security.

Last year, a friend of mine was in the planning phase of expanding their business by harnessing the power of electronic commerce — selling their products over the Internet. The business, Adeler Jewlers, is highly reputable and the kind of business which believes today’s quality is tomorrow’s customers. They had concerns that poor online security could jeopardize their customer’s financial information they’ve been entrusted to protect. They hand-craft works of art worn by celebrities and everyday people — but they know nothing of system availability, electronically ensuring confidentiality, or ensuring data integrity. After all, they’re gifted artisans and gemologists not techno-geeks.

They were planning to use a service provider which specialized in electronic commerce. Prior to interviewing the service provider, they asked me for advice and to suggest interview questions because I have many years of experience working in secure operational environments. My first inclination was to ramble on about high-availability, data integrity, patch management but I stopped before I blurted out my first geek word. Instead I discussed it in terms of how operational security impacts their business. If a system is unavailable, you lose business because customers can’t use your website. Now I was speaking this business woman’s language. In the end, she found a service provider that answered her questions and met her very high standards of customer care.

More recently, my company’s Security Blanket® customer base is expanding into various industries. These customers range in size, available resources, and experience but all share a common goal of protecting their systems. Some of these customers are venturing into new areas — faced with new security policies and requirements which they’ll address themselves.

Whether you’re like my friend’s business who will be using a third-party or you will be implementing the operational security yourself, I’ve put together some information to help you on your new venture — or should I say adventure?

Defining the Security Policy

Since my friend’s business was going to be involved with electronic commerce, I suggested to her that the service provider she chose be PCI DSS compliant. The Payment Card Industry’s (PCI) Security Standards Council is an open global forum. Launched in 2006, it is responsible for the development, management, education, and awareness of the PCI Security Standards, including the Data Security Standard (DSS). Not technically a policy; it is however, a requirement set forth by the Payment Card Industry if you want to do business with their card holders online.

Questions:

Depending on your business, what kind of certifications does the service provider possess? If you’re deploying an electronic commerce solution, ask to see evidence of their annual PCI DSS assessment. Typically, providers are certified annually but undergo quarterly assessments to ensure they are maintaining the appropriate level of security. If the service provider says they perform self-assessments, then I would ask how often they perform them and to provide evidence that they were in fact performed — but that’s just me. (PCI DSS § 12.1.2 & 12.1.3)
Personnel — What is the service provider’s hiring policy? What kind of system administrators will be managing your customer’s information? These people have access to your customer’s information and will be an extension of your business so their actions reflect your business. (PCI DSS § 12.7)

The Architecture

The architecture must be clearly documented. For organizations responsible for their own operational security, it is imperative that you document the architecture and routinely update it. I’ve always said that “...the existence of quality documentation separates the men from the boys.” Besides, you can’t securely manage assets if you don’t know they exist or how they’re expected to function.

Service providers adhering to the PCI DSS are required to submit a Report On Compliance (ROC) which includes architectural diagrams. However, it is highly unlikely they would divulge these diagrams to their customers and I wouldn’t expect them to for security’s sake. Which is fine because most of their customers wouldn’t understand what they meant anyhow. Nonetheless, as a potential customer you can still ask them questions about their architecture such as redundancy characteristics which facilitate the continuity of your business in the event of failures or disasters.

Questions:

What kind of physical security is in place? Is the room in which the system hosting your applications restricted? Are only authorized personnel allowed access to your systems? (PCI DSS § 9)
Is the level of service to be provided clearly understood? Does the architecture allow the system to be available to customers 99.999% of the time? If the service provider does not meet the agreed upon level of service, how are you compensated for potential loss of business?

Plans & Procedures

In the real world, would you board an aircraft for a transatlantic flight if you knew neither the pilot or maintenance crew performed their pre-flight checklists? Of course not!

Plans help enforce your policy while procedures are typically part of your plans. While there are many plans, one of the most important is the Business Continuity Plan (BCP). In the old days, we called them Disaster Recovery Plans but with today’s technology and the right architecture, you can continue business despite a disaster. Hence, the term continuity.

An Incident Response Plan should also be in place in the event of a system breach. If customer card holder information is compromised, the incident response plan is to be implemented immediately.

Questions:

Are media back-ups stored in a secure location, preferably an off-site facility, such as an alternate or back-up site, or a commercial storage facility? How is the media transported? How is older media destroyed? (PCI DSS § 9.5 – 9.10)
Is there an incident response plan in place? Is the plan tested routinely? How are card holders informed if there was an incident? (PCI DSS § 11 & 12.9.1)

Configuring & Maintaining the Systems

Configuring systems for production use involves allocating hardware (or more commonly in today’s Green IT environments — provisioning virtualized guests), installing operating systems, configuring the applications, and certifying them for use.

Configuring operating systems consistently, in accordance with security guidelines, is where Security Blanket® excels. Our customers, which include service providers, as well as organizations configuring their own systems to be PCI DSS compliant, consider Security Blanket® invaluable. As a matter of fact, a customer recently stated “...I just completed in 8 minutes what I have been working on for the last several weeks.”

I recently wrote an article describing the activities required by system administrators to become PCI DSS compliant — check out “PCI DSS from a Linux Sysadmin’s Perspective”.

Once the systems are configured and tested, they should be baselined. This is essentially a snapshot of how the system is configured which can be used to compare against a future snapshot to identify any changes to the system. Identifying changes is an invaluable forensics tool for use in Incident Response Plans as well as evidence for change management processes.

If you’re researching a potential service provider, these activities are really under the hood and wouldn’t be of much interest to you. However, I would ask a few questions regarding provisioning and how your customer information co-exists with their other customers.

Questions:

When they provision resources for new customers, is it disruptive to your systems? If your systems become slow, how easily can they provision additional resources to your configuration?
In situations where your customer information co-exists with another business’s customers, to what degree is that information isolated or contained?

Managing Change

Operational systems must have the characteristic of being changeable. They must be able to adapt to new security guidelines, new architectural components, upgraded or replaced technology, and growth due to increased demands on the system.

“Change Management is an IT Service Management discipline. The objective of Change Management in this context is to ensure that standardized methods and procedures are used for efficient and prompt handling of all changes to controlled IT infrastructure, in order to minimize the number and impact of any related incidents upon service.”^[Wikipedia]

Changes are inevitable but the associated risk must be mitigated. For example, consider that new versions of software are always being released. These new versions might include new features or they fix vulnerabilities. What is riskier: running a vulnerable version of software and risk being attacked or upgrading the software to eliminate the vulnerability? Most logical people would choose to upgrade. However, it is imperative that this new version of software be thoroughly tested in an environment which mimics production to ensure the “fix” didn’t break other features. This process of updating software is often referred to as patch management.

Questions:

How do they ensure your systems are protected with the latest patches? When they apply patches, will your systems continue to be available? (PCI DSS § 6.1)
Are all new software updates (patches) thoroughly tested prior to deployment into production? (PCI DSS § 6.3.1)

Summary

Regardless of whether you’re responsible for your own systems or you’ve entrusted someone else, operational security is critical. Even the latest and greatest most sophisticated technology could be hindered or worse rendered useless if poor operational security exists. In my opinion, the number one ingredient to operational security is trained, disciplined personnel. They need the right tools which empower them rather than over complicate their already stressful jobs.

As a family friend, I am honored Mrs. Wendy Adeler Hall asked for my advice but I am a bit disappointed her father, Jorge Adeler, did not afford me the opportunity to meet Anna Kournikova — who recently wore a diamond pendant designed and hand made by him at a charity event hosted by Capitol File.

Anna Kournikova & Jorge Adeler

For my female readers, there are some photographs of Argentinian heartthrob and face of Polo — Nacho Figueras. To read some fun facts and see more photographs, check out the Adeler Jewelers Facebook Fan Page.

I know an engineer’s life is never glamorous but I can live with that. A geek’s idea of a glamorous life is attending technical conferences where our community can get together and share ideas. I have the privilege of speaking in San Jose, California, at the 24th Large Installation System Administration (LISA) Conference (November 7–12, 2010). I’ll be speaking at a “Guru Is In” session on Friday, November 12th at 2:00 pm PT.

If you’ve never been to a LISA conference, I encourage you to come. You will meet some great people in an environment that fosters the exchange of ideas. If you’re not a member of USENIX or SAGE, I encourage you to join even though you’ll probably never see Anna Kournikova or Nacho Figueras at a meeting.

NOTE: All PCI DSS section references are from v1.2.1 of the standard.

Protect Linux Against Overflow Exploits

2010-08-02T07:01:00.040-04:00

An overflow is an anomaly where a program, while writing data to a memory buffer, overruns the buffer's boundary and overwrites adjacent memory—which could be maliciously exploited. While the onus is on software developers to perform proper bounds checking there are some things you can do on an operational system to help protect against code which has been overlooked during development.

Most overflows are addressed during the development process because testing will often uncover erratic program behavior, memory access errors, incorrect results, and unexpected program terminations (crashes). Those which aren't identified and make it into production are often abused by attackers who manage to inject hostile code into this memory.

Program Memory Basics

Kernels are complicated and as such can be difficult to understand. For the purposes of this post, I will try to keep it simple but if any of my facts are incorrect please, add a comment so we all might learn. With that said, running programs establish two key areas in random access memory (RAM): stack and data segment.

The stack is a last in, first out (LIFO) structured linear list. In this structure, elements can be added or taken off from only one end, called the “top”.

The data segment comprises of “data”, BSS, and heap.

The Data area contains global and static variables used by the program that are initialized. This segment can be further classified into initialized read-only area and initialized read-write area.
The BSS segment also known as uninitialized data starts at the end of the data segment and contains all uninitialized global variables and static variables that are initialized to zero by default. (Geek Trivia: Historically, BSS (from Block Started by Symbol) was a pseudo-operation in UA-SAP (United Aircraft Symbolic Assembly Program), the assembler developed in the mid-1950s for the IBM 704.)
The Heap space is dynamic memory. The heap grows due to the dynamic allocation of memory storage for use by a program during the runtime of that program and shrinks when it is released. Dynamically allocated memory exists until it is released either explicitly by the programmer, or by a garbage collector (e.g., Java and Python).

One approach to buffer overflow protection is called executable space protection. This approach prevents the execution of code on the stack or the heap. An attacker may use buffer overflows to insert arbitrary code into the memory of a program, but with executable space protection, any attempt to execute that code will cause an exception.

Hardware Support

Some CPUs support a feature called NX ("No eXecute") or XD ("eXecute Disabled") bit, which in conjunction with software, can be used to mark memory pages of data (such as those containing the stack and the heap) as readable and writeable but not executable.

Generically and on AMD processors, this ability is called NX, while on Intel processors it is called XD. AMD has also marketed this technology as “Enhanced Virus Protection”. In the context of this post, when I refer to the “NX feature” I am including XD.

To determine if your CPU has such support, check the value of flags in /proc/cpuinfo to see if it includes pae (physical address extensions) and nx:

$ egrep '^flags' /proc/cpuinfo
flags      : fpu vme de pse tsc msr pae mce cx8  apic mtrr pge mca cmov pat pse36
clflush mmx fxsr sse sse2 constant_tsc up pni monitor nx

You will get a flags line for each CPU available to your operating system. If your operating system resides on a virutalized guest and you know your hardware supports it but you don't see the appropriate flags, check your virtual machine settings. You will typically see a setting like “Enable PAE/NX”.

It should also be noted that other processors which do not support PAE are AMD K6 and earlier, Transmeta Crusoe, VIA C3 and earlier, and Geode GX and LX. VMware Workstation versions older than 4.0, Parallels Workstation versions older than 4.0, and Microsoft Virtual PC and Virtual Server do not support PAE on the guest.

Linux Kernel Support

If your hardware supports this feature, you should ensure that this protection is enabled in kernels running on 32-bit x86 systems. Other processors, such as Itanium, POWER, and 64-bit x86 (both AMD64 or Intel 64), have included such support since inception and the standard kernel for those platforms already supports the feature.

Most Linux distributions bundle NX support with a PAE-enabled kernel (kernel-PAE). However, some people don't install the Physical Address Extension (PAE) kernel because they think it is just to provide support for physical memory above 4GB. To install the package use either yum(8) or Zypper:

# yum install kernel-PAE

Reboot using the new kernel and then use the -r option to the uname(1) utility to ensure the running kernel is PAE-enabled (you will see the letters ‘PAE’ in the kernel name).

These are the recommended procedures detailed in section 2.2.4.4 of NSA's Guide to the Secure Configuration of Red Hat Enterprise Linux 5.

I have received phone calls from Security Blanket customers attempting to address DISA UNIX STIG PDI “GEN003540 – Disable Executable Stack”. The guideline states:

“Linux kernels must support the NX feature. Red Hat Enterprise 4 and SuSE 9.1 and later do support this feature. This will be a finding on systems prior to the above releases. This is a manual review.”

However, the guideline provides no procedures on how to perform the “manual review”. I hope this blog post helps those of you who must perform this manual check. The upcoming release of Security Blanket scheduled for the Fall of 2010 will include some modules to perform these checks for you.

NX memory protection has always been available in Ubuntu for any systems that had the hardware to support it and ran the 64-bit or 32-bit server kernel. The 32-bit PAE desktop kernel (linux-image-generic-pae) in Ubuntu 9.10 and later, also provides the PAE mode needed for hardware with the NX CPU feature. For systems that lack NX hardware, the 32-bit kernels now provide an approximation of the NX CPU feature via software emulation that can help block many exploits an attacker might run from stack or heap memory.

By default, Red Hat-based and Fedora systems have enabled “Exec Shield”. To read the announcement made by the developer of this kernel patch, Ingo Molnar, click here.

Exec Shield's legacy CPU support approximates (Ingo Molnar's word for it) NX emulation by tracking the upper code segment limit. This imposes only a few cycles of overhead during context switches, which is for all intents and purposes immeasurable.

Add the following lines to /etc/sysctl.conf:

kernel.exec-shield = 1
kernel.randomize_va_space = 1

If you don't want to reboot, use the -w option to the sysctl(8) utility to set these parameters in the current running kernel.

Enabling Exec Shield is a recommended procedure detailed in section 2.2.4.3 of NSA's Guide to the Secure Configuration of Red Hat Enterprise Linux 5.

Prevent Core Dumps

In the event a program experiences memory access errors or unexpectedly terminates (crashes), the kernel can create a core dump file. The most notorious cause of core dumps (and dreaded by developers) is the segmentation violation (SIGSEGV).

The core dump consists of the recorded state of the working memory, including processor registers, which may include the program counter and stack pointer, memory management information, and other processor and operating system flags and information.

Because the core dump may contain sensitive information, many security guidelines recommend preventing them from occurring on production systems. Here are some guidelines which recommend disabling core dumps:

NSA's Guide to the Secure Configuration of Red Hat Enterprise Linux 5, Sections 2.2.4.2 and 2.2.4.2.1
CIS RHEL4 Benchmark (1.0.5), Section 8.10 - Disable Core Dumps
CIS RHEL5 Benchmark (1.1.2), Section 9.10 - Disable Core Dumps
CIS SUSE Benchmark (2.0), Section 8.11 - Disable Core Dumps
DISA UNIX STIG, PDI GEN003500 - Disable Core Dumps

To disable core dumps for all users, add or correct the following line in /etc/security/limits.conf:

* hard core 0

In addition, to ensure that core dumps can never be made by setuid programs, edit /etc/sysctl.conf and add or correct the line:

fs.suid_dumpable = 0

Summary

So if your hardware or virtualization framework supports it, enable it and ensure your kernel is taking advantage of it. Security Blanket has many modules to address the above security guidelines. If you are interested in an easier way to implement NSA's Guide to the Secure Configuration of Red Hat Enterprise Linux 5, keep an eye out for our 2010 Fall release.

Thank you Wikipedia for the hardware specifics!

Sysadmin Day: The Journey of a Geek

2010-07-26T14:56:00.025-04:00

System Administrator Appreciation Day (a.k.a., SysAdmin Day) is right around the corner. This Friday (July 30, 2010) will be the 11th one—and it is always the last Friday of July. In this technological society, most people are absolutely oblivious to the role system administrators play in their lives. I'd like to share my experiences from my humble beginnings to my present day position as a Principal Engineer. Perhaps, by the end of this post you'll have a better understanding of what kind people sysadmins are.

The Early Years

I grew up in Gainesville, Missouri, a small mid-western town, which according to a 2000 census grew to a population of 632. Despite its small size, Gainesville had a wonderful school system with dedicated teachers. As a matter of fact, it was one of the first school systems in Missouri in the 1980's to have computer related classes. This included office automation and even programming on Apple IIe systems.

When I was in junior high, I remember my friend Michael and I would go to the grade school building after school to work on their computers. Michael's father was a janitor for the school and he would let us in after hours. We'd spend hours on those computers.

When I was 14 or 15 years old, a girl gave me a disk which contained an Apple BASIC program which when executed displayed the following text 100 times: "Will you go out with me?" However, being a geek my FIRST response to her was, "You know you could have done that much easier with a for...loop instead of typing that line 100 times. That was crazy—that must have taken you forever!" To this day, I still don't pick up on social queues very well.

Through the school, I was able to learn both Apple BASIC and UCSD Pascal by my junior year of high school. There were no more classes to take so I was allowed to be a teacher's assistant my senior year—which probably translates the same today as it did then: "teacher's pet". I even worked on a program for the school to record student attendance.

Like most kids in the county, if I wanted to enjoy extra curricular activities I would have to get a job or even two. Unlike most kids who spent their money on cars and girls, I spent mine on computers. First, I bought a Commodore 64 which I connected to a black and white tube television. I am color blind so I didn't care. Within a year, I upgraded to a Commodore Amiga and I "acquired" a copy of Lattice C—I was so excited to learn the C programming language. In the 1980's, I didn't have Internet access and there were few books on the language which I could buy. I struggled but within a year or so I was pretty proficient. Many late nights were spent tinkering with this stuff, while my peers were out having fun on dates and playing sports. I focused on implementing math formulas as algorithms which the computer could solve if I provided certain variables. I also read many computer magazines which in those days provided various programs written in BASIC, so I converted some of them to the C language to improve my skills. These days, this process is called "porting".

Despite this dedication to computer science, my other classes suffered and I received poor grades, especially in English. I just was not interested in what I considered "Artsy Fartsy" stuff. Ironically, today I have a real passion for reading. My family didn't have the funds to send me to college and my lousy grades made it very difficult to receive a college loan or even a grant. So I joined the U.S. Navy when I was 18 years old.

I served eight years as a cryptologist at various duty stations throughout the world. Much of that work is still classified. I worked on many different kinds of equipment and I worked with some great people. To drop a few names, I worked on DEC PDP-11(s), AT&T 3B15(s), DEC VAX, Sun, Silicon Graphics' running IRIX, and even a Cray. You haven't lived until you've performed system backups on 9 track reel-to-reel tape systems, loaded punch cards, or manually entered bad sectors into a disk drive's manufacturer's defect table (MDT).

I remember when we thought we had a cutting-edge local area network because it consisted of thicknet interconnecting many IBM desktops running Microsoft Xenix. Yes, I said "Microsoft". This was before Santa Cruz Operation (SCO) emerged and Linux as we know it today, didn't emerge until the early 1990's.

In this configuration, I remember I could login remotely to someone's desktop and send messages to their console. I would login to the desktop of a pretty Air Force female employee and send alerts which informed her that she needed to immediately seek the assistance of a system administrator. Then she would come back to our shop and we could talk to her. Well we tried to talk to her but we mainly just geeked out and she walked away shaking her head and rolling her eyes.

Present Day

In 1996, I was honorably discharged from the U.S. Navy and I started work for a U.S. government contractor in England. Later, I moved to northern Virginia near the company's headquarters. Many years later and several companies later, I still live in northern Virginia. I've worked in both public and private sectors and various industries to include telecommunications, large integrators, and software development companies.

I've worked as a team lead, a manager, and even served as a Director of Operations for a period of time. Despite working as a sysadmin for so many years, I eventually fell back into design, engineering, research, and development. I've always preferred the hands-on aspect of building things and solving problems.

I think it is critical that software developers and architects consider the "manageability" characteristics of any system. Designers are focused on reliability and efficiency among other things however, I don't feel that all software today is as manageable as it could be. Companies spend a lot of money to study human–computer interaction to help improve the end-user's experience which hopefully translates into improved productivity. I would like to see more companies integrating experienced sysadmins into the development process. They offer great advice from proper logging techniques to product deployment into today's operational environments.

Another thing I've learned over the years is that not everyone is cut out to be a sysadmin regardless of how strong their desire and how much effort they put into it. I equate it to the sports nut—no matter how much they KNOW about the game it doesn't make them qualified to be a professional player. There are certain innate traits they must possess. Likewise, sysadmins are a disciplined group of problem solvers. They are also dedicated, loyal, and even a bit arrogant. Some sysadmins are better suited for certain roles within our community. Some are better at providing desktop support while others are probably best left in the data center where they can't interact with end-users. By mid-career most sysadmins begin to gravitate toward their particular field of interest such as storage area networks, large mainframes, distributed environments and cluster technology, and even virtualization.

The most rewarding moments of my career are when I reconnect with a young sysadmin who worked with me several years earlier. They tell me about their current job and the amazing path they've traveled. Their passion and excitement for the work is as it was years earlier. I think it is so important for senior sysadmins to assume the role of mentor. When a new sysadmin arrives, they should not only be shown how the shop does things but the rationale behind it. It is important to nurture those research and problem solving skills early on. Often times, I wouldn't even assist a sysadmin until they had done some initial research like review the manual or researched it on the Internet. If they came to me a second time to ask a previously answered question, I would ask them why they didn't write it down. I encouraged them to be self-sufficient and empowered.

I am fortunate to work with a gentleman whom I've worked with on and off over the last 11 years. He is now a senior sysadmin integrated into our development team. He not only assists with the testing of the product, he provides valuable input to make the product more manageable. I am still in touch with many of my previous co-workers. I think what people will discover is that it is a tight knit community and most seasoned managers know that if a couple of sysadmins leave the company, generally this results in the entire group leaving.

Today, I work for a GREAT company with GREAT people. They really understand and appreciate the value of their engineers and the importance of considering the operational aspects of systems. Last year, they even spent money to produce a humorous video celebrating System Administrator Appreciation Day:

I will be speaking in San Jose, California at the 24th Large Installation System Administration Conference (November 7–12, 2010). I’ll be speaking at a “Guru Is In” session on Friday, November 12th at 2:00 pm PT. If you've never been to a LISA conference, I encourage you to come. You will meet some great people in an environment that fosters the exchange of ideas. If you're not a member of USENIX or SAGE, I encourage you to join.

This Friday be sure to thank a sysadmin. Without them, your Twitter, Facebook, cellphones, text messages, etc, etc... would NOT work.

Lock down Heartburn: Windows to Linux Migration

2010-07-20T10:38:00.022-04:00

Organizations migrating from Microsoft® Windows® to Linux® often have heartburn over their concerns for already established security configurations. Regardless of the reasons for the migration, or even adding Linux to the inventory an obvious question is raised: “Are my current system administrators (sysadmins) capable of managing Linux?” Some Windows sysadmins might be wondering “Are they going to replace me, or train me to handle Linux?” while others might be opposed to the new architecture and are simply inflexible about learning something new.

Many Linux vendors offer some great training, resources, and professional services to assist in the migration. The top three that come to mind are Red Hat, Novell, and IBM. For starters, I would recommend reading IBM's nine-part developerWorks series by Chris Walden on moving your operational skills from a Windows® to a Linux® environment.

The process of lock down (hardening) is difficult, tedious, and time consuming even for an administrator working on an operating system they're familiar with. This process requires knowledge as to where to configure the item and often how to configure the item. This process becomes even more complicated when a system administrator is confronted with an unfamiliar operating system because the “where” and “how” are different. By “how”, I mean the necessary tool or technique required to make the change.

Every good system administrator understands the basic, generic activities required to maintain the systems under their purview. Activities such as user account management, access control, backups and disaster recovery, and so on. More to the point, system administrators understand the rationale in using core, fundamental technologies such as discretionary access controls—that's why I said they only need to know the “where“ and “how” not necessarily the “why”.

Every organization should have a security policy and if you don't—SHAME ON YOU! If you don't have one, create one but don't let technology set your policy. What I mean by this is some organizations will use downloaded how-tos, lock down recipes, configuration guides to configure their systems. Once completed, they document how their systems are configured and call it a “policy.”

A policy is high-level and defines what is required to protect your assets and ensure business continuity. Its constraints are typically legal in nature (e.g., Privacy Act of 1947, HIPPA, or FISMA). Additional constraints can be those imposed by an organization's mission objectives such as system availability and timely service. That's why it is important NOT to set your policy based on your technology's capability. If your technology is insufficient, use something different. Don't compromise. Okay, I will step down from the soap box now...

Enforcing the Policy in Linux

So what you need is the ability to pick a collection of security controls based on your site security policy. For example, pick "Disable USB Storage Devices" or "Minimum Password Length". This is exactly how Security Blanket works. It is policy driven so you don't need to worry about the underlying operating system.

For example, a Windows administrator would disable USB storage devices by editing the registry with the "regedit" utility to set:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\USBSTOR\DWORD "start" value=4

A Linux administrator, would use a text editor to modify the /etc/modprobe.conf file to add something like:

remove usb_storage /sbin/modprobe -r usb_storage

In Linux, you can also edit the GRand Unified Bootloader (GRUB) configuration file and add "nousb" to the end of a bootable kernel definition.

The Solaris operating system is different, too. Confusing? Yes, it is. That's why Security Blanket provides you the ability to create profiles which comprise modules. The module to disable USB storage is aptly named "Disable USB and PCMCIA Devices" and has a simple drop down to choose from: all devices or storage devices only.

Security Blanket's documentation is extensive. The Modules Guide details exactly “where” and “how” the configuration will be made. When Security Blanket scans and applies changes, it logs in great detail what files it is scanning and editing. Security Blanket customers love the detailed logs and documentation. It is also a great way to educate Windows administrators to the ways of Linux.

Security Blanket modules were built to support industry standard guidelines. It is intended to help you satisfy many security guidelines such as those published by the Center for Internet Security (CIS). So if you've configured your Windows systems according to CIS Benchmarks and you'd like to apply similar CIS Benchmarks to Linux—Security Blanket is the right choice. Right out of the box Security Blanket has many such profiles. If you'd like to make a copy of the CIS profile and customize it to meet your site security policy, it is only a few clicks away.

If you'd like more information on Security Blanket, visit our website or send an email to TCSSBSales@TrustedCS.com.

PCI DSS from a Linux Sysadmin's Perspective

2010-07-10T06:27:00.123-04:00

A cursory glance at the PCI DSS might lead one to believe that the majority of work required to comply with the standard belongs to network, database, and application/middleware administrators as well as software developers.

Of course, every system administrator knows that there is always a great deal of work required by them anytime an application or service is deployed. During my recent review of the PCI DSS v1.2.1 to ensure our Security Blanket® modules support the standard, I identified several requirements which will significantly increase a sysadmin's workload.

There is no doubt that the explosion of the Internet has facilitated electronic commerce—changing the way we do business. “E-commerce” consists of the buying and selling of products or services over electronic systems such as the Internet and other computer networks.¹ The easy use of credit cards online has further fueled this commerce but requires security controls to protect consumers.

The Payment Card Industry's (PCI) Security Standards Council is an open global forum. Launched in 2006, it is responsible for the development, management, education, and awareness of the PCI Security Standards, including the Data Security Standard (DSS).²

Getting Started

Obviously, the first step is to download the PCI DSS specification. I will save you a little time by directing you to requirement 2.2. This will keep you quite busy while the rest of your team takes care of their parts.

PCI DSS requirement 2.2 states, “Develop configuration standards for all system components. Assure that these standards address all known security vulnerabilities and are consistent with industry-accepted system hardening standards.”

The phrase “industry-accepted system hardening standards” should get the gears in your head turning. The test procedure (2.2.a) for this requirement states “Examine the organization’s system configuration standards for all types of system components and verify the system configuration standards are consistent with industry accepted hardening standards—for example, SysAdmin Audit Network Security (SANS), National Institute of Standards Technology (NIST), and Center for Internet Security (CIS).”

If you choose to follow the CIS Benchmarks, you will soon discover that you've got a lot of work ahead of you. The Center for Internet Security (CIS) is a non-profit enterprise that helps organizations reduce the risk of business and e-commerce disruptions resulting from inadequate technical security controls. There are many benchmarks including:

CIS Red Hat Enterprise Linux 5.0 - 5.1 Benchmark v1.1.2
CIS Debian Benchmark v1.0.0
CIS SUSE Linux Enterprise Server 9/10 Benchmark v2.0.0
CIS Solaris 10 Benchmark v5.0.0

These detailed benchmarks will require you to assess your systems, develop an implementation plan, test applications after you've implemented the controls, and use a back-out plan in case an implemented security control broke an application. Fortunately, this is what Security Blanket® does very well. Security Blanket will not only assess your systems but will apply the appropriate changes to your operating system and has a quick undo feature if you find yourself in trouble.

At this point, your fellow administrators might be giggling about the amount of work you just found yourself waist-deep in. If you want to wipe that grin off their face, inform them that CIS has many benchmarks for their world to include:

CIS MySQL 4.1/5.0/5.1 Benchmark v1.0.2
CIS Oracle Database 11g Benchmark v1.0.1
CIS Apache Tomcat Server Benchmark v1.0.0
CIS Apache HTTP Server 2.2.x Benchmark v3.0.0

System Minimization

Those of you who follow this blog or have listened to one of my webcasts are probably tired of hearing me chant “If you don't need it, disable it or remove it!” like a mantra. I even posted Minimize Attack Surfaces which describes how to check for system services configured to start during boot.

The Twenty Critical Controls for Effective Cyber Defense: Consensus Audit Guidelines (CAG), Critical Control 13: Limitation & Control of Network Ports, Protocols, and Services is all about reducing your attack surface. Likewise, the PCI DSS has similar requirements:

2.2.2 Disable all unnecessary and insecure services and protocols (services and protocols not directly needed to perform the device’s specified function).
2.2.4 Remove all unnecessary functionality, such as scripts, drivers, features, subsystems, file systems, and unnecessary web servers.

Patch Management

As every system administrator knows, the process of patching or upgrading software to newer versions to address vulnerabilities is a seemingly never ending task. This process however, raises two important questions: (1) Will newer software versions behave differently resulting in broken applications? (2) Will the installation of new software alter previously implemented security controls (i.e., security permissions and related configuration settings)?

PCI DSS requirement 6.1 states, “Ensure that all system components and software have the latest vendor-supplied security patches installed. Install critical security patches within one month of release.” The authors of the DSS recognize the need to sufficiently test your applications with the newer software prior to production deployment—hence, the phrase “...within one month of release.” This should also remind management to provide adequate test systems and resources to facilitate such testing. A few extra servers for testing is much cheaper than downtime or loss of customer confidence.

Detecting configuration changes and correcting deviations is another area which Security Blanket excels. Many existing customers love the fact that Security Blanket brings their systems back into compliancy after applying patches.

Configuration Drift

Throughout the life cycle of every system configurations vary due to new applications, upgraded software, and change in personnel. A more sinister source of change might be a result of system compromise. Regardless of the source of change, you must know that something has changed. A manifestation of system changes may be application malfunctions or performance degradation. Even worse are those changes which have not manifested in anyway—instead attackers are silently mining your customer's data. That's why it is so critical to detect system changes.

One method of detection is described in PCI DSS requirement 11.5 which states “Deploy file-integrity monitoring software to alert personnel to unauthorized modification of critical system files, configuration files, or content files; and configure the software to perform critical file comparisons at least weekly.”

This is another area which Security Blanket excels. In addition to capturing SHA-1 fingerprints of critical files, it also let's you know when discretionary access controls (permissions) have changed. The baseline component of Security Blanket also captures information on all installed software packages, network routing tables, attached hardware devices, host-based firewall rules, and other vital pieces of information.

Another nice feature of Security Blanket is the ability for you to compare not only the same system at two points in time to identify changes but the ability to compare two different systems to identify changes. This is critical to ensure that all nodes in a cluster are similarly configured or a more simpler load-balanced or fail-over configuration.

Summary

This post only scratches the surface of what is required of Linux administrators responsible for systems which must adhere to the PCI DSS. The standard is detailed and requires good planning and a commitment to maintain the final achieved security posture. Security Blanket can assist organizations in configuring and maintaining their operating system level security. For for more information, check our website.

1. "Electronic Commerce". Retrieved from Wikipedia July 10, 2010 (http://en.wikipedia.org/wiki/Electronic_commerce)
2. "About the PCI Security Standards Council" (https://www.pcisecuritystandards.org/about/index.shtml)

Broken Links in Linux File Systems can be a Security Risk

2010-06-11T08:26:00.059-04:00

BROKEN LINKS in Linux file systems are not just annoying — they can also be a security risk. In a previous post, I discussed the potential dangers of unowned files and in this post I will talk about those annoying, resource consuming broken links usually considered simple file system “lint”.

I recently spoke to a Security Blanket® customer and they asked me, "Why does Security Blanket report on these broken links?" I responded with a typical, technical explanation of...

"It [Security Blanket] was trying to determine the file's existing access controls by using stat(2) not lstat(2). This call was unsuccessful because the target file was non-existent therefore, we want you to be aware of this 'lint'."

Although I could not see their face, I am sure their eyes were rolling at my somewhat cryptic response. Later that day, I contemplated the existence of broken links and realized they are a potential security risk — in the form of a Trojan Horse.

Common Uses of Links

First of all, what is a link? Most desktop users use "shortcuts" but in the underlying file system they are actually links. For example, "File A" resides in the document folder and "File B" is created on the desktop but actually points to "File A".

Another common implementation is for backward compatibility. Consider an application which was compiled and associated with a specific shared library. When a new version of the library is installed, the older application still looks for the previous shared library. The new shared library has all of the objects as before but to keep the older application from barfing a link is created. Consider the following:

# cd /usr/lib
# ls -l libxml2.so*
lrwxrwxrwx 1 root root      16 Jun  1 12:08 libxml2.so.2 -> libxml2.so.2.7.1
-rwxr-xr-x 1 root root 1385248 Aug  7  2009 libxml2.so.2.7.1

The earlier libxml2.so.2 library is linked to libxml2.so.2.7.1 — indicated by the "l" in the first character of the mode field and the "->" in the name field.

Another scenario links entire directories. This is typically done by system administrators to compensate for legacy applications or even an inability to properly configure applications to utilize existing directory structures. For example, consider a homegrown application which stores its data in /usr/local/myhomegrownappdata. Now, there is no more space on the /usr file system and there is no more disk space to create a new file system. However, there is plenty of space on /home. Unfortunately, the homegrown application's data directory is hard-coded so the system administrator can't configure it to use a different path. So, the quick-and-dirty is to create the directory /home/myhomegrownappdata and link /usr/local/myhomegrownappdata/ to it. Ugh. The horror.

A Trojan Horse — “The Trouble With Tribbles”

A broken link is created when the source (target) file is removed but the link remains. Consider the above example of "File A" in the user's document folder and "File B" on the desktop which is linked to it. If "File A" is removed, "File B" is a broken link. Let's see this action at the command line:

greenlantern:~# cd /tmp
greenlantern:/tmp # touch jamie
greenlantern:/tmp # ln -s jamie adams
greenlantern:/tmp # ls -li jamie adams
2859232 lrwxrwxrwx 1 root root 5 Jun 11 09:22 adams -> jamie
2859231 -rw-r--r-- 1 root root 0 Jun 11 09:22 jamie

I created a file "jamie" with the touch(1) command and then created a symbolic link from "adams" to "jamie". Then I used the ls(1) command with the -i option to show their inodes (first column). Now, if I remove the "jamie" file:

greenlantern:/tmp # rm -f jamie
greenlantern:/tmp # test -r adams; echo $?
1
greenlantern:/tmp # find -L . -type l -ls
2859232    0 lrwxrwxrwx   1 root     root            5 Jun 11 09:22 ./adams -> jamie

After removing the file jamie, the return code ($?) of the test(1) command shows that I am unable to read "adams" because it was linked to "jamie". A useful technique to find all broken links, is to use find(1)'s -L option as shown above.

Now, a Trojan Horse could potentially be introduced if a target file "jamie" is created in the same path. Below I did just that:

greenlantern:/tmp # echo "Dangerous Stuff" > jamie
greenlantern:/tmp # ls -li jamie adams
2859232 lrwxrwxrwx 1 root root  5 Jun 11 09:22 adams -> jamie
2859233 -rw-r--r-- 1 root root 16 Jun 11 09:32 jamie
greenlantern:/tmp # test -r adams; echo $?
0

Note that the previous file "jamie" had an inode of 2859231 but this new file's inode is 2859233. The link doesn't care because it is path-based. Using the test(1) command I can read the link again. Even if this is a result of non-malicious actions, the fact remains that the user or application is no longer using the original file.

Systems using Security Enhanced Linux (SELinux) in enforcing mode would have an advantage since the mandatory access controls applied to the file is inode-based. Despite discretionary access controls on a newly created file, it would not result in access to the process following the link — until the appropriate security contexts have been manually re-established.

Where do these troublesome “Tribbles” come from?

When installing software, many packages contain a "post-install" and a "pre-uninstall" routine. This means, when a package is installed the "post-install" routine may add additional links to provide backward compatibility. However, since these links were created during the "post-install" they are not part of the package's manifest. When packages are removed, only files in the manifest are deleted unless there is some special clean up functions in the "pre-uninstall". However, few packagers include such clean up routines.

Then there is the notorious sharing-caring community of users which may all link to one user's resource files. Then one day, that user is removed and the community is left with broken links.

Regardless of their origins, broken links are not only annoying little Tribbles which consume file system resources they also pose a security risk in the form of a Trojan Horse. A good system administrator will try to control these little Tribbles by using best practices, common sense, and educating their users.

Security Blanket's baseline feature can assist in detecting removed files or ones which have been replaced by using cryptographic hashes. For more information, visit our website.

Tool Talk Webcast: Operating System Security ... a "Fireside Chat"

2010-05-23T19:02:00.018-04:00

Check out this FREE webcast hosted by SANS Tool Talk !

What is a Tool Talk?

"SANS Tool Talks are an opportunity for you to hear from Information Security Vendors. At SANS we believe that you cannot accomplish Information Security tasks without tools. A surprising number of security professionals have no idea what technology is available in the marketplace. Tool Talks are designed to give you a solid understanding of a problem, and how a vendor's commercial tool can be used to solve or mitigate that problem."

Overview of the Webcast

Featuring: Ed Hammersla and Jamie Adams

Operating System (OS) security is key to building a strong foundation for enterprise security. OS lock down or system hardening is a recommended best practice when securing the enterprise, yet most organizations are reluctant to fully lock down their systems for fear that their applications may fail, or system performance will degrade.

Whether you have 5 servers or 500, implementing and maintaining OS security is a time and resource intensive task. Listen in on this insightful discussion between two veteran security experts on best practices for effective and efficient OS lock down using Security Blanket, an automated OS lock down tool, by Trusted Computer Solutions.

Speaker Bios:

Ed Hammersla is Chief Operating Officer of Trusted Computer Solutions (TCS), a proven leader in providing cyber security solutions which prevent government and commercial organizations from being compromised. In this role, Mr. Hammersla applies over 30 years of industry experience implementing strategies which assist the company in providing the highest quality security solutions for mission critical needs. Mr. Hammersla is considered an industry thought leader in how to deliver information securely and quickly to those that need it the most. As a frequent contributing speaker at technology conferences, Mr. Hammersla has played a key role in evangelizing the need to share information across multiple security domains long before it gained widespread public attention.

Jamie Adams is a Principle Secure Systems Engineer for Trusted Computer Solutions. As a lead engineer, Jamie has been instrumental in the development and ongoing evolution of the Security Blanket product. Jamie has over 20 years experience as a software developer and systems engineer for classified and unclassified systems. He has been involved in the design, deployment, operation and accreditation of large scale, highly available mission critical systems in both the public and private sectors. His deep background with applying security guidelines throughout enterprise system environments has provided Jamie with a wealth of knowledge regarding best practices for reducing vulnerabilities and risk.

Surviving the DISA UNIX STIGs

2010-04-30T13:23:00.031-04:00

Understanding the DISA UNIX STIGs is difficult for first-timers and sends chills down the spines of system administrators who have used them before. They are probably the most detailed set of security controls available which apply to a wide variety of operating systems.

The biggest complaints from system administrators are that the STIGs are time consuming and require benign or seemingly useless settings. Personally, I think they are great. They are worth the time because they will improve your security posture and most system administrators learn a great deal during the implementation process.

The Security Technical Implementation Guides (STIGs) are published by the U.S. Defense Information Systems Agency's (DISA) Field Security Office (FSO). There are three pieces:

UNIX STIG V5R1 (April 2006) — This is the main document and it gives plenty of explanations, rationale, and general information.
UNIX Security Checklist (published quarterly) — details the requirements by line item to include what to look for in configuration files as well as commands to be executed.
System Readiness Review (SRR) Scripts (published quarterly) — scan the operating system for compliancy. To download, you must have DoD PKI access.

Getting Started

Too often system administrators download the SRR scripts, run the scan, and then perform the manual review. Then they bounce between editing configuration files and re-running the scan. Before you run any script on your system, there are some things I would recommend.

First and foremost, make sure you have a Disaster Recovery Plan or some sort of Business Continuity Plan (BCP). The plan must contain detailed procedures for performing backups and recovery. Wherever possible, try to have on-line, off-line, and off-site backups. On-line could include storage mirroring technologies and off-line could be some sort of media which isn't connected to equipment. If at all possible, perform encrypted backups and then store them in a media safe on-site and off-site. The off-site backups could be as simple as taking the off-line media outside of your data center and preferably not in the same building.

The biggest tip I can offer is to REMOVE ALL SOFTWARE WHICH IS NOT NEEDED. This is where you gain great insight to your system's architecture because you will learn what components rely on what. For example, if you are not using any features or utilities of Samba — remove it. I hear some administrators complain about the results of an SRR because it found problems with their Samba configuration. Their argument is "We aren't even running the daemon!" If that's the case, simply remove the package — it will reduce your findings and the amount of software you must constantly patch, upgrade, and report on.

Lastly, have a STRONG, aggressive patch management plan. Make sure you have a complete backup of your system and patch everything you can. Next, download the "UNIX Security Checklist" and familiarize yourself with Section 3 of the document.

Once you've done all of the above, feel free to perform your first SRR scan — which will take about 3 to 5 minutes to complete.

Locking Down and Testing

Once the Start-SRR script and manual review scripts have been run, you are ready to begin configuring the system to address failures. This is where it can get confusing for some people.

Each test result is bound to a line-item, or more correctly a Potential Discrepancy Indicator (PDI). For example, the SRR reported that PDI GEN003600 failed:

Beginning GEN003600 on 04/25/10 at 06:38.
Network parameters are not securely set.
Finished GEN003600 on 04/25/10 at 06:38.
::::::::::::::
/root/Desktop/Script.February/centos-5_2-i386.localdomain/GEN003600.Examples
::::::::::::::
GEN003600: net.ipv4.tcp_max_syn_backlog tunable is set less than 1280.
::::::::::::::
/root/Desktop/Script.February/centos-5_2-i386.localdomain/GEN003600.log
::::::::::::::
::::::::::::::
/root/Desktop/Script.February/centos-5_2-i386.localdomain/GEN003600.Result
::::::::::::::
PDI Number: GEN003600
Finding Category: CAT II
Reference: UNIX STIG: 3.20.5
Description: Network parameters are not securely set.
Status: Open

For example:
GEN003600: net.ipv4.tcp_max_syn_backlog tunable is set less than 1280.

Now you must figure out how to configure the system so it will pass subsequent SRR scans. Section 3 of the Checklist document simply states the following:

GEN003600 – Network Security Settings
Perform the following to ensure the network security settings are enabled for each operating system. The command is listed with the expected response below it.
....

Linux
# sysctl –a | grep net.ipv4.ip_forward
0
# sysctl –a | grep net.ipv4.tcp_max_syn_backlog
1280
# sysctl –a | grep net.ipv4.conf.all.accept_source_route
0
# sysctl –a | grep net.ipv4.icmp_echo_ignore_broadcasts
1
...

Unfortunately, the guideline does not tell you how to set the value correctly. Most system administrators would check the manual page on sysctl(8) and learn that you can use the -w option to set the parameter in the running kernel. However, if you reboot the system the kernel setting would be lost because it was not placed in the /etc/sysctl.conf file.

Situations like this are compounded when dealing with multiple operating systems which require different methods to implement the security setting.

Once the setting has been configured, test the applications on the system. If any problems arise, the change must be reverted and you must document why it was not implemented. You must also document any reasons or other configurations which mitigate or reduce the risk of not configuring the system accordingly.

There are 368 PDI(s) so, you better get yourself a cup of coffee.

Maintaining

Once the system has been configured and tested, it should be baselined and its current configuration well documented. Each time new software is introduced or existing software is patched, another baseline should be taken and compared to the first. Identify any changes and correct the security settings to ensure you maintain the same security posture. As a matter of fact, DISA recommends running weekly baselines to detect any changes in the system.

Automation

This laborious process is one of the reasons Security Blanket® is becoming so popular. Security Blanket will automatically configure your system so issues such as GEN003600 are completely transparent — regardless of the operating system it is running on. Security Blanket's engineering team has already conducted the research for you, documented where changes will be made in your system, as well as provide detailed logging when it does make changes to your system. You can perform assessments or apply the configurations which take 30 seconds or less on most systems. Security Blanket also has the ability to quickly undo settings.

Security Blanket addresses all of your baselining requirements, too.

There are numerous modules which you can easily include in a profile which is used to enforce your security policy. There are out-of-the-box profiles to address guidelines such as the DISA UNIX STIGs and CIS Linux Benchmarks. With a simple click you can create a copy of those profiles and customize them to suit your site's needs. If you're skeptical, check out the free trial.

openSUSE and SUSE Linux Meets or Exceeds DISA and CIS Security Guidelines

2010-04-23T09:55:00.003-04:00

While developing the Security Blanket® lock down product for SUSE, our engineering team has confirmed that all of the appropriate technical controls required to meet the guidelines were present. In addition, we identified some deficiencies in the guidelines. The guidelines, intended for earlier versions of the operating system, were not aware of some superior technologies available in version 11.

I wrote a white paper detailing the findings.

Novell® SUSE® Linux® and openSUSE® are growing in popularity in many different industries. One possible reason for the growing popularity is its cost-effective use of scalable Linux solutions combined with Novell’s commitment to interoperability and usability. On November 3, 2006, Novell signed a landmark agreement with Microsoft®. Part of that agreement was focused on the improvement of the interoperability of SUSE with Microsoft Windows®. One of the tenants of the openSUSE Project is to “...explicitly look[s] beyond the technical community to the broader non-technical community of computer users interested in Linux.” Recently SUSE Linux was chosen as the primary operating system (OS) by Burton Snowboards, Keller Homes (a nationally ranked home builder), and Sherwin-Williams nationwide retail stores.

Download the entire whitepaper »

Is McAfee's blunder a result of insufficent testing?

2010-04-22T09:57:00.006-04:00

In light of the recent McAfee blunder, I wonder if the rush to deliver new signatures to thwart a zero-day attack resulted in insufficient testing. The fact that most companies have some sort of automatic update enabled does not give enough time to test the updates themselves.

Would other, non-signature, network-based technologies have given more time? More time for companies to do their own minimal or subset of testing first?

Why do we still rely on older technologies and techniques when attackers don't?

Traditional Intrusion Detection Systems (IDSs), based on rules or signatures, have the problem of signature availability: it can take days or weeks to update the IDS with information about a new vulnerability or attack. They also have a problem with scale: new attacks are appearing at an increasing rate. Attackers have gotten smarter and adopted the idea of polymorphism – changing one byte in an attack renders it undetectable by traditional detection methods. Smarter signature-based systems can still be fooled by more sophisticated polymorphic techniques. IDSs based upon Anomaly Detection (AD), on the other hand, do not require an explicit description of each new malicious behavior, and adapt themselves to changing network conditions.

Read the full whitepaper »

Keeping Linux File Systems Clean

2010-04-18T04:42:00.032-04:00

SYSTEM ADMINISTRATORS ROUTINELY "CLEAN" their file systems by removing unused files and organizing files into directories. The term "clean" is also associated with performing file system checks to ensure structural integrity with tools such as fsck.ext3(8). In addition to these critical activities, you should routinely "clean" file system access controls — ownerships and permissions.

Unowned Files and Directories

An unowned file or directory is one in which no owner or group has been assigned. All accounts and group have a numerical identifier associated with them — referred to as the user identifier (uid) and group identifier (gid).

For example, I have an account called darth (uid=502) and it belongs to two groups: darth (gid=502) and vader (gid=503). I created the file /tmp/deathstar which is owned by darth and assigned group vader:

[root@localhost tmp]# id darth
uid=502(darth) gid=502(darth) groups=502(darth),503(vader)

[root@localhost tmp]# touch deathstar
[root@localhost tmp]# chown darth:vader deathstar 
[root@localhost tmp]# ls -l deathstar 
-rw-r--r--. 1 darth vader 0 2010-04-17 17:30 deathstar

Once the group vader is deleted from the system, the file no longer has a group assigned to it. The ls(1) command shows the number 503 because it no longer has a group mapped to gid 503. When I delete the account darth, the number 502 is displayed because there is no longer an account with uid 502:

[root@localhost tmp]# groupdel vader
[root@localhost tmp]# ls -l deathstar
-rw-r--r--. 1 darth 503 0 2010-04-17 17:30 deathstar

[root@localhost tmp]# userdel -r darth
[root@localhost tmp]# ls -l deathstar
-rw-r--r--. 1 502 503 0 2010-04-17 17:30 deathstar

This may not seem like a big deal but if a new account is created with uid 502 it will own the file.

Unowned files and directories can also show up on your system when you extract an archive created on a system which has accounts your system does not have. Most notorious is the use of the tar(1) command as root without the -o option.

To locate all files on your system with no user or group assigned use the following command as root:

# find / \( -nouser -o -nogroup \) -print

Uneven Permissions

Uneven permissions are when a group has more permissions than the owner or when the other (world) permissions are greater than the group's or owner's. For example, if a file has "-r--rw-r--" (octal 0464) the group has write permission but the owner does not.

From right-to-left, the permissions map to other-group-owner. Uneven permissions can be tested using simple mathematics to extract each digit and a simple test expression to ensure each digit is greater than the one to the right. Below is an example Python script:

#!/usr/bin/python

import os

for root, dirs, files in os.walk('/'): 
    testFiles = []
    for name in files:
        fullName = os.path.join(root, name)
        testFiles.append(fullName)
   
    for name in dirs:
        fullName = os.path.join(root, name)
        testFiles.append(fullName)

    for name in testFiles:  
        fileMode = os.lstat(name).st_mode
        usrMode = (fileMode / 8**2) % 8 
        grpMode = (fileMode / 8) % 8 
        othMode = fileMode % 8 
        
        if othMode > grpMode or othMode > usrMode or grpMode > usrMode: 
            print "Mode=%d%d%d %s" % (usrMode, grpMode, othMode, name)

Uneven permissions are usually created by the misuse of the chmod(1) command, a bad umask, or a program which doesn't convert from decimal to octal correctly when assigning permissions.

Automation

Besides unowned files and uneven permissions, keep an eye on setuid and setgid files, contents of home directories, and excessive permissions on binaries (executables) and libraries. Many system administrators go to a lot of trouble to tighten permissions only to have them changed after packages are updated.

Security Blanket® can detect these problems AND correct them — consistently and automatically. There are several hundred security modules which you can easily include in a profile which is used to enforce your security policy. There are out-of-the-box profiles to address guidelines such as DISA UNIX STIGS and CIS Linux Benchmarks. With a simple click you can create a copy of those profiles and customize them to suit your site's needs. If you're skeptical, check out the free trial.

Fine-grained Access Control on Linux File Systems

2010-04-15T18:56:00.047-04:00

IMPLEMENTING FINE-GRAINED ACCESS CONTROLS is easy when you understand the basics. All system administrators are familiar with traditional user-group-other permissions applied to files — at least I certainly hope so! Discretionary Access Controls (DAC) are a fundamental security mechanism and POSIX.1e Access Control Lists (ACL) provide administrators the ability to grant access to additional users and groups without having to grant access to other (a.k.a, the world).

In my opinion, it's easier to understand ACLs if you can see a practical use for them. Let's say you have an application that monitors system logs but the monitoring application doesn't have root access. The log's owner is root and the group is assigned root — the permissions are 0600 which means only the root account can read the log file and write to the log file.

[root@spartacus ~]# ls -lh /var/log/messages
-rw------- 1 root root 185K 2010-04-16 07:33 /var/log/messages

The monitoring application runs as the analyzer user account. However, in order to grant access to this account we would have to either add analyzer to the root group or grant other (world) permissions to the log. The first option is a bad idea because you don't want any account besides root to belong to the root group. The second idea is bad because once you grant permission to other — any account will have the same permissions!

We only want to grant the analyzer account permission to read the log file — plain and simple. We can use the setfacl(1) command to accomplish this. First, if I switch to the analyzer account I am unable to read the log file:

[root@spartacus ~]# su - analyzer
[analyzer@spartacus log]$ tail -1 /var/log/messages
tail: cannot open `/var/log/messages' for reading: Permission denied

With the setfacl(1) command, we can grant access to the anlayzer account by adding an ACL entry:

[root@spartacus ~]# setfacl -m user:analyzer:r-- /var/log/messages

The getfacl(1) command now shows an additional entry besides the base ones. Also, the ls(1) command now shows a plus sign at the end of the permissions column.

[root@spartacus ~]# getfacl /var/log/messages
# file: var/log/messages
# owner: root
# group: root
user::rw-
user:analyzer:r--
group::---
mask::r--
other::---

[root@spartacus ~]# ls -l /var/log/messages
-rw-r-----+ 1 root root 189403 2010-04-16 07:33 /var/log/messages

When we switch to the analyzer account, we can now read the log file.

[root@spartacus ~]# su - analyzer

[analyzer@spartacus ~]$ tail -1 /var/log/messages
Apr 16 07:33:28 spartacus auditd[1774]: Audit daemon rotating log files

To remove ACL entries, use the -x option to the setfacl(1) command.

You can also establish default access controls on directories which is useful if you want all files in a directory to inherit a certain set of permissions.

Linux File Systems

2010-04-12T12:00:00.023-04:00

LINUX FILE SYSTEMS are an essential operating system resource. Modern file systems and disk drive technology are robust and reliable — so, most administrators put little effort into planning or worrying about them once the operating system is configured. This makes me both smile and cringe.

I smile because the advancements in technology facilitate out-of-the-box reliability and performance for common installations regardless of the system administrator's skills. The implementation allows applications which utilize them to be reliable and require little day-to-day maintenance.

I cringe because when it comes time for a system administrator to do proper planning and implementation of a very large or unique solution with a finite set of resources, many administrators lack the skills. Storage Area Networks (SAN), advanced server hardware, and virtualization have provided a seemingly unlimited pool of resources.

When it comes to file systems, there are several different types. For the purposes of this blog post, I will focus on disk file systems which include ext3 and ext4. However, there are other types such as special and network-based file systems.

I'd like to offer some thoughts on file system planning and implementation. They're intended to get you thinking about some important aspects of using file systems.

Planning — The Layout and Creation of File Systems

Defining a file system layout involves considering the use of each file system in the operating system. Most system administrators will simply choose the default which assigns the entire disk partition as one, big monolithic file system (/).

Consider creating separate file systems for data which is constantly growing and shrinking such as /tmp, /var/log, and /var/audit. If you are going to have a large number of Linux user accounts, consider a separate file system such as /home — this will isolate non-privileged users which could potentially fill up the root file system (/).

Next, think about the size and amount of data to be stored on the file system. For example, if the file system will be dedicated to a database you may have fewer files but can be very large. If the file system will be comprised of many smaller files, then consider adjusting the inodes-per-bytes ratio. I've seen systems where the "df -h" command reports plenty of space however, the "df -ih" command reports no more space for inodes. If your file system needs change, some parameters can be adjusted later with tune2fs(8) command.

Implementation — Mounting File Systems

Once the file system is created, it is "mounted" to make it available to the operating system. Typically, file systems are mounted using its options specified in the /etc/fstab. Again, the majority of default mount options suffice for most environments.

Consider file systems which may be used for user home directories. You may want to prevent these users from executing binaries they've copied into their home directories — use the 'noexec' and 'nosuid' mount options on their file system to prevent this. Another useful location for the 'noexec' and 'nosuid' options would be a file system dedicated to logging.

File systems dedicated to serving static content such as web pages and images are great candidates for the 'noexec' and 'nosuid' options as well. If the pages are seldom updated, consider mounting the file system as read-only using the 'ro' option.

By default, file systems such as ext2 and newer support POSIX.1e Access Control Lists (ACL). These ACLs provide fine-grained access control beyond the base user-group-other permissions. However, many novice administrators do not understand the technology so excessive permissions could be hidden from them. If you are not comfortable or well-versed in POSIX ACLs, I recommend mounting file systems with the 'noacl' option to disable support for them. Some security professionals might disagree with me but in my opinion a misuse of such technology could be very dangerous.

Identifying Network Ports in Linux

2010-03-24T11:39:00.152-04:00

ONE OF THE MOST CRITICAL SKILLS a system administrator should have is managing network ports on their servers. This includes knowing what network ports are open, correlating them to a running process, restricting access to that port, and disabling (closing) the port if it is not needed.

Security Blanket® customers come from a variety of industries and their system administrators range from novice to expert. Recently, I assisted some customers who were seeing error messages such as "bind failed: port already in use". After explaining that the console port they've chosen was already assigned to another process and they should choose another port or kill the other process, I decided this would be a good blog post. Of course, these are only a handful of techniques I've used over the years so, please comment on this post and let us know what techniques you use.

First of all, let's see what one of my favorite Internet resources says about ports. Wikipedia says:

"In computer networking, a port is an application-specific or process-specific software construct serving as a communications endpoint used by Transport Layer protocols of the Internet Protocol Suite, such as Transmission Control Protocol (TCP) and User Datagram Protocol (UDP). A specific port is identified by its number, commonly known as the port number, the IP address it is associated with, and the protocol used for communication."

It should be noted that ports range from 1 to 65535 and ports 1–1023 are considered privileged and "well-known". The Internet Assigned Numbers Authority (IANA) is responsible for the global coordination of the DNS Root, IP addressing, and other Internet protocol resources. As such, IANA considers Registered Ports to be 1024–49151 and Dynamic and/or Private Ports to be 49152–65535.

Ports are often referred to by the service name which uses them. For example, TCP port 25 is well-known as the "smtp" service port. A list of ports and services can be found in your /etc/services file. Much like your /etc/hosts correlates textual host names to IP addresses; the /etc/services file correlates ports to service names.

However, ports can be assigned to any service you'd like. To illustrate this point, consider TCP port 8080. This is commonly used in many products for web-based administration. The products and services vary but the /etc/services considers the most common to be "webcache". For this reason, my techniques will use command line options (i.e., –n & –P) which prevent the look up of the service name so, the port number will always be reported.

Knowing What Ports are Open

This is the most important step. System administrators must know every port their operating system has open. If at all possible, you should install the lsof(8) command. I prefer lsof(8) but some sites don't permit it on their systems. If that's the case, you can use the netstat(8) command.

Use the netstat(8) command to determine which TCP & UDP ports are open:

# /bin/netstat -an --program |/bin/egrep '^(udp|tcp.*LISTEN)'

tcp  0      0 0.0.0.0:111                 0.0.0.0:*     LISTEN   833/rpcbind         
tcp  0      0 0.0.0.0:22                  0.0.0.0:*     LISTEN   1132/sshd           
tcp  0      0 127.0.0.1:631               0.0.0.0:*     LISTEN   914/cupsd           
tcp  0      0 127.0.0.1:25                0.0.0.0:*     LISTEN   1327/sendmail: acce 
tcp  0      0 0.0.0.0:60353               0.0.0.0:*     LISTEN   3044/rpc.statd      
tcp  0      0 :::111                      :::*          LISTEN   833/rpcbind         
tcp  0      0 :::80                       :::*          LISTEN   2463/httpd          
tcp  0      0 :::22                       :::*          LISTEN   1132/sshd           
tcp  0      0 ::1:631                     :::*          LISTEN   914/cupsd           
tcp  0      0 :::443                      :::*          LISTEN   2463/httpd          
udp  0      0 0.0.0.0:111                 0.0.0.0:*              833/rpcbind         
udp  0      0 0.0.0.0:1008                0.0.0.0:*              833/rpcbind         
udp  0      0 0.0.0.0:631                 0.0.0.0:*              914/cupsd           
udp  0      0 10.0.2.15:123               0.0.0.0:*              1140/ntpd           
udp  0      0 127.0.0.1:123               0.0.0.0:*              1140/ntpd           
udp  0      0 0.0.0.0:123                 0.0.0.0:*              1140/ntpd           
udp  0      0 0.0.0.0:676                 0.0.0.0:*              3044/rpc.statd      
udp  0      0 0.0.0.0:33834               0.0.0.0:*              3044/rpc.statd      
udp  0      0 0.0.0.0:68                  0.0.0.0:*              1104/dhclient       
udp  0      0 0.0.0.0:60100               0.0.0.0:*              857/avahi-daemon: r 
udp  0      0 0.0.0.0:5353                0.0.0.0:*              857/avahi-daemon: r 
udp  0      0 :::111                      :::*                           833/rpcbind         
udp  0      0 :::1008                     :::*                   833/rpcbind         
udp  0      0 fe80::a00:27ff:fe2d:a55:123 :::*                   1140/ntpd           
udp  0      0 ::1:123                     :::*                   1140/ntpd           
udp  0      0 :::123                      :::*                   1140/ntpd

The two most important columns are the fourth column and the last column. The fourth one identifies the network interface address and port it is listening on. For example, "127.0.0.1:25" says that TCP port 25 is listening on the loopback (127.0.0.1) interface only. Records such as "0.0.0.0:22" means that any request to connect to port 22 can be handled by any of the operating system's network interfaces—zeroes act as a wild card. Those ports preceded by all colons such as ":::443" listen on all interfaces with IPv6 addresses—the colons act as wild cards in IPv6.

The last column contains the process identifier (pid) and program which controls the open port. The --program (or -p) option is the only way to see this information. In our example, TCP port 22 is controlled by the sshd program and it's current pid is 1132. Using the ps(1) command, you can get some information on the process:

# /bin/ps --pid 1132 -F
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root      1132     1  0  2004   784   0 11:38 ?        00:00:00 /usr/sbin/sshd

Often times, system administrators will install new software and configure it using its default port. However, in some cases when they attempt to start the new service, the chosen port is already in use by another process. You can use the netstat(8) command as shown above to find the port or you can use lsof(8) which is quicker and easier. For example, to identify the program and pid controlling TCP port 22 you can simply execute:

# /usr/sbin/lsof -P -i :22
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    1132 root    3u  IPv4  13220      0t0  TCP *:22 (LISTEN)
sshd    1132 root    4u  IPv6  13222      0t0  TCP *:22 (LISTEN)

Taking Action

At this juncture, you must decide if you are going to leave the port open or close it. If you decide to leave it open, you should restrict access; which means limiting what networks or remote machines can connect to your port. Many organizations have a network infrastructure in place to control access. However, I strongly recommend taking it a step further by configuring the service to only listen on specific interfaces and use host-based firewalls for granular access control.

▸ Restricting Access

When it comes to configuring a service to listen on a specific interface, many applications have simple options to do this in their configuration file. For example, to configure the Apache Web (HTTP) server use the Listen directive in the httpd.conf file. Similarly, you can restrict the OpenSSH SSH daemon by using the ListenAddress parameter in its configuration (sshd_conf) file.

Next consider, taking advantage of the Linux kernel's firewall. You can use your operating system's graphical administration tools to establish your rules. For more granular control, use the command line iptables(8) administration tool. To determine if you have existing rules, you can execute "/sbin/iptables -L" as root.

Once you've restricted access to the port, it is critical that you confirm the restrictions. In my opinion, the best open source tool to do this is Network Mapper (Nmap). Use this tool from various points within and outside of your network to test your access controls. Nmap will even attempt to identify the service as well as the version of the service. Here is an example:

# nmap -sV -A spartacus

Starting Nmap 5.21 ( http://nmap.org ) at 2010-03-24 16:31 EDT
Nmap scan report for spartacus (192.168.1.30)
Host is up (0.00016s latency).
Hostname spartacus resolves to 2 IPs. Only scanned 192.168.1.30
Not shown: 994 closed ports

PORT    STATE SERVICE  VERSION
22/tcp  open  ssh      OpenSSH 5.3 (protocol 2.0)
| ssh-hostkey: 1024 2a:03:cc:09:fd:85:c6:1b:3f:80:dc:61:6f:19:89:38 (DSA)
|_2048 1d:90:d3:b4:44:06:22:f9:59:bd:d6:22:23:79:1a:6e (RSA)

111/tcp open  rpcbind  2-4 (rpc #100000)
| rpcinfo:  
| 100000  2,3,4    111/udp  rpcbind  
| 100024  1      41016/udp  status   
| 100000  2,3,4    111/tcp  rpcbind  
|_100024  1      40028/tcp  status   

443/tcp open  ssl/http Apache httpd 2.2.14 ((Fedora))
|_html-title: Test Page for the Apache HTTP Server on Fedora

631/tcp open  ipp      CUPS 1.4

Device type: general purpose
Running: Linux 2.6.X
OS details: Linux 2.6.19 - 2.6.31
Network Distance: 0 hops
Service Info: Host: spartacus; OS: Unix

▸ Close the Port

If you've decided that you do not need this service, then you should stop the service in order to close the port. As a last resort, you can use the kill(1) command to send a terminate signal to the process. However, it is best to identify the specific service, use the service(8) command to stop it, and then use the chkconfig(8) utility to prevent the service from starting during system boot.

For our example, we found that TCP port 40028 was being controlled by a program named rpc.statd. The first step is to identify the service which this program controls. Usually, the program and service name share the same—in Red Hat-based systems httpd program's service name is httpd.

# /sbin/chkconfig --list |/bin/grep statd
# /sbin/chkconfig --list |/bin/grep rpc.statd

However, in this example there is no service name which is even close to the program name. So, now we'll take a look in the directory which contains the system wide service scripts. I searched all of the scripts for our program:

# /bin/grep -l statd /etc/init.d/*
/etc/init.d/nfslock

Now, we can use the chkconfig(8) command to determine what run-levels the service is configured to run:

# /sbin/chkconfig --list nfslock
nfslock         0:off 1:off 2:off 3:on 4:on 5:on 6:off

We can shutdown the service immediately with the service(8) command and prevent it from starting during system boot with the chkconfig(8) command as follows:

# /sbin/service nfslock stop
Stopping NFS statd:                                        [  OK  ]

# /sbin/chkconfig nfslock off

In my opinion, the best technique is to remove the software from the system. You can use the rpm(8) command to identify the software package a particular file belongs. So, in this example we want to identify the name of the package which the nfslock service belongs.

# /bin/rpm -q --file /etc/init.d/nfslock
nfs-utils-1.2.1-4.fc12.i686

Summary

The importance of knowing what ports are open can't be emphasized enough — then closing unneeded ports and restricting access to the ones you do need. These points are emphasized in the "Twenty Critical Security Controls for Effective Cyber Defense: Consensus Audit Guidelines". Specifically Critical Control 3 and 13.

For more information, check out my previous blog post: "Minimize Attack Surfaces".

Disabling USB Thumbdrives in openSUSE 11

2010-02-01T13:41:00.074-05:00

Disabling the auto-mounting and restricting the use of USB thumbdrives (a.k.a., flash drives) in Novell® SUSE® Linux 11 and openSUSE® Linux 11 is easy.

We want to disable USB storage devices and permit other USB devices such as the mouse and keyboard. To do this, we’ll remove the USB storage support module from the kernel using modprobe. Edit the /etc/modeprobe.conf file and add the following line to the end of the file:

remove usb-storage /sbin/modprobe -f usb-storage

On a single system, this is a simple edit but it will become cumbersome to configure many different systems. Security Blanket® Enterprise Edition automates this configuration on Novell® SUSE® & openSUSE®, Fedora™, Red Hat®, Oracle® Enterprise Linux, CentOS, and Solaris™—all through the same, single policy. Here is a video tutorial on how to create and implement such a policy:

Personally, I love USB thumbdrives but many security guidelines recommend disabling them. Besides a classified environment, it might make sense to deploy this technique on appliances and desktops shared by many people. If you don’t disable them, at least consider using built-in file system encryption or use an IronKey. Either way, remember that good physical and personnel security is always a must.

Performing a Quick Scan

2010-01-25T09:53:00.016-05:00

Assessing a Linux system’s security against the DISA UNIX STIG is quick and easy with Security Blanket. In addition to scanning, you can even apply the profile to configure the system to meet the guidelines.

In November 2009, I posted a sneak preview of Security Blanket v4.0. Since then, the product was released. I’ve been busy authoring white papers, conducting research, and working with customers. Now that I have more time to better maintain this blog, look for a series of tutorial videos demonstrating Security Blanket’s power.

I recorded this video using Istanbul and Firefox on my Fedora 10 desktop. Great job to all of the Istanbul and Firefox developers—the combination of these tools makes my life easy!

On March 23, 2010, check out the the free webcast with SANS: “Analyst Webcast: Automated Operating System Lockdown: Security Blanket 4.0 Review.”

If there are other features of the product you’d like to see, add a comment to this post or drop us an email at SecurityBlanket@TrustedCS.com.

Lock Down: Seeing is Believing

2010-01-04T13:23:00.008-05:00

WHEN IT COMES TO SECURITY SOFTWARE, SEEING IS BELIEVING. After reading SANS NewsBites - Volume: XI, Issue: 101, I was shocked and angered to learn that U.S. taxpayers paid $30 million for software that didn't even work!

A self-proclaimed software programmer who convinced the CIA that he had developed software capable of deciphering hidden messages in Al Jazeera broadcasts appears to have been responsible for an elevation in the national security level in late 2003, causing the grounding of international flights and the evacuation of the Metropolitan Museum of Art. Dennis Montgomery managed to convince a CIA Directorate of Science and Technology employee that his technology and the information it generated were credible. The information was passed to top government officials. Only later did it become evident that Montgomery had not shared his algorithms with anybody in the Government, nor was anyone in the government clear about how the information was obtained. Montgomery also reportedly received a no-bid US $30 million contract for "compression" and "automatic target recognition" technology that he claimed could analyze surveillance video from drones and identify weapons in people's hands. A man who used to work with Montgomery says he helped fake about 40 demonstrations of the software. (See http://www.wired.com/threatlevel/2009/12/montgomery-2.)

The Security Blanket development team has worked hard to build a quality product which system administrators want and need. We get positive feedback from our customers all the time as well as enhancement requests. But don’t take our word for it—our customers have told SANS that Security Blanket automates a number of the “Twenty Critical Security Controls for Effective Cyber Defense”. As a result, SANS has placed Security Blanket on their User Vetted Tools list.

Sure, we’ve got numerous customer testimonials we could post on a web page or place in marketing brochures. We do demonstrations at trade shows and have hosted webinars on the product. However, there is nothing like seeing a product run on your own system so, we encourage you to try Security Blanket for free. That’s right—get a trial of our product for free and run it on your system.

For a trial with basic features, go here. For a full-featured trial, send a request to SecurityBlanket@TrustedCS.com or call 1-443-255-5454.

DISA Unix SRR Vulnerability

2009-12-09T09:45:00.066-05:00

“U.S. Defense Information Systems Agency (DISA) Unix Security Readiness Review (SRR) root compromise” vulnerability was reported to the CERT Coordination Center September 21, 2009 and assigned number #433821. On December 3, 2009, the U.S. Computer Emergency Response Team (US-CERT) made it publicly available. It was assigned CVE-2009-4211 on December 4, 2009 and placed in the National Vulnerability Database. However, the CVE is a bit misleading because it only identifies Solaris 10 x86 as vulnerable. The original vulnerability report states, “It was tested on Solaris/x86 only but is expected to be applicable to all Unix/Linux versions supported by the software.”

On December 6, 2009, the DISA Field Security Office sent an email requesting organizations to not use the SRR scripts until the problem has been corrected. Below is a copy of the email our team received:

-----Original Message-----
From: XXXXX, XXXXXXXXXXX X CTR DISA FSO
[mailto:xxxxxxxx@disa.mil] On Behalf Of IASE
Sent: Sunday, December 06, 2009 2:27 PM
Subject: Temporary Removal of UNIX SRR Scripts (UNCLASSIFIED)

Classification:  UNCLASSIFIED
Caveats: NONE

Due to an issue discovered with the UNIX SRR scripts, we ask 
that you immediately stop running any version of the UNIX 
scripts (Oct 2009 or older).  An updated version will be 
made available soon and information on how to access the 
updates will be provided.

Classification:  UNCLASSIFIED
Caveats: NONE

Just within the last week or so, I have seen a media frenzy and a wave of blogposts and reposts of information related to the issue. Unfortunately, some of the content has been modified which has lead to further confusion. So, I’d like to take a deeper look at the issue and and discuss some possible things to mitigate the risk. Let’s start by examining the executive summary of the vulnerability report:

“Unprivileged local users can obtain root access on Unix systems where the DISA SRR scripts are run. If a remote user can introduce a file into the filesystem (e.g. anonymous ftp, http upload, cdrom, samba share, etc.), root access may be obtained by remote, and potentially anonymous, users.”

First of all, what is the DISA Unix SRR?: “The U.S. Defense Information Systems Agency (DISA) publishes Security Readiness Review scripts (SRRs) to ensure systems and software meet security baselines required by the Department of Defense. The SRRs are commonly run on military systems and DISA makes them available to other government agencies and the general public (at their own risk) at http://iase.disa.mil/stigs/stig/index.html.”
What do they mean by “If a remote user can introduce a file into the filesystem…”?: If the server is configured to use applications such as Samba, a user can save files on the server with any name they choose. If the server was configured with a web application or an FTP server, the user might be able to save a file on the server with any name they choose.
How could running the SRR scripts make my systems vulnerable?: The SRR scripts take an inventory of all files on the system. The scripts are interested in which versions of executables reside on your system to include ‘java’, ‘snort’, and ‘php’. So, when the SRR scripts find such files, it executes them with an argument to report their version (i.e., php -v). Since the SRR scripts are always run as root, the files are executed with root privileges.

So, let’s consider an insider threat scenario in which Samba is used on a Linux-based file server. Samba provides file shares to its internal employees—these shares appear as drive Z:\ on their Microsoft Windows desktop. A user can simply drag-and-drop any file they wish into their share (Z:\). If the SRR scripts found a file named ‘php’ in John Q. Public’s home directory which he created, the SRR script would execute the command as root. The SRR would expect output from the command (/home/jqpublic/php -v) to be something like:

PHP 5.2.9 (cli) (built: Apr 17 2009 03:29:14) 
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies

The command may report the above to avoid detection and it may also do some very malicious things to your system.

Executing a command to determine its version during a scan is a lot like placing your hand on a stove to see if it is hot rather than checking the stove’s gauges first. Almost all modern operating systems deploy software in the form of packages. When the package is installed, it extracts and copies libraries, configuration files, and executables to their appropriate location within the operating system. The package is also registered with the operating system and can be queried to determine the version as well as which files were included in the package.

The following techniques are presented for discussion purposes only. It is strongly recommended you follow the DISA FSO’s recommendation of not running the SRR scripts until a fix is provided.

Since packages can only be installed as root, consider the following questions:

Did you receive the package from a trusted source? Did you verify the package’s signature before installing it?
If a command is found on the system, does it belong to a package? If so, has the file changed since it was installed?

To find out if a file was installed as part of a package, you can query the package database in Linux with the rpm(8) command as follows:

# /bin/rpm -q --file /usr/bin/php
php-cli-5.2.9-2.fc10.x86_64

# /bin/rpm -q --file /home/jqpublic/php
file /home/jqpublic/php is not owned by any package
# echo $?
1

In the above example, you can see that /usr/bin/php belongs to the ‘php-cli’ but the /home/jqpublic/php does not belong to any package. Also, note the return code of the rpm command ($?) was 1. When the file belongs to a package, the return code will be 0.

You can also query the package database to obtain the version:

# /bin/rpm -q php-cli --qf "%{VERSION}-%{RELEASE}\n"
5.2.9-2.fc10

Any command not part of a package should be considered suspect. And if the command is part of a package, you want to make sure that the file has not changed since it was installed. When the package was installed, it also registered information about the files including MD5 digest. So, if you used the -V option you can verify a package:

# /bin/rpm -V php-cli; echo $?
0

The return code of the command was zero indicating the package was still intact. I added a single space character in a comment line of a Python script and verified the ‘python’ package:

# /bin/rpm -V python; echo $?
S.5....T    /usr/lib64/python2.5/xmlrpclib.py
1

It reports that the file size (S), the MD5 checsum (5), and the modification time (T) all differ from when the package was installed.

In Solaris, to determine if a file belongs to a package the pkgchk(1M) command could be used. For example, does /usr/bin/perl belong to any package:

# /usr/sbin/pkgchk -l -p /usr/bin/perl
Pathname: /usr/bin/perl
Type: symbolic link
Source of link: ../perl5/5.8.4/bin/perl
Referenced by the following packages:
        SUNWperl584core
Current status: installed

To tie these techniques together here is an example Linux script which searches your system for the commands identified in the vulnerability. It then reports whether it is part of a package, verifies the package integrity, and uses the file(1) utility to try and determine what type of file it is.

#!/bin/bash

PATH=/bin:/usr/bin:/usr/sbin
export PATH

if [ "`uname -s`" != "Linux" ]; then
    echo "Sorry, script written for Linux."
    exit 1
fi
if [ "`id -u`" -ne 0 ]; then
    echo "Sorry, you must run this script as root"   
    exit 1
fi


IFS=$(echo -en "\n\b")
for tFile in `find / -regextype posix-egrep \
     -regex '.*/(java|openssl|php|snort|tshark|vncserver|wireshark)$' \
     -type f -print`
do
    echo $tFile
    printf ": Package - "
    pkginfo="`rpm -q --file $tFile --qf \"%{NAME}|%{VERSION}-%{RELEASE}\"`"
    if [ $? -eq 0 ]; then 
        pkgName=`echo $pkginfo |cut -f1 -d"|"`
        pkgVers=`echo $pkginfo |cut -f2 -d"|"`
        printf "%s (%s)\n" $pkgName $pkgVers
        printf ": Integrity Check - "
        rpm -V $pkgName
        if [ $? -eq 0 ]; then
            echo "OK"
        fi
    else
        echo "No associated package found"
    fi
    printf ": File type - " 
    file -b $tFile
    echo 

done
IFS=" "
echo
exit 0

Another thing to consider is the permissions of such commands. If a command has world-writeable permissions, any user can overwrite its contents. There are several guidelines provided by DISA and other organizations.

Finally, it is strongly recommended that system administrators routinely baseline their systems to identify new or modified packages and files. Security Blanket can assist with baselining, restricting file permissions, and other aspects of security. It is important to note that Security Blanket uses this methodology in determining software versions; it actually accesses the package database via an application programming interface in lieu of running shell commands. For more information, check out the following links:

How can Security Blanket help with Linux lock down?

2009-11-24T07:16:00.112-05:00

Performing “lock down” on openSUSE, Red Hat, Fedora, and CentOS Linux is often misunderstood. Even more misunderstood is Security Blanket’s role in the lock down process.

To put you in the right frame of mind, consider the following questions:

What exactly am I trying to protect? Really, think about this question before you just yell out something generic like “My Webserver”. What kind of Web server is it? What kind information does it serve? What else must run on this machine?
What and how do I configure the operating system and applications to meet the goals I’ve identified?
How do I implement these configuration changes consistently across all platforms, regardless of the operating system? How I do ensure that it stays that way?
How do I ensure that the configuration changes I made are actually protecting the system the way I want?

Too often, people jump to question number two. Even worse, I’ve seen people immediately downloading “How-tos”, lock down recipes, and install new tools. Then once the’ve discovered what can be configured, they go back and define their security goals.

Some people choose to install additional software such as grsecurity or ModSecurity for the Apache Web Server. However, Security Blanket is focused on configuring what is already part of the operating system distribution.

Security Blanket simply configures operating system features. For example, if you want to make sure that no file systems are mounted with the nosuid option, user accounts have a minimum password length of eight characters, and that certain services are turned off—Security Blanket will configure these things for you. It also does do some low-level to middleware application settings such global PHP settings and Samba.

You can easily reverse any changes that have been made and there is even a scan-only mode, called an “assessment”. Checking, configuring, and reversing a setting in the operating system is performed by a Security Blanket module. Currently, there are about two hundred modules for you to choose from. Some modules accept parameters such as the module which configures the minimum password length. In Security Blanket terminology, a grouping or collection of modules is called a “profile”; which is used to implement and enforce your site security policy.

Here is where it gets cool: when you build a profile, it can be applied to any of the supported operating systems. So, you could use your openSUSE profile on a Red Hat and Solaris platform, too. The module knows how to configure the underlying operating system and if it isn’t applicable, it will report not applicable. For example, there is a module to disable the Kudzu service but it isn’t applicable to Solaris.

The modules will even check to see if a software package is installed and if it isn’t, it will report not applicable. For example, if you choose to disallow “guest ok” in Samba but Samba isn’t installed; it will only configure the machines which have Samba and the others will just be reported as not applicable.

When it comes to disabling services, Security Blanket is aware of corresponding software packages and services for each operating system. For example, if you wanted to disable server-side NFS services consider all of the different software package and service names:

OS	Package	Service Names
Fedora 10 & 11	nfs-utils	nfs
RHEL4 & 5	nfs-utils	nfs
Solaris 10	SUNWnfssr	`svc:/network/nfs/mapid:default svc:/network/nfs/cbd:default svc:/network/nfs/server:default`
SUSE 11	nfs-kernel-server	nfs

On Solaris systems, Security Blanket even uses SMF to manage the services.

Summary

You can create new security profiles based on proven industry standards or your own site security needs. When you create or select profiles, you are building a system security policy that contains modules with configured options. Security Blanket provides the following industry-standard profiles:

CIS Benchmarks - contains requirements from the Center for Internet Security (CIS) benchmarks.
DCID - contains requirements from the Director of Central Intelligence Directives (DCID).
DISA UNIX STIG - contains modules that address DISA UNIX Security Technical Implementation Guides (STIGs) security recommendations.
FERC CIP - contains modules that address the Critical Infrastructure Protection (CIP) standards for electricity distributors.
JAFAN - the Joint Air Force-Army-Navy (JAFAN) manual established the security policy and procedures for storing, processing, and communicating classified Department of Defence (DoD) Special Access Program (SAP) information in information systems.
NISPOM - contains modules that address the National Industrial Security Program Operating Manual (NISPOM).
PCI DSS - Payment Card Industry Security Standards Council includes members from Visa, MasterCard, American Express, Discover, and JCB International Credit Card Company. This council administers the Data Security Standard (DSS).
Web Services Protection - contains modules that address standard security functionality to provide protection.

For more information on building profiles, see my previous blog post.

You can use Security Blanket’s enterprise console to centrally manage all of your systems. This includes remotely assessing, configuring, baselining, and storing your assessment and baseline reports.

The enterprise console has role based access controls so, you can grant your security officer permission to browse centrally located reports as well as identify which system administrators are allowed to actually configure the systems.

For more details on Security Blanket, plan on being part of the upcoming free webcast.

Version 4.0 Sneak Preview - Building Profiles

2009-11-19T11:11:00.055-05:00

Security Blanket v4.0.0 will include lock down support for openSUSE 11, Novell SUSE 11, and Fedora 11. Additionally, our completely redesigned console streamlines the process of building complex profiles to satisfy your site’s security policy.

For those of you who don’t know, Security Blanket is an operating system hardening tool. Through a centralized web console, it enables you to periodically check and re-check the state of security in your enterprise-wide systems and apply changes by configuring the operating systems based on site security requirements.

With a rich set of configuration options available for assessing the security state of your systems, Security Blanket assists you in defining and applying (configuring) security settings applicable to your systems.

In this post, I will describe two key terms: “Profiles” and “Modules” as well as provide some screen shots to show how you can customize your profiles.

Terminology

Profiles

Profiles conveniently manage a collection of modules and their associated parameters. Security Blanket profiles are completely customizable, allowing you to use common tools across multiple systems such as larger enterprises. You can create new security profiles based on proven industry standards or your own site security needs. When you create or select profiles, you are building a system security policy that contains modules with configured options. Security Blanket provides the following industry-standard profiles:

CIS Benchmarks - contains requirements from the Center for Internet Security (CIS) benchmarks.
DCID - contains requirements from the Director of Central Intelligence Directives (DCID).
DISA UNIX STIG - contains modules that address DISA UNIX Security Technical Implementation Guides (STIGs) security recommendations.
FERC CIP - contains modules that address the Critical Infrastructure Protection (CIP) standards for electricity distributors.
JAFAN - the Joint Air Force-Army-Navy (JAFAN) manual established the security policy and procedures for storing, processing, and communicating classified Department of Defence (DoD) Special Access Program (SAP) information in information systems.
NISPOM - contains modules that address the National Industrial Security Program Operating Manual (NISPOM).
PCI DSS - Payment Card Industry Security Standards Council includes members from Visa, MasterCard, American Express, Discover, and JCB International Credit Card Company. This council administers the Data Security Standard (DSS).
Web Services Protection - contains modules that address standard security functionality to provide protection.

Depending on the needs of your system, choose one of these industry-standard profiles and you can be confident that you are applying some of the best practices and security recommendations from the most well respected sources.

Modules

Modules are the building blocks of security profiles within Security Blanket. Each module represents a specific security lock down procedure (which in turn represents one or more system configuration operations). Modules can perform operations that range from setting file permissions and adjusting password policy to altering core operating system behavior through the use of kernel options. Some modules have adjustable parameters, enabling you to modify the value (for example, “Password Policy Length Minimum”).

You can add modules or remove them from your profile. Even though you can remove a module from a profile, the module remains part of the pool. Modules with adjustable parameters allow you to modify their parameter values. Security Blanket provides you with the freedom to load a profile and run Security Blanket against modules instead of having to reconfigure the application each time it is executed.

Managing Profiles

From the main, “Profiles” page, you can manage all of your profiles:

When modifying a profile, you can add or remove modules as well as change the parameters of modules you have already added.

For example, if you click the “Modify” button for the “Password Policy Length Minimum” module you can change its parameter value:

If you are creating a custom profile from scratch or using a variation of one of the industry standard profiles, it can be challenging to find the right modules. Security Blanket includes two-hundred some modules therefore, we have included a powerful search tool to find the right ones for your profile:

Search Criteria Option	Description
Word Search	Use this field to enter key words that appear in the module description, or the module name.
Category	Use this drop-down list to select a category under which the module falls. A category contains a collection of individual modules that are related by a common security category (such as Password Policy). In previous Security Blanket releases, these are referred to as "Module Groups".
Platform	Use this drop-down list to select an application platform under which the module falls. Common Platform Enumeration (CPE™) is a structured naming scheme for platforms. The asterisk () is a wild card, meaning all inclusive. For example, cpe:/a:apache:http_server: indicates all Apache HTTP servers. Using “cpe:/o:redhat:fedora:10” as an example, the “o” stands for “operating system”. Using “cpe:/a:apache:http_server:*” as an example, the “a” stands for “application”. Refer to http://cpe.mitre.org/specification/index.html for more information on CPEs.
Compliancy	Use this drop-down list to select a specific compliancy guideline for which the module satisfies.

Summary

Security Blanket v4.0.0 is scheduled to be released in December 2009. Check out our free webinar on December 8 to demonstrate the product. If you have any questions, please send us an email at SecurityBlanket@TrustedCS.com.

Configure Linux Mozilla Firefox to Meet DISA UNIX STIG

2009-11-13T12:07:00.098-05:00

Configuring Mozilla® Firefox® to Meet DISA UNIX STIG can be frustrating. If you’re using Mozilla® Firefox® v3 on a Linux desktop such as openSUSE® or Red Hat® and you’re required to follow DISA’s guidelines, there are many changes you must make. If you’re the system administrator, you must ensure each user’s preferences are configured the same. To make matters worse, the guideline instructs you to use the graphical interface to check each browser setting of each user. This is just not practical if you have many users which are probably constantly undoing these annoying settings.

When I began writing a Security Blanket® module to automate this process, I had to determine where the browser was storing its per-user configuration settings and what they were. I discovered that each user that had used Firefox®, a preferences file was created in their home directory ($HOME/.mozilla/firefox/*/prefs.js).

In order to determine the settings, I monitored the file as I used the graphical interface to make appropriate changes and then cross-referenced them with Mozilla’s documentation. For example, when I disabled JavaScript, the user_pref("javascript.enabled", false) line would be inserted or modified accordingly in the preferences file.

The following is a list of DISA UNIX STIG line items and their associated Mozilla properties:

GEN004040 - Browser Software Update Feature (Mozilla property: app.update.auto)
GEN004100 - Browser Allows Active Scripting (Mozilla properties: javascript.enabled & security.enable_java)
GEN004120 - Browser Data Redirection Warning (Mozilla property: security.OSCP.enabled)
GEN004160 - Browser Certificate Warning (Mozilla property: security.OSCP.enabled)
GEN004180 - Browser Home Page (Mozilla property: browser.startup.homepage)
GEN004200 - Browser SSL Configuration (Mozilla properties: security.enable_ssl2, security.enable_ssl3, & security.enable_tls)
GEN004260 - Browser Cookie Warning (Mozilla property: network.cookie.cookieBehavior)
GEN004280 - Browser Form Data Warning (Mozilla property: security.warn_submit_insecure)
GEN004300 - Browser Secure and Non-secure Content Warning (Mozilla property: security.warn_viewing_mixed)
GEN004320 - Browser Leaving Encrypted Site Warning (Mozilla properties: security.warn_leaving_secure & security.warn_entering_secure)

Next, I wanted to define a dictionary of the required settings and make only those settings in the preferences file; while preserving all other settings the user has chosen. Below is an example Python (v2.4+) script to show how this can be done:

#!/usr/bin/env python
import os
import re
import sys
import pwd
import shutil

DIRLOC = '.mozilla/firefox'
PREFS  = 'prefs.js'

REQD_OPTS = { 'javascript.enabled' : 'false',
             'security.enable_java' : 'false',
             'security.OCSP.enabled' : '1',
             'browser.startup.homepage' : 'about:blank',
             'app.update.auto' : 'false',
             'security.enable_ssl2' : 'true',
             'security.enable_ssl3' : 'true',
             'security.enable_tls' : 'true',
             'network.cookie.cookieBehavior' : '2',
             'security.warn_entering_weak' : 'true',
             'security.warn_leaving_secure' : 'true',
             'security.warn_entering_secure' : 'true',
             'security.warn_submit_insecure' : 'true',
             'security.warn_viewing_mixed'  : 'true' }

##########################
def get_moz_pref_files():
    dirs = []
    for pwent in  pwd.getpwall():
        homedir = "%s/%s" % (pwent.pw_dir, DIRLOC)
        if os.path.isdir(homedir):
            dirs.append(homedir)

    unique = list(set(dirs)) 

    pref_files = []
    for mdir in unique:
        for root, dirs, files in os.walk(mdir):
            for name in files:
                if name == PREFS:
                    pref_files.append(os.path.join(root, name))

    return pref_files

########################
def fix_prefs():
    for pref_file in get_moz_pref_files():

        sys.stdout.write("\n%s\n" % pref_file)
        try:
            in_obj = open(pref_file, 'r')
        except (IOError, OSError), err:
            msg = ":: ERROR: Unable to open %s : %s\n" % (pref_file, err)
            sys.stdout.write(msg)
            continue

        try:
            out_obj = open(pref_file + '.new', 'w')
        except Exception, err:
            in_obj.close()
            msg = ":: ERROR: Unable to create %s.new : %s\n" % (pref_file, err)
            sys.stdout.write(msg)
            continue

        # Initialize counters
        key_count = {}
        for key in REQD_OPTS.keys():
            key_count[key] = 0

        # Count matches
        lines = in_obj.readlines()
        in_obj.close()
        for idx, line in enumerate(lines):
            line = line.rstrip('\n')
            if line.startswith('user_pref('):
                line = line.strip(' ')

                # Use Regex to extract the key-value pairs
                # Sample: user_pref("security.warn_entering_secure", false);
                pat = re.compile('user_pref\("(\S+)",\s+(.*)\);')
                mat = pat.match(line)
                key = mat.group(1)
                cur_value = mat.group(2)
                cur_value = cur_value.strip('"')

                if REQD_OPTS.has_key(key):
                    if REQD_OPTS[key] == cur_value: 
                        key_count[key] += 1

                    if REQD_OPTS[key] != cur_value:
                        line = 'user_pref("'+key+'", '+ REQD_OPTS[key]+');'
                        sys.stdout.write(":: Modified line %d - %s\n" % (idx+1, line))
                        key_count[key] += 1

                    if key_count[key] > 1:
                        line = 'discard'

            if line != 'discard':
                out_obj.write(line+'\n')
            
        for key in REQD_OPTS.keys():
            if key_count[key] == 0:
                line = 'user_pref("' + key + '", ' + REQD_OPTS[key] + ');'
                sys.stdout.write(":: Adding - %s\n" % line)
                out_obj.write(line+'\n')

        out_obj.close()

        # Swap file with temp file that was created
        try:
            shutil.copymode(pref_file, pref_file + '.new')
            shutil.copy2(pref_file + '.new', pref_file)
            os.unlink(pref_file + '.new')
        except (OSError, IOError), err:
            msg = ":: ERROR: Unable to update file: %s\n" % (err)
            sys.stdout.write(msg)
            continue

        sys.stdout.write(":: Done\n") 

if __name__ == '__main__':
    fix_prefs()

Of course, you can modify the REQD_OPTS{} dictionary to include whatever settings you’d like. You could also modify the get_moz_pref_files() function’s dirs[] list to include any home directories you’d like to search.

Here is the Python script in action on my trusty Fedora box:

/home/jadams/.mozilla/firefox/l1i51nwu.default/prefs.js
:: Modified line 30 - user_pref("browser.startup.homepage", about:blank);
:: Modified line 322 - user_pref("security.warn_viewing_mixed", true);
:: Adding - user_pref("app.update.auto", false);
:: Adding - user_pref("security.warn_entering_weak", true);
:: Adding - user_pref("security.warn_leaving_secure", true);
:: Adding - user_pref("security.enable_ssl2", true);
:: Adding - user_pref("security.enable_ssl3", true);
:: Adding - user_pref("security.enable_java", false);
:: Adding - user_pref("security.warn_entering_secure", true);
:: Adding - user_pref("security.warn_submit_insecure", true);
:: Adding - user_pref("javascript.enabled", false);
:: Adding - user_pref("security.enable_tls", true);
:: Adding - user_pref("network.cookie.cookieBehavior", 2);
:: Adding - user_pref("security.OCSP.enabled", 1);

/root/.mozilla/firefox/u69w287n.default/prefs.js
:: Adding - user_pref("app.update.auto", false);
:: Adding - user_pref("security.warn_entering_weak", true);
:: Adding - user_pref("security.warn_leaving_secure", true);
:: Adding - user_pref("security.enable_ssl2", true);
:: Adding - user_pref("security.enable_ssl3", true);
:: Adding - user_pref("security.enable_java", false);
:: Adding - user_pref("security.warn_entering_secure", true);
:: Adding - user_pref("security.warn_submit_insecure", true);
:: Adding - user_pref("javascript.enabled", false);
:: Adding - user_pref("security.enable_tls", true);
:: Adding - user_pref("browser.startup.homepage", about:blank);
:: Adding - user_pref("security.warn_viewing_mixed", true);
:: Adding - user_pref("network.cookie.cookieBehavior", 2);
:: Adding - user_pref("security.OCSP.enabled", 1);

Security Blanket® performs many other functions associated with these settings to include scan only, changing the preference files, detailed logging of the performed actions, and reversing/undoing the changes. For more information on available Security Blanket® modules, see our Modules Guide.

U.S Navy DADMS & NIAP CCEVS Evaluation Lists

2009-11-09T16:21:00.080-05:00

NIAP CCEVS

“The National Information Assurance Partnership (NIAP) is a United States government initiative to meet the security testing needs of both information technology consumers and producers that is operated by the National Security Agency (NSA) and the National Institute of Standards and Technology (NIST).”[NIAP]

“The Common Criteria Evaluation and Validation Scheme (CCEVS) is a United States Government program administered by the NIAP to evaluate information technology (IT) product conformance to the Common Criteria international standard.”[CCEVS]

SECURITY BLANKET IS NOW ON THE NIAP list of security products to be evaluated for conformance to the Common Criteria (ISO/IEC 15408) and the U.S. Navy's DADMS.

I have been following Security Blanket-related discussions on news groups and blogs over the last year or so. Many times people would ask if Security Blanket is on the “list” but no one seemed to know the specific “list” Security Blanket needed to be on before they were authorized to purchase it. I found it humorous when I discussed this “list” with existing and potential customers over the phone because most of them weren't quite sure either.

This red tape can be confusing and frustrating for a system administrator who just wants to purchase the right product to do the job but their purchasing department tells them “No” because of a “list”. So, I thought I would post some information regarding “lists”.

In October of 2009, Security Blanket was added to the U.S. Department of Navy's Application and Database Management System (DADMS) list. The DADMS is the authoritative list of software applications for use within the U.S. Navy and U.S. Marine Corps. Each application on the DADMS list has been examined and put into compliance with the Navy’s application reduction initiative.

On November 6, 2009, Security Blanket Enterprise Edition began its evalaution for conformance to “Evaluation Assurance Level 2 (EAL2), Augmented with ALC_FLR.2 (PP_IDS_SCA_BR_V1.3), VID#10411.” To see Security Blanket and other security products currently being evaluated go to http://www.niap-ccevs.org/in_evaluation/.

If you have any questions regarding these “lists” or others you would like to see Security Blanket on, send us an email at SecurityBlanket@Trustedcs.com.

Establishing a Password Policy in openSUSE

2009-11-05T07:30:00.048-05:00

ESTABLISHING A PASSWORD POLICY in openSUSE 11 was much easier than I had expected. We have been working on porting Security Blanket to openSUSE 11; which will be available in v4.0.0 scheduled for December 2009.

First of all, there are many aspects to establishing a password policy. For example, you must consider the age of a password and the complexity of a password. Of course, there are two distinct camps in which one can subscribe. The first camp believes you should change passwords frequently and the other believes you can change them less frequently as long as they are long and complex.

I am just going to offer one technique to enforce complex passwords when a user is required to change their Linux account password. Specifically, those of you who must satisfy FISMA (NIST SP 800-53/IA-5) or follow the DISA STIGs:

GEN000580 - Password Length
GEN000600 - Password Character Mix (Mixed case)
GEN000620 - Password Character Mix (Digits)
GEN000640 - Password Character Mix (Special)
GEN000800 - Password Reuse

When a user is required to change their Linux password, the operating system performs this action using the Pluggable Authentication Modules (PAM) framework. In SuSE, PAM loads pam_pwcheck(8) to do password strength checking. It uses a limited number of functions from pam_cracklib(8). However, in order to set the necessary parameters to meet the above security guidelines we must disable the pam_pwcheck(8) and enable pam_cracklib(8). Fortunately, SuSE provides a great utility called pam-config(8) to enable, disable, and set parameter values.

In order to satisfy the above guidelines, you can run the following commands as root:

# pam-config -d --pwcheck
# pam-config -a --cracklib
# pam-config -a --cracklib-minlen=14
# pam-config -a --cracklib-lcredit=-1
# pam-config -a --cracklib-ucredit=-1
# pam-config -a --cracklib-dcredit=-2
# pam-config -a --cracklib-ocredit=-2

# pam-config -a --pwhistory
# pam-config -a --pwhistory-remember=5

The last setting uses pam_pwhistory(8) to prevent users from using any of the previous five passwords. The guidelines state that it should be five or more and the default value is ten. So, if you choose not to execute the last command above you will be fine.

Just for the record, I personally feel that a minimum password length (minlen) of fourteen is ridiculous but I didn't establish the guidelines.

Another PAM library, pam_passwdqc(8), is also available which does similar checking and it provides pass phrase support. The Security Blanket team chose not to configure the system using it because it isn't part of the base operating install and most of the DISA STIGs already have established check procedures for pam_cracklib(8) configurations. The pam_passwdqc(8) library looks fine but I found the documentation a bit confusing and with very few examples.

As I said before, there are many more aspects to a site password policy and to automate these, check out the Security Blanket's Modules Guide.

SC Magazine Review

2009-10-13T15:34:00.031-04:00

In the October 2009 Issue of SC Magazine, Security Blanket v3.1 received an excellent review. We are proud to have a great review by such a prominent magazine. However, I wanted to elaborate on a few things which several of our new customers asked about during the pre-sales process.

First, there was confusion regarding supported operating systems because the article stated that we only support Red Hat and CentOS. This is true only for the web-based enterprise console but not the client machines which it manages. The client machines can be Red Hat 4 & 5, CentOS 4 & 5, Oracle Enterprise Linux 4 & 5, Fedora 10, and Solaris 10. Supported architectures include 32bit, 64bit, Intel, x86, and SPARC.

We are currently working on Security Blanket v4.0 which includes support for openSUSE 11 & Novell SUSE 11, Fedora 11, and we’ve finished regression testing Red Hat 5.3. This is scheduled to be released in December of 2009.

The article also stated that “The agent must be polled, however, so this tool is not a real-time reporting solution.” This is true because Security Blanket was never intended to be a real-time monitoring tool. Security Blanket will never probe your network but instead will only perform functions on designated machines when you tell it do so. This includes fulfilling immediate requests or you can schedule actions to be ran during non-peak hours. When these actions are complete, all of your reports are available from the console.

You can define a client’s peak hours and a load threshold within its agent. This means you can define a window of time the client should never be polled and if its busier than the designated load threshold it won’t respond to console requests. This is very useful in preventing a self-denial-of-service caused by an overeager security officer who should have waited to scan it after midnight!

For more information on Security Blanket, check out our website. Also, we will have representatives at the LISA ’09 Conference in Baltimore, Maryland so, come by and talk to us.

Root of Evil?

2009-09-05T07:07:00.203-04:00

IN THE LINUX OPERATING SYSTEM, the root account has unlimited power. If not used carefully, unexpected, evil things can happen to your system. I could tell you horror stories involving mistyped commands while logged in as root. Now, imagine this power in the hands of attacker—executing any command they desire with methodical precision.

CAG: Critical Control 8

“The second common technique used by attackers is elevation of privileges by guessing or cracking a password for an administrative user to gain access to a target machine. If administrative privileges are loosely and widely distributed, the attacker has a much easier time gaining full control of systems, because there are many more accounts that can act as avenues for the attacker to compromise administrative privileges.”

Only use the root account when absolutely necessary

Too often system administrators immediately jump to root level to perform routine tasks—this is when mistakes occur. The administrator is in a hurry and one wrong space, sitting in the wrong directory, or typed an incorrect parameter all can result in evil things the moment the [ENTER] key is pressed. I used to tell junior administrators:

“Stop and remove your hands from the keyboard. Now, slow down and think about what you are doing. When in doubt, do not hit that [ENTER] key. Your lightening speed on the keyboard does not impress me and as far as completing your assigned tasks, I want quality not quantity.”

This advice extends to experienced administrators, too. They’ve done these tasks so often, sometimes they can have a lackadaisical or even cavalier attitude. Since senior administrators have more commands in their repertoire, this situation is more toxic than a junior administrator who may only know a fraction of commands.

Protect the root account from compromise

Whenever possible, software developers should avoid writing applications with root level access. Typically system administrators are not developers and developers are not system administrators. System administrators are often responsible for hosting many applications on the same server.

It is critical that you work together through out the product’s life cycle and ensure the development environment is as close to the production as possible. If you don’t, there could be deployment problems.

When the management team is standing over you about to go into panic mode, crunch time can result in relaxing access controls in order to get an application to run. Bad move! Tell management to relax and fetch you a cup of coffee while you sort it out… they love that.

Always run applications, such as Apache web server, as non-root accounts. This way if the application is compromised, root level access is not immediately given to the attacker. Consider using technologies such as sudo, SELinux, and AppArmor.

Also, establish common groups through traditional discretionary access controls and POSIX.1e access control lists (ACL).

Know who is using the root account

First and foremost, you should never log directly into the root account. Log into your designated user account and use the switch user command to become root. If personnel login in directly as root, you have no audit trail to help identify who became root and performed administrative tasks. Grant system administrator accounts membership in the wheel group and use pam_wheel(8) to allow only group members to switch to the root account.

The only time you should login directly as root is when you are physically on the console and entering single-user mode to perform critical maintenance.

To prevent this, modify the securetty(5) configuration file to include only authorized devices root can directly login. Set PermitRootLogin to ‘no’ in the Secure Shell daemon configuration file. If you are running an FTP site, be sure to edit your ftpusers(5) file.

Many people are confused by the term console as it pertains to the securetty(5) file. On many virtual machines, console is not the physical device you directly login. Additionally, in some hardware configurations the console isn't always the console. To identify your desired device name to store in the securetty(5) file, login in as root before editing the file and use the who(1) command to identify your current login device. Now, add that device to the securetty(5) file.

Lastly, edit the syslog.conf(5) or syslog-ng.conf(5) to log all authpriv messages to /var/log/secure and if possible, configure auditd(8). Don’t forget to routinely examine /var/log/secure to see who has gained access to root.

Summary

There are many other techniques and things to consider when protecting your root account. These are just a taste to get you thinking about how important this account is to the system. It is imperative that you control administrative privileges such as root. For more ideas and information, see Critical Control 8: Controlled Use of Administrative Privileges of the Consensus Audit Guidelines. Security Blanket automates many of these tasks but there is no substitute for a well-trained, disciplined team of system administrators.

Free Webcast: Maintaining OS Security in Complex IT Environments

2009-08-18T11:24:00.025-04:00

We will be hosting a complimentary (free) webcast titled “Maintaining OS Security in Complex IT Environments.” Taking place on Wednesday, September 9, this webcast will cover ways to effectively retain compliance across an enterprise while remaining flexible in order to repurpose systems and reverse changes that impact productivity. Topics covered in this session will include:

Determining the mission impact of your data on organizational operations, assets or individuals
Operational, technical and management security controls
Updating security posture as guidelines change Best practices for maintaining enterprise-wide OS security
Security compliance auditing and validation

In addition, TCS will discuss how using Security Blanket, an automated system lock down solution, can help an organization maintain OS compliance in a matter of minutes. Security Blanket’s administration console allows system administrators to manage servers, group servers and run profiles that automate user-defined or industry-standard best practices for hardening Linux and Solaris operating systems. Attendees who download the presentation will receive a complimentary Security Blanket Modules Guide. The guide includes descriptions of each security module included in the product and is cross-referenced to the specific industry guidelines it satisfies.

WHO: Greg Hall, Secure Systems Engineer, Trusted Computer Solutions

WHAT: Maintaining OS Security in Complex IT Environments

WHEN: Wednesday, September 9, 2009 at 11:00 a.m. EDT

WHERE: Webcast; Registration for the event can be accessed at TCS Webinar Registration.

Greg Hall is an experienced systems engineer whom I have had the privilege of working with for many years. His experience as a system administrator and engineer brings real-world experience and depth to Security Blanket.

If you are familiar with the Consensus Audit Guidelines (CAG), you will see that many traditional system administrator roles and responsibilities already map to several critical controls to include:

Critical Control 2: Inventory of Authorized and Unauthorized Software
Critical Control 3: Secure Configurations for Hardware and Software on Laptops, Workstations, and Servers
Critical Control 6: Maintenance, Monitoring, and Analysis of Audit Logs
Critical Control 9: Controlled Access Based on Need to Know
Critical Control 10: Continuous Vulnerability Assessment and Remediation
Critical Control 11: Account Monitoring and Control

If you have the time, I strongly encourage you to attend. I also encourage you to share your ideas by joining the discussion on the SANS Security Leadership Blog.

Minimize Attack Surfaces

2009-08-12T19:09:00.093-04:00

I hear this question often from System Administrators: “Where and how do I begin locking down my system?!?” From the moment they begin, they discover there are countless published tips & tricks, web postings, articles, publications, and guidelines on the topic. Everyone has opinions and will passionately defend them to the end.

Dr. Andrew Tanenbaum once said, “The nice thing about standards is that there are so many of them to choose from.” In this post, I am not going to dispute anyone’s opinion on system lock down. I applaud everyone who is passionate about security and those who endeavor to lock down their systems. However, I will present a basic approach which I believe no one will dispute.

If you do not need it, remove it. Just in case you didn’t understand it the first time, I will repeat it: “If you do not need it, remove it.”

I know some of you are saying, “That’s extreme. I have disabled the service and I am using a system-based firewall and am already blocking those ports. Besides, if I remove it and need the software for something—it is a real pain in the butt to get it back on the system. I am on an isolated network and getting software on and off the system is difficult.”

First of all, what if your firewall rules fail to parse and the daemon doesn’t start? Secondly, the less stuff you have on your system the less you have to worry about disabling, configuring, and patching! Third, why waste system resources on unused services? Lastly, if you don’t have a mechanism or process in-place to patch your system—then shame on you.

When the Twenty Critical Controls for Effective Cyber Defense: Consensus Audit Guidelines were released, I said to myself “I can get on board with this kind of thinking.” In particular, Critical Control 13: Limitation & Control of Network Ports, Protocols, and Services.

Real World Example

Time to roll up your sleeves and try out a technique focused on Red Hat-based systems. Let’s get started with examining services configured to start during system boot using the chkconfig(8) command.

# /sbin/chkconfig --list |/bin/grep ":on"
…
autofs     0:off  1:off  2:off  3:on  4:on 5:on 6:off
bluetooth  0:off  1:off  2:off  3:on  4:on 5:on 6:off
…

The difficult part is determining if you need the service or not. First, identify what software package is associated with the service (replace bluetooth with the service in question):

# /bin/rpm -q --file /etc/init.d/bluetooth
bluez-utils-3.7-2.el5.centos

Now, you will need to do some research on this package and service to determine if you need it. In this example, it is fairly obvious. If you are running a public web server then you probably don’t need the bluetooth service. So, configure it to not start when the system boots:

# /sbin/chkconfig bluetooth off

If you want to disable the service immediately, use the service(8) utility. Even better, remove the package with /bin/rpm -e bluez-utils!

Don’t stop there. You should verify the network ports are truly down with a utility like lsof(8). The -i tcp and -i udp options should suffice. I would even use INSECURE.ORG’s Network Mapper (nmap) tool.

Security Blanket Can Help

Security Blanket offers over sixty four modules to disable network, peer-to-peer, file sharing, and general services—many of them enabled by default and administrators have no idea they are even running. It also has several modules to prevent the disclosure of the service versions which might otherwise give an attacker an edge.

There are even modules to adjust your kernel’s TCP/IP settings to help protect against denial-of-service attacks and packet spoofing.

Security Blanket has over one hundred and ninety modules at your disposal. Download our Modules Guide for free; it contains a cross-reference of services, package names, and even a description of the service to help you.

Most importantly, Security Blanket automates this configuration process so you can rest assured that all your systems are configured consistently.

What is a Baseline?

2009-08-05T11:57:00.153-04:00

Security Blanket provides the ability to perform baselines in order to track change. This feature is ancillary to Security Blanket’s primary function but is an affordable alternative to other products which only function as a configuration management tool. As it pertains to configuration management, let’s define the baseline process.

The identification of significant states within the revision history of a configuration item is the central purpose of baseline identification.^[1]

Hold up! What? Let me offer an explanation in terms most system administrators would understand. You’re the proud parent of a teenage son or daughter. You’re about to leave them home alone for three days for the first time. Before you leave, you take detailed photographs of each room in your house. When you return, you walk into each room with photographs in hand and examine the room carefully to see if anything has changed. To ground or not ground, that is the question.

Now, fast-forward to the present. You’ve just finished building a series of complex systems hosting an important application. You’ve locked down the systems and passed all of the required security audits and your application is working. So, this is the initial state, good state if you will.

At this point, you would want to take that photograph. Just to name a few things, this photograph should include a detailed inventory of installed software packages and versions, a list of critical files, networking configuration, and general hardware configuration.

As time marches forward, you may have new software updates, removals, additions, and new administrators. All of these could potentially change the behavior of your system which you worked so hard to configure. When a system or application malfunctions, system administrators immediately begin the fault management process which consists of identification, isolation, and remediation.

Like most administrators, my first response is always “What changed?!?! It was working just fine yesterday!” Wouldn’t it be cool to quickly take another photograph and compare it to the one you took when everything was working correctly, the good state? This is typically referred to as a “baseline comparison” and it can help identify or eliminate configuration changes as the culprit.

As a best practice, system administrators should periodically perform a baseline comparison to identify changes that could potentially become a fault. In the case of authorized, expected changes the baseline comparison can be used as evidence to your change management process that in fact, a specific change has been completed. Of course, if all of the changes were authorized you would now consider the last photograph (baseline) as the current, good state.

In many high-availability configurations, you may have systems working in parallel or as a simple fail-over configuration. In these situations, it is critical that the two system configurations be as similar as possible. So, it would be pretty cool if the baseline (photograph) was in a structured format that allowed the comparison of two systems. Well, Security Blanket can do all of these things for you.

Tracking File Changes

In addition to tracking changes to your network interfaces, routing, hardware, and firewall configuration (iptables) you must monitor critical system configuration files, libraries, and executables. Not only must you track the ownership and permissions of these files you must be able to detect if the contents have changed.

The most common method of tracking file changes is to record the cryptographic check sum. This is typically done through the use of a cryptographic hash such as SHA1 and the resultant, hexadecimal string is recorded. This is most often referred to as the file’s fingerprint.

Consider the following scenario. Bring up a terminal window and execute the following command:

$ openssl sha1 /etc/hosts
SHA1(/etc/hosts)= 8117e722ee3c230be6b68b2c1bebc955bfa62e31

Now edit your /etc/hosts and add an extra space or character anywhere in the file. Execute the above command again and notice the long, hexadecimal string has changed.

By default, when you perform a baseline with Security Blanket it records fingerprints (SHA1) of all files in /etc, /bin, /usr/bin, /usr/sbin, /sbin, and others. This process takes about thirty to sixty seconds depending on the speed of your machine. Additionally, Security Blanket offers a configurable exclusion list. The directories included in this list will not be inventoried. This is useful in large systems which have shared, attached storage.

Compliancy

Several organizations which provide guidance on secure configurations recommend monitoring configuration changes. In particular “Critical Control 2: Inventory of Authorized and Unauthorized Software”^[2] and the U.S. Defense Information Systems Agency UNIX Security Technical Implementation Guide (DISA UNIX STIG). Specifically, the DISA UNIX STIG recommends:

Create & Maintain a System Baseline to help maintain system integrity:

GEN000140 - Create and Maintain System Baseline: "Confirm with the SA that a system baseline (all device files, all sgid and suid files, and system libraries and binaries), to include cryptographic hashes of files in the baseline, has been created and is maintained."
GEN002380 - SUID Files Baseline
GEN002440 - SGID Files Baseline: "If the ownership, permissions, and location of files with the sgid/suid bit set are not baselined with the IAO, then this is a finding."

Once you have created your baseline, move a copy of this baseline off of the system to protect it.

GEN000160 – System Baseline Backup on Write-protected Media

Periodically check the system configuration (at least once a week):

GEN000220 - System Baseline for System Libraries and Binaries Checking
GEN002400 - System Baseline for SUID Files Checkling
GEN002460 - System Baseline for SGID Files Checking: "Confirm with the SA that filesystems are checked at least weekly for unauthorized system libraries or binaries or unauthorized modification to authorized system libraries or binaries."
GEN002260 - System Baseline for Device Files Checking: "If the system is not checked weekly against the system baseline for extraneous device files, then this is a finding. Ask the SA to show the previous weeks baseline of files."

Security Blanket in Action

Once you’ve downloaded, installed, and registered Security Blanket, bring up its graphical user interface by selecting Applications → System Tools &rarr Security Blanket. Now, click on the Baselining button on the left toolbar. If you’ve never used this before or have never scheduled a baseline, you will be presented a dialog box asking if you would like to schedule regular baselines.

From the screen above, you can easily perform a baseline or compare two baselines. All reports are in XML format and are stored in /var/lib/security-blanket/baselines. You can perform a baseline from the command line, too using the -b option.

Security Blanket includes schema definition files and standard style sheets to render the reports in HTML. Here are some sample reports from my Fedora 10 box using Security Blanket v3.1.0:

Baseline Report [ HTML | XML (~4MB) ]
Baseline Comparison Report [ HTML | XML ]

Summary

As you can see, performing system baselines can be very helpful and Security Blanket’s baselining feature compliments its system lock down and assessment technology. Security Blanket's Enterprise Edition will centralize this function and allow you to compare the same machine at any two points in time (states) or two different machines at any two points in time.

I have been told by existing Security Blanket customers that this feature is quite useful when deploying systems to child organizations or even their own customers. Once they have locked down and certified a system, before it is shipped to their customer a baseline is performed.

The Security Blanket team plans to expand the information collected in v4.0 scheduled for November 2009. Tentative plans include filesystem layouts, drive configurations to include logical volume layouts, generic user account lists, and open TCP/UDP ports. If you have suggestions or questions, please make a comment regarding this post or email us directly at SecurityBlanket@TrustedCS.com.

[1] CMMI Product Team, “Chpt 7, Maturity Level 2: Managed, Configuration Management, SP 1.3,”, in Capability Maturity Model Integration, Version 1.1 (CMMI-SE/SW/IPPD/SS, V1.1): Staged Representation, Carnegie Mellon Software Engineering Institute.

[2] Twenty Critical Controls for Effective Cyber Defense: Consensus Audit Guidelines, v2.0, http://www.sans.org/cag/