Threat update #2

Linux/Cdorked.A malware (backdoor) has been detected recently. It is known to affect Apache, nginx & Lighttpd web servers. Detailed analysis can be found here and is continued here.

It is also worth to note another Linux malware (rootkit) that has been recently found in the wild and was analyzed by CrowdStrike.

Threat update #1

Exploit for CVE-2013-2094 (Linux local privilege escalation vulnerability) is publicly available here.

SSL essentials for system administrators

X.509 standard specifies format for public key certificates.

Structure of a public key certificate is expressed in a formal language: ASN.1.

Creating self-signed certificate:

1) create private key & public key (without passphrase)

$ openssl genrsa -out server.key 2048

2) only private key was generated, but public key can be always extracted from it:

$ openssl rsa -in server.key -pubout

3) create certificate-signing request (CSR). It will contain public key of the entity requesting the certificate and information about the entity.

$ openssl req -new -key server.key -out server.csr

4) sign your own certificate using CSR and your private key (for real certs it would be private key of Certificate Authority)

$ openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

5) examine the contents of your certificate

$ openssl x509 -text -in server.crt

Adding SSL support in Apache2:

<IfModule mod_ssl.c>
Listen 443
NameVirtualHost *:443
<VirtualHost *:443>
    SSLEngine On
    SSLOptions +StrictRequire
    SSLCertificateFile /etc/httpd/conf/ssl/server.crt
    SSLCertificateKeyFile /etc/httpd/conf/ssl/server.key
</VirtualHost>
</IfModule>

SSL deployment best practices can be found here.

Example certificate in text format:

Certificate:
Data:
Version: 1 (0x0)
Serial Number:
    aa:c9:e7:2b:f7:fe:38:e2
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=PL, ST=xxx, L=xxx, O=xxx, OU=xxx, CN=www.mysecuresite.com/emailAddress=xxx
Validity
    Not Before: Feb  7 13:28:40 2013 GMT
    Not After : Feb  7 13:28:40 2014 GMT
Subject: C=PL, ST=xxx, L=xxx, O=xxx, OU=xxx, CN=www.mysecuresite.com/emailAddress=xxx
Subject Public Key Info:
    Public Key Algorithm: rsaEncryption
	Public-Key: (2048 bit)
	Modulus:
	    00:df:02:92:d7:f4:0d:08:a0:78:14:f3:d4:a6:ff:
	    d7:36:ce:1d:75:90:92:af:c6:79:b1:e9:7a:60:2e:
	    b3:8a:77:94:79:c0:4f:a2:07:50:c0:9b:9e:38:9d:
	    a2:80:a9:c6:97:24:3d:80:da:13:15:63:f0:46:aa:
	    98:4d:3a:0b:37:94:4b:d9:bf:fd:8e:b6:7e:b7:46:
	    a1:58:05:20:9c:e0:6f:85:21:35:8a:14:04:7c:11:
	    0b:ed:1e:d5:24:98:0d:60:48:3f:40:51:38:17:05:
	    46:5a:2f:78:0d:64:f7:83:0a:e2:86:63:e0:23:18:
	    9a:ca:29:32:70:43:4e:41:ca:c9:57:5a:c8:9a:a7:
	    3d:50:63:96:b5:99:5a:c5:21:29:39:a5:6e:37:fb:
	    60:14:a4:a4:c7:4d:98:dc:1e:fa:c5:3e:b7:e1:ad:
	    2e:e5:c2:83:30:af:81:5d:0b:b2:19:26:6a:04:0b:
	    0b:e1:a6:70:75:ee:a2:fa:32:8d:b6:89:c8:ad:9f:
	    f0:25:66:29:27:e9:16:bf:1e:2f:ab:d4:29:af:33:
	    46:a1:c8:84:36:9a:f9:67:1a:e7:0e:e3:ac:bf:a4:
	    54:e6:e4:8d:dc:e2:1b:ad:3f:b9:3f:8c:af:7e:72:
	    27:ea:6b:6f:c7:cd:54:5a:4b:b0:e8:0a:2b:b4:dc:
	    aa:b1
	Exponent: 65537 (0x10001)
Signature Algorithm: sha1WithRSAEncryption
57:96:60:2f:97:92:a7:b4:23:9d:0d:17:62:33:e2:bb:df:34:
c1:5c:43:df:63:c1:78:53:8c:a7:e5:ad:25:3a:5f:1d:f7:70:
2b:0b:28:33:61:f9:94:6f:4f:19:e0:2d:fc:fd:40:ad:8b:5c:
cc:b3:7d:0c:ed:85:05:ad:d5:ec:9f:61:61:d7:e4:89:12:5d:
f4:97:c4:a5:b2:37:4f:15:5b:da:f2:80:a0:5d:a7:91:5f:77:
1b:ee:2e:03:3f:10:3e:2c:bf:7e:f9:c0:5b:21:04:97:bb:42:
c9:f9:b6:30:99:8d:1a:35:7c:8e:6e:2a:91:c1:c0:2c:06:53:
eb:f5:72:a1:44:f5:b5:e0:b8:b8:3f:cd:34:2c:10:0b:69:ea:
11:a8:98:00:28:f8:36:64:18:53:73:fd:7f:35:81:cd:5a:7e:
a4:34:2f:8b:b9:54:46:94:6d:d4:13:13:e2:16:df:73:9b:0d:
fe:ce:1f:53:5c:ff:cd:91:d6:60:7d:bf:65:35:67:f6:57:a9:
8a:f8:ff:d3:23:4e:a0:83:41:5f:22:dc:4c:39:06:02:84:0f:
82:21:72:21:e7:d9:14:5d:9a:66:36:38:ba:15:af:97:45:06:
dc:da:9a:e3:6d:e4:61:36:42:ba:82:39:44:a4:35:cb:75:56:
2d:36:b0:9a

most important fields:
1) Issuer: #certificate authority, Verisign for example (Issuer == Subject in case of self-signed certificates)
2) Subject: CN: #common name, it's the part that must match the host being authenticated (it should be domain name of your server)

Certificates are commonly stored in two formats:
1) DER binary with extensions: .cer, .crt, .der
2) PEM (Privacy Enhanced Mail), base64 encoded DER certificate, enclosed between "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" with extension .pem

Getting certificate in PEM format from remote server:

$ openssl s_client -connect <ip:port> < /dev/null 2> /dev/null | openssl x509

Showing remote certificate in text format:

$ openssl s_client -connect <ip:port> < /dev/null 2> /dev/null | \
    openssl x509 -noout -text

Creating fingerprints (sha1):

$ openssl s_client -connect <ip:port> < /dev/null 2> /dev/null | \
    openssl x509 -noout -fingerprint

Simple debugging technique:

$ openssl s_client -connect <ip:port> -state -debug

Google maintains database of SSL certificates that were found by it's web crawlers. To check whether given certificate is available in Google's database (Google Certificate Catalog) do this:
calculate sha1 fingerprint of chosen certificate:

$ openssl s_client -connect <ip:port> < /dev/null | \ 
    openssl x509 -outform DER | openssl sha1

look up a TXT record with calculated fingerprint as a name in the certs.googlednstest.com domain:

$ dig +short 405062e5befde4af97e9382af16cc87c8fb7c4e2.certs.googlednstest.com TXT

You will get three numbers: the first number is the day that Google's crawlers first saw that certificate, the second is the most recent day, and the third is the number of days we saw it in between.
Only Certificates that satisfy following criteria will appear in Google's catalog: it must by correctly signed (either by a CA or self-signed); it must have correct domain name, that is, one that matches the one we used to retrieve the certificate.

Useful looping constructs in Bash

Handy "looping" constructs which I frequently use in my Bash scripts & one-liners:

1) loop over files

for FILE in /etc/*; do echo "etc file: $FILE"; done

OR:

for FILE in `find / -type f -name conf`; do 
    echo "File with 'conf' in name: $FILE"; 
done

2) loop over lines in file

cat /etc/hosts | while read LINE; do echo $LINE; done

OR:

while read LINE; do echo $LINE; done < /etc/hosts

3) loop over words in a file

for WORD in `cat /etc/hosts`; do echo $WORD; done

4) loop over elements in array

A=(a b c)
for i in ${A[@]}; do
    echo $i
done

Some more advanced (but also useful) constructs:

1) looping over array of functions & executing each function

FUNC_LIST=(func1 func2)

function func1 {
    echo 'func1 here'
}

function func2 {
    echo 'func2 here'
}

for func in ${FUNC_LIST[@]}; do
    eval $func
done 

2) passing array to function

ARR=(el1 el2)

function func {
    declare -a argArray=("${!1}")
    echo "${argArray[@]}"
}

func ARR[@] 

Linux kernel hacking for System Administrators (part I)

Generally speaking kernel hacking is full time job and most of us, sys admins do not possess required programming skills and experience (or just time) to actively help in developing Linux kernel.

Nevertheless knowledge about kernel internals and familiarity with some of it's features can be of a great help for seasoned system administrator. In this series of articles I will try to demystify kernel's functionality that can help in managing & troubleshooting server's related issues.

In the first part I will present how Kernel Probes (Kprobes) mechanism can be used.

We will create kernel module that inserts jprobes (special type of KProbe). Jprobes allow us to insert kprobe on kernel's function entry point to get convinient access to function's arguments. Let's say that I want to get better insight on what's going on when specific ioctl (DRM_IOCTL_I915_GETPARAM sent to Intel's graphics card driver in my example) is called. To accomplish this we will need: linux kernel source, Makefile (that will build our module) and source code of our module (let's call it tracingModule.c).

Makefile:

obj-m += tracingModule.o

module:
	make clean 
	make -C /lib/modules/`uname -r`/build M=`pwd` modules

all:	module

clean:
	make -C /lib/modules/`uname -r`/build M=`pwd` clean

Module's source code:

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/sched.h>
#include <linux/kprobes.h>
#include <linux/kallsyms.h>
#include <linux/moduleparam.h>
#include <linux/slab.h>
#include <drm/i915_drm.h>
#include <drm/drmP.h>

/*
 our tracing function that will be called before original
 (i915_getparam that is responsible for handling 
 DRM_IOCTL_I915_GETPARAM ioctl) function.
 Here it lists values of parameters sent to the function.
*/
static int my_i915_getparam(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
    drm_i915_getparam_t *args = data;

    printk("ioctl(DRM_IOCTL_I915_GETPARAM, (drm_i915_getparam_t *) 0x%p"
	"{ param = %d, value = %d })\n", data, args->param, *(args->value));

    jprobe_return();
    return 0;
}

/* jprobe struct .entry field is pointing to our tracing function */
static struct jprobe my_getparam_jprobe = {
    .entry = (kprobe_opcode_t *) my_i915_getparam
};

/* setting up jprobe on chosen symbol (function name) */
int probeInit(struct jprobe *probe, const char *symbolName)
{
    int ret;

    probe->kp.addr = 
    (kprobe_opcode_t *) kallsyms_lookup_name(symbolName);
    if(!probe->kp.addr) {
	    printk("Couldn't find %s to plant jprobe\n", symbolName);
	    return -1;
    }

    if ((ret = register_jprobe(probe)) < 0) {
	    printk("register_jprobe failed, returned %d\n", ret);
	    return -1;
    }
    printk("Planted jprobe at %p (%s), handler addr %p\n",
    probe->kp.addr, symbolName, probe->entry);

    return 1;
}

/* standard module initialization function. Invoked by 
the kernel on loading the module. It registers our jprobe
on specified (i915_getparam) function.
*/
int init_module(void)
{
    if(probeInit(&my_getparam_jprobe, "i915_getparam") == -1)
	printk("Failed to insert jprobe. Exiting.\n");

    printk("jprobe registered.\n");

    return 0;
}

/* standard module cleanup function. Invoked by 
the kernel on unloading the module. It unregisters our jprobe.
*/
void cleanup_module(void)
{
    unregister_jprobe(&my_getparam_jprobe);
    printk("jprobe unregistered\n");
}

MODULE_LICENSE("GPL");

To build and load the module:

$ make
$ sudo insmod ./tracingModule.ko

Executing dmesg will show you details that your jprobe has printed.

[one-liner] shows whether your CPU supports 64bit mode

grep -q ' lm ' /proc/cpuinfo; [ 0 -eq 0 ] && echo '64bit supported'


it shows whether your CPU supports 64 bit (x86-64) mode. uname -a only shows whether you have 64 bit (x86-64) or 32bit (i386) OS installed, this one-liner answers question: Can I install 64bit OS on this machine?

Software flaw #3: CVE-2012-4298

CVE-2012-4298 is classical example of signedness vulnerability which I have explained in flaw #1. Affected software is Wireshark versions 1.8.x before 1.8.2.

Here's my analysis of the vulnerable code:

 
static void vwr_read_rec_data_ethernet(wtap *wth, guint8 *data_ptr, guint8 *rec,
		int rec_size, int IS_TX)
{
    ...

    // [1] msd_length is signed!
    gint16 msdu_length,actual_octets; /* octets in frame */

    ...

    // [2] msdu_length is initialized with external data (from received packet)
    m_ptr = &(rec[0]); /* point to the data block */
    s_ptr = &(rec[rec_size - vwr->STATS_LEN]); /* point to the stats block */
    msdu_length = pntohs(&s_ptr[vwr->OCTET_OFF]);

    ...

    /* [3] sanity checking is done but because msdu_length is signed, values 
	   such as -1 will pass the check */
    if (msdu_length > (rec_size - (int)vwr->STATS_LEN)) {
        msdu_length = (rec_size - (int)vwr->STATS_LEN);
    }

    ...

    /* [4] wrongly validated data is casted to size_t (unsigned int) type and
	   used as memcpy parameter potentially causing overflow of buffer 
	   pointed by data_ptr */
    memcpy(&data_ptr[bytes_written], m_ptr, msdu_length);
    ...
}
 

Mitigation:

Declare msdu_length as guint16 instead of glint16