Linux kernel hacking for System Administrators (part I)

Generally speaking kernel hacking is full time job and most of us, sys admins do not possess required programming skills and experience (or just time) to actively help in developing Linux kernel.

Nevertheless knowledge about kernel internals and familiarity with some of it's features can be of a great help for seasoned system administrator. In this series of articles I will try to demystify kernel's functionality that can help in managing & troubleshooting server's related issues.

In the first part I will present how Kernel Probes (Kprobes) mechanism can be used.

We will create kernel module that inserts jprobes (special type of KProbe). Jprobes allow us to insert kprobe on kernel's function entry point to get convinient access to function's arguments. Let's say that I want to get better insight on what's going on when specific ioctl (DRM_IOCTL_I915_GETPARAM sent to Intel's graphics card driver in my example) is called. To accomplish this we will need: linux kernel source, Makefile (that will build our module) and source code of our module (let's call it tracingModule.c).

Makefile:

obj-m += tracingModule.o

module:
	make clean 
	make -C /lib/modules/`uname -r`/build M=`pwd` modules

all:	module

clean:
	make -C /lib/modules/`uname -r`/build M=`pwd` clean

Module's source code:

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/sched.h>
#include <linux/kprobes.h>
#include <linux/kallsyms.h>
#include <linux/moduleparam.h>
#include <linux/slab.h>
#include <drm/i915_drm.h>
#include <drm/drmP.h>

/*
 our tracing function that will be called before original
 (i915_getparam that is responsible for handling 
 DRM_IOCTL_I915_GETPARAM ioctl) function.
 Here it lists values of parameters sent to the function.
*/
static int my_i915_getparam(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
    drm_i915_getparam_t *args = data;

    printk("ioctl(DRM_IOCTL_I915_GETPARAM, (drm_i915_getparam_t *) 0x%p"
	"{ param = %d, value = %d })\n", data, args->param, *(args->value));

    jprobe_return();
    return 0;
}

/* jprobe struct .entry field is pointing to our tracing function */
static struct jprobe my_getparam_jprobe = {
    .entry = (kprobe_opcode_t *) my_i915_getparam
};

/* setting up jprobe on chosen symbol (function name) */
int probeInit(struct jprobe *probe, const char *symbolName)
{
    int ret;

    probe->kp.addr = 
    (kprobe_opcode_t *) kallsyms_lookup_name(symbolName);
    if(!probe->kp.addr) {
	    printk("Couldn't find %s to plant jprobe\n", symbolName);
	    return -1;
    }

    if ((ret = register_jprobe(probe)) < 0) {
	    printk("register_jprobe failed, returned %d\n", ret);
	    return -1;
    }
    printk("Planted jprobe at %p (%s), handler addr %p\n",
    probe->kp.addr, symbolName, probe->entry);

    return 1;
}

/* standard module initialization function. Invoked by 
the kernel on loading the module. It registers our jprobe
on specified (i915_getparam) function.
*/
int init_module(void)
{
    if(probeInit(&my_getparam_jprobe, "i915_getparam") == -1)
	printk("Failed to insert jprobe. Exiting.\n");

    printk("jprobe registered.\n");

    return 0;
}

/* standard module cleanup function. Invoked by 
the kernel on unloading the module. It unregisters our jprobe.
*/
void cleanup_module(void)
{
    unregister_jprobe(&my_getparam_jprobe);
    printk("jprobe unregistered\n");
}

MODULE_LICENSE("GPL");

To build and load the module:

$ make
$ sudo insmod ./tracingModule.ko

Executing dmesg will show you details that your jprobe has printed.

The power of Awk

Recently I've been busy with analyzing results of Challenge 5 of the Honeynet Forensic Challenge. The winner has written nice script for analyzing auth.log file. The script is written in Python and can be found here. I decided to implement such script in Awk. It turned out tha Awk is perfectly suited to this kind of work, here's the script:

cat auth.log | awk '

    $0 !~ /sshd/ {next};

    {ip=$11};

    /invalid/ {ip=$13};

    /Failed/ {

        Fails[ip] += 1;

        if(Fails[ip] == 1)

            StartTime[ip] = $1 " " $2 " " $3;

        EndTime[ip] = $1 " " $2 " " $3;

    };

    /Accepted/ {

        if(Fails[ip] > 0) {

            Accepts[ip] += 1;

            Details[ip] = sprintf("%s %s %s\n", Details[ip], $1 " " $2 " " $3, $9);

        }

    };

    END {

        for(key in Fails) {

            if((Fails[key] > 0) && (Accepts[key]/Fails[key] < 0.1 )) {

                print "Attacker IP: " key

                print "Start time: " StartTime[key]

                print "End time: " EndTime[key]

                print "Login attempts: " Fails[key]

                if(Accepts[key] > 0) {

                    print "Successfull attempts: " Accepts[key];

                    print Details[key];

                }

                printf "\n"

            }

        }

    }'

About 26 lines of code as opposed to 136 lines of Python version!

And here's one-liner (shorter but less verbose) version of the script:

 $ cat auth.log | awk '$0 !~ /sshd/ {next}; {ip=$11}; /invalid/ {ip=$13}; /Failed/ {Fails[ip] += 1 }; /Accepted/ {if(Fails[ip] > 0) Accepts[ip] += 1 }; END { for(key in Fails) {if((Fails[key] > 0) && (Accepts[key]/Fails[key] < 0.1 )) print key": "Fails[key] "\t Successes: " Accepts[key]} }'