Useful looping constructs in Bash

Handy "looping" constructs which I frequently use in my Bash scripts & one-liners:

1) loop over files

for FILE in /etc/*; do echo "etc file: $FILE"; done

OR:

for FILE in `find / -type f -name conf`; do 
    echo "File with 'conf' in name: $FILE"; 
done

2) loop over lines in file

cat /etc/hosts | while read LINE; do echo $LINE; done

OR:

while read LINE; do echo $LINE; done < /etc/hosts

3) loop over words in a file

for WORD in `cat /etc/hosts`; do echo $WORD; done

4) loop over elements in array

A=(a b c)
for i in ${A[@]}; do
    echo $i
done

Some more advanced (but also useful) constructs:

1) looping over array of functions & executing each function

FUNC_LIST=(func1 func2)

function func1 {
    echo 'func1 here'
}

function func2 {
    echo 'func2 here'
}

for func in ${FUNC_LIST[@]}; do
    eval $func
done 

2) passing array to function

ARR=(el1 el2)

function func {
    declare -a argArray=("${!1}")
    echo "${argArray[@]}"
}

func ARR[@] 

Linux kernel hacking for System Administrators (part I)

Generally speaking kernel hacking is full time job and most of us, sys admins do not possess required programming skills and experience (or just time) to actively help in developing Linux kernel.

Nevertheless knowledge about kernel internals and familiarity with some of it's features can be of a great help for seasoned system administrator. In this series of articles I will try to demystify kernel's functionality that can help in managing & troubleshooting server's related issues.

In the first part I will present how Kernel Probes (Kprobes) mechanism can be used.

We will create kernel module that inserts jprobes (special type of KProbe). Jprobes allow us to insert kprobe on kernel's function entry point to get convinient access to function's arguments. Let's say that I want to get better insight on what's going on when specific ioctl (DRM_IOCTL_I915_GETPARAM sent to Intel's graphics card driver in my example) is called. To accomplish this we will need: linux kernel source, Makefile (that will build our module) and source code of our module (let's call it tracingModule.c).

Makefile:

obj-m += tracingModule.o

module:
	make clean 
	make -C /lib/modules/`uname -r`/build M=`pwd` modules

all:	module

clean:
	make -C /lib/modules/`uname -r`/build M=`pwd` clean

Module's source code:

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/sched.h>
#include <linux/kprobes.h>
#include <linux/kallsyms.h>
#include <linux/moduleparam.h>
#include <linux/slab.h>
#include <drm/i915_drm.h>
#include <drm/drmP.h>

/*
 our tracing function that will be called before original
 (i915_getparam that is responsible for handling 
 DRM_IOCTL_I915_GETPARAM ioctl) function.
 Here it lists values of parameters sent to the function.
*/
static int my_i915_getparam(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
    drm_i915_getparam_t *args = data;

    printk("ioctl(DRM_IOCTL_I915_GETPARAM, (drm_i915_getparam_t *) 0x%p"
	"{ param = %d, value = %d })\n", data, args->param, *(args->value));

    jprobe_return();
    return 0;
}

/* jprobe struct .entry field is pointing to our tracing function */
static struct jprobe my_getparam_jprobe = {
    .entry = (kprobe_opcode_t *) my_i915_getparam
};

/* setting up jprobe on chosen symbol (function name) */
int probeInit(struct jprobe *probe, const char *symbolName)
{
    int ret;

    probe->kp.addr = 
    (kprobe_opcode_t *) kallsyms_lookup_name(symbolName);
    if(!probe->kp.addr) {
	    printk("Couldn't find %s to plant jprobe\n", symbolName);
	    return -1;
    }

    if ((ret = register_jprobe(probe)) < 0) {
	    printk("register_jprobe failed, returned %d\n", ret);
	    return -1;
    }
    printk("Planted jprobe at %p (%s), handler addr %p\n",
    probe->kp.addr, symbolName, probe->entry);

    return 1;
}

/* standard module initialization function. Invoked by 
the kernel on loading the module. It registers our jprobe
on specified (i915_getparam) function.
*/
int init_module(void)
{
    if(probeInit(&my_getparam_jprobe, "i915_getparam") == -1)
	printk("Failed to insert jprobe. Exiting.\n");

    printk("jprobe registered.\n");

    return 0;
}

/* standard module cleanup function. Invoked by 
the kernel on unloading the module. It unregisters our jprobe.
*/
void cleanup_module(void)
{
    unregister_jprobe(&my_getparam_jprobe);
    printk("jprobe unregistered\n");
}

MODULE_LICENSE("GPL");

To build and load the module:

$ make
$ sudo insmod ./tracingModule.ko

Executing dmesg will show you details that your jprobe has printed.

[one-liner] shows whether your CPU supports 64bit mode

grep -q ' lm ' /proc/cpuinfo; [ 0 -eq 0 ] && echo '64bit supported'


it shows whether your CPU supports 64 bit (x86-64) mode. uname -a only shows whether you have 64 bit (x86-64) or 32bit (i386) OS installed, this one-liner answers question: Can I install 64bit OS on this machine?

Software flaw #3: CVE-2012-4298

CVE-2012-4298 is classical example of signedness vulnerability which I have explained in flaw #1. Affected software is Wireshark versions 1.8.x before 1.8.2.

Here's my analysis of the vulnerable code:

 
static void vwr_read_rec_data_ethernet(wtap *wth, guint8 *data_ptr, guint8 *rec,
		int rec_size, int IS_TX)
{
    ...

    // [1] msd_length is signed!
    gint16 msdu_length,actual_octets; /* octets in frame */

    ...

    // [2] msdu_length is initialized with external data (from received packet)
    m_ptr = &(rec[0]); /* point to the data block */
    s_ptr = &(rec[rec_size - vwr->STATS_LEN]); /* point to the stats block */
    msdu_length = pntohs(&s_ptr[vwr->OCTET_OFF]);

    ...

    /* [3] sanity checking is done but because msdu_length is signed, values 
	   such as -1 will pass the check */
    if (msdu_length > (rec_size - (int)vwr->STATS_LEN)) {
        msdu_length = (rec_size - (int)vwr->STATS_LEN);
    }

    ...

    /* [4] wrongly validated data is casted to size_t (unsigned int) type and
	   used as memcpy parameter potentially causing overflow of buffer 
	   pointed by data_ptr */
    memcpy(&data_ptr[bytes_written], m_ptr, msdu_length);
    ...
}
 

Mitigation:

Declare msdu_length as guint16 instead of glint16

SSH remote port forwarding

Second kind of forwarding with ssh is so called remote port forwarding. This time the service available on ssh client can be forwarded to be available on ssh server. Suppose that we have machine with httpd (or any other TCP service) somewhere behind NAT and we want to make it available on our public standing ssh machine:

 ssh-client$ ssh -R 8080:localhost:80 root@ssh-server-ip-addr

What it does is forwarding ssh-client's httpd server (port 80) to port 8080 on ssh-server.

From now on, connecting to ssh-server-ip-addr:8080 will effectively connect us with ssh-client:80

Use cases for this functionality:
- remote system administering of machine behind NAT (see my serverfault's answer)
- encrypted forwarding service to another machine

Vulnerability analysis with gdbserver

Here's my workflow when I'm doing vulnerability analysis under debugger. Thanks to gdbserver I can debug on target machine (OS image launched in Virtualbox) and use gvim and pyclewn just by dropping statically compiled gdbserver into target machine.

ON WORKING MACHINE:

# in case you don't already have it
 apt-get install dpkg-dev

# get source of your gdb (includes gdbserver)
 apt-get source gdb

# compile gdbserver
 cd gdb-x.y/gdb/gdbserver/
 LDFLAGS=-static ./configure
 make

# sent it to target machine (for example:)
 scp ./gdbserver root@192.168.x.y:

ON TARGET MACHINE:

# compile your binary (exim4 for me)
 cd exim-4.69/
 cp src/EDITME Local/Makefile
 vi Local/Makefile
    set/modify BIN_DIRECTORY
    set/modify CONFIGURE_FILE
    set EXIM_USER
    set EXIM_GROUP
    comment out EXIM_MONITOR
    add: CFLAGS += -g somewhere in the file
 make
 make install

ON WORKING MACHINE:

# copy binary from target machine to your working machine:
 cd exim-4.69/src
 scp 192.168.x.y:/usr/local/exim/bin/exim-4.69-2 ./

ON TARGET MACHINE:

# start gdbserver:
 ./gdbserver :3332 /usr/local/exim/bin/exim-4.69-2 -d -bd -oX 3333

 OR attach to existing process:

 ./gdbserver :3332 --attach  

ON WORKING MACHINE:

 cd exim-4.69/src
 gdb exim-4.69-2
     target remote 192.168.x.y:3332
     break main
     continue

 OR you could do it from gvim + pyclewn:

 cd exim-4.69/src
 pyclewn
     e exim.c
     Cmapkeys
     Cfile exim-4.69-2
     Ctarget remote 192.168.x.y:3332
     Cbreak main
     Ccontinue

Software flaw #2: integer overflow vulnerability

Following code illustrates integer overflow condition:

#include <stdio.h>
#include <memory.h>
#include <stdlib.h>

unsigned char *createTable(unsigned int w, unsigned int h, unsigned char *initialRow);

int main(int argc, char **argv)
{
    unsigned char row[2] = { 'a' };
    unsigned char *wholeTable;
    unsigned int user_provided_w = 0x400;
    unsigned int user_provided_h = 0x1000001;

    wholeTable = createTable(user_provided_w, user_provided_h, row);

    free(wholeTable);
    return 0;
}

unsigned char *createTable(unsigned int w, unsigned int h, unsigned char *initialRow)
{
    unsigned int n;
    int i;
    unsigned char *buf;

    n = w * h;
    buf = (char *)malloc(n);
    if(!buf)
        return NULL;

    for(i=0; i<h; ++i)
        memcpy(&buf[i*w], initialRow, w);

    return buf;
}
 
The purpose of createTable(...) function is to take width and height and an initial row and create table in which all rows are initialized with initialRow. However we can observe that there can be integer overflow condition (bolded line), when width and height will be big enough. Lets assume width = 0x400 and height = 0x1000001 in this situation n will be equal to 1024 (in decimal), so only 1024 bytes will be allocated. Following for loop will be iterated 0x1000001 times so heap buffer overflow will occur.