VM Forensics and Security

After graduating from GWU's Masters of Forensic Science program this year, it was nice to see they recognized my co-publication in "Digital Forensics Magazine".

J.D. Durick (MFS, 2010), co-published an article with Eric Fitterman titled: "Ghost in the Machine: Forensic Evidence Collection in the Virtual Environment." Digital Forensics Magazine, Issue 2, Spring 2010.

The newsletter can be found here:  GWU_Forensics_newsletter.pdf

I added an offset flag so that the user can start from any byte location within the image, results will only show you from that offset on (Default offset is 0).  Additionally, I added more error checking and detailed commenting.  Furthermore, the blocksize has to be a multiple of 512 and the hex string is max'd to size of 16 bytes.  I uploaded it to sourceforge and can bytelocator-0.2 can be found there.  Here is an example of it running:

Added functionality to determine which partition is bootable (or active) as well locate the end of sector marker of the master boot record.  Here is a quick snapshot of the tool:

root@redbox:/data/projects/C/dd2vmdk-0.1.1# ./dd2vmdk -i /data/images/vmpersonal.dd -v /tmp/output.vmdk

Author:   JD Durick <jd@vmforensics.org>
Program:  dd2vmdk
Website:  http://vmforensics.org
Version:  0.1.1

Starting the MBR analysis...
Finding which partition is bootable
Getting cylinders value from partition1...
Getting ending cylinders value from partition1...
Getting sectors per track value from partition1...
Getting number of heads value from partition1...

Image Geometry specifications:
================================
1.  Image location:  /data/images/vmpersonal.dd
1.  VMDK destination:  /tmp/output.vmdk
2.  Number of sectors = 62914560
3.  Number of cylinders:  1023
4.  Number of heads per track:  255
5.  Number of sectors per track: 63
6.  File size in bytes: 32212254720 (30.00 GB)
7.  Size of each sector: 512
8.  Bootable partition:  Partition 1

Two byte signature word:  55AA (End of MBR)

It can be downloaded HERE or you can grab the source code via this link:

svn co https://dd2vmdk.svn.sourceforge.net/svnroot/dd2vmdk dd2vmdk

Related articles by Zemanta

• Computer Forensics: Identifying Disk Differences - Broken Mirrors (blogs.sans.org)
• Partition Recovery When Disk Administrator Corrupts Windows Partitions (badcreditdebtmanagement.com)
• Windows MBR and Advanced Format Drives (e512) (blogs.sans.org)
• Partition Assistant v2.1 (lockergnome.com)

I am in the process of make a bunch of smaller tools instead of one big tool.  mbrChunker will not be broken up into smaller pieces and makes the code more manageable.  dd2vmdk is a *nix-based program that allows you to mount raw disk images (created by dd, dcfldd, dc3dd, ftk imager, etc) by taking the raw image, analyzing the master boot record (physical sector 0), and getting specific information that is need to create a vmdk file.  Version 0.1 has the ability to extract data from the master boot record of any raw image, extracted information such as heads
 cylinders, sectors per track, etc. and create a working vmdk file (monolithic flat disk). 

Running the tool on a dd image of an existing VMDK (30GB) file, these are the results:

root@redbox:/data/projects/C/dd2vmdk# ./dd2vmdk -i /data/images/vmpersonal.dd -v /tmp/dualboot.vmdk
image path = /data/images/vmpersonal.dd
vmdk path = /tmp/dualboot.vmdk
sizeof(off_t) = 8.
total size:  32212254720

Author:   JD Durick <jd@vmforensics.org>
Program:  dd2vmdk
Website:  http://vmforensics.org
Version:  0.1

Image Geometry specifications [/data/images/vmpersonal.dd]
================================
1.  Image location:  /data/images/vmpersonal.dd
1.  VMDK destination:  /tmp/dualboot.vmdk
2.  Number of sectors = 62914560
3.  Number of cylinders:  1023
4.  Number of heads per track:  255
5.  Number of sectors per track: 63
6.  File size in bytes: 32212254720
7.  Size of each sector: 512

You can download  dd2vmdk-0.1.tar.gz HERE and HERE.

Related articles by Zemanta

• Computer Forensics: Identifying Disk Differences - Broken Mirrors (blogs.sans.org)
• How To Perform A Backup Recovery in Linux (brighthub.com)

The hex byte searching functionality was flawed and had to be rewritten. Additionally, it could not handle files larger than 2-3gb in size but now can handle files of any size given the new redesign. Added a -b flag to give the user customization on how big the buffer will be when searching for your hex string. All three functions, dd2vmdk, MBR analysis, and hex byte string searching work fine under any Linux OS. 

Download:

http://vmforensics.org/mbrChunker/mbrChunker-0.3.17.tar.gz

Looks like someone beat me to the punch and converted the Open source VMFS drivers originally written in java to C.  In addition, he added some new features to his vmfs toolset such as extents and allow you to access VMFS through the standard Linux VFS with the help of the FUSE framework.  I suggest all who work with VMFS partitions to check out vmfs-tools.  To get the tools compiled on my ubuntu box, I had to do the following:

root@redbox:/tmp/vmfs-tools-0.2.1# apt-get install libfuse-dev
root@redbox:/tmp/vmfs-tools-0.2.1# apt-get install asciidoc
root@redbox:/tmp/vmfs-tools-0.2.1# apt-get uuid-dev

I was missing these three and was unable to get it compiled until I grabbed these packages via apt-get.  The debugvmfs tool appears to be the best tool for analyzing the VMFS partition.  Here is the options for the programs:
root@redbox:/tmp/vmfs-tools-0.2.1/debugvmfs# ./debugvmfs -h
debugvmfs v0.2.1
Syntax: debugvmfs <device_name...> <command> <args...>

Available commands:
  - cat : Concatenate files and print on standard output
  - ls : List files in specified directory
  - truncate : Truncate file
  - copy_file : Copy a file to VMFS volume
  - chmod : Change permissions
  - mkdir : Create a directory
  - df : Show available free space
  - show_file_blocks : Show file blocks
  - get_file_block : Get file block
  - check_file_blocks : Check file blocks
  - check_vol_bitmaps : Check volume bitmaps
  - show_heartbeats : Show active heartbeats
  - read_block : Read a block
  - dump_block : Dump a block in hex
  - get_block_status : Get block status
  - alloc_block_fixed : Allocate block (fixed)
  - alloc_block : Find and Allocate a block
  - free_block : Free block
  - show_bitmap_item : Show a bitmap item
  - show : Display value(s) for the given variable
  - shell : Opens a shell

Using it on my test VMFS dd image, I get the following:

root@redbox:/tmp/vmfs-tools-0.2.1/debugvmfs# ./debugvmfs /mnt/mnt/evidence/vmfstest.img  ls
.
..
.fbb.sf
.fdc.sf
.pbc.sf
.sbc.sf
.vh.sf
esxconsole-4b2abf93-d71f-2613-1a26-00137268d17a
maldns
Bodog
converted
rootdns
winxp_personal
windows xp - web
xp-web
bodog
win2003


Looking at the TODO file, here is what we can expect from them down the road:
Expected for v0.3.0:
- Test suite
- Make debugvmfs more of a VMFS debugging tool and less of a vmfs-tools testing tool
- Better fsck.vmfs

Expected for v0.5.0:
- Stable internal API
- mkfs.vmfs

Expected for v1.0.0:
- Full write support.

Any time:
- Check if we want to read/update separately metadata header from the metadata
  itself (for example: vmfs_inode_read/vmfs_inode_write)
- Add locking/unlocking/update functions for metadata headers
- Improve build system and portability


The source code looks pretty extensive and they put some time into this project.  I definitely recommend you giving their tool a test drive.

Related articles by Zemanta

• Newly formatted 1 TB+ shared VMFS datastore went missing after few days of operation. (edugeek.net)
• ESX4 cannot create datastore (edugeek.net)

Unfortunately, I will not be attending BH/defcon this year however there are going to be some good talks.  One of which you shouldn't miss is the one by Christiaan Beek on Virtual Forensics.  He gave a similar talk at BH-Europe however has added some new things for the BH USA version.  Christiaan has a pretty cool site over at http://securitybananas.com/ and definitely recommend you checking out his site.  Another talk that looks to be interesting is from Claudio Criscione on pentesting virutalization, information on this talk can be found here.  Acutally the whole second day in Augustus room 3 and 4 has talks on Cloud Virutalization - all look to be interesting and wishing I was going after all.  Defcon, (I kind of miss the old days of the alexis park hotel) is at the Riviera and looks pretty decent, here is the schedule for the weekend party!

Bytelocator v0.1.2 was released, added a -b flag so the user could give the block size value for the buffer to be search rather than be hard coded.  Cleaned up some code and added some more documentation.  Works fine if you are looking for the starting location of any hex byte value, functionality will probably be integrated with other another tool to address mbr analysis.

Was dinking around with the VDDK API's and saw this post and thought I would post if for anyone interested in understanding why you are not able to create a second VMFS partition via fdisk and only via the VI client.  It appears that you need to remove the lock that ESX has on the device! Thought I would pass this tip along...will play with it later tonight...

http://www.virtualvcp.com/content/view/67/26/

Related articles by Zemanta

• ESX4 cannot create datastore (edugeek.net)
• File Systems Used in Pc Operating Systems (computersight.com)
• Try a live Distro without burning on CD or running in VM (amit-agarwal.co.in)

Made some changes to byelocator v0.1 and the hex functionality now works by analyzing every 1k chunks as opposed to trying to read in the entire file which was insane.  It is working fine and will get you the starting byte location of any hex string up to 16 bytes long.  Error checking is limited but will be added soon.  You can download bytelocator over at sourceforge.  Here is a small snapshot of the program after it has searched a 6gb file:

root@redbox:~/workspace/bytelocator/Debug# ./bytelocator -x BF1B0650 -i /data/images/sixgbimage.img

Total size of /data/images/sixgbimage.img file:  6448619520 bytes
Parsing the hex string now: BF1B0650

Hex string BF1B0650 was found at starting byte location:  18
Hex string BF1B0650 was found at starting byte location:  193885738
Hex string BF1B0650 was found at starting byte location:  194514442
Hex string BF1B0650 was found at starting byte location:  525033370
Hex string BF1B0650 was found at starting byte location:  1696715251
Hex string BF1B0650 was found at starting byte location:  1774337550
Hex string BF1B0650 was found at starting byte location:  2758859834
Hex string BF1B0650 was found at starting byte location:  3484416018
Hex string BF1B0650 was found at starting byte location:  3909721614
Hex string BF1B0650 was found at starting byte location:  3999533674
Hex string BF1B0650 was found at starting byte location:  4018701866
Hex string BF1B0650 was found at starting byte location:  4077977098
Hex string BF1B0650 was found at starting byte location:  4098838010


Quick stats:
================
Number of bytes that have been read:  6448619520
Number of signature matches found:  13
Total number of bytes in hex string:  4

Related articles by Zemanta

• # - the Unix truth (homepages.cwi.nl)
• visual studio memory (ask.metafilter.com)

bytelocator is a small C command-line utility that allows you to search for hex byte patterns within any binary file. Its provides information on the byte location within a binary file based on the hex byte pattern you provided.  Additionally, it can be used on images or files where forensics analysis might be needed such as hex patterns within a master boot record.  I just created a project on sourceforge and bytelocator can be found here.  As stated, it is pre-alpha stage so it works and gets you the information you need however contains bugs.  Enjoy.

Had an issue with some code when I was trying to read in files over 4GB in size.  Don't know why, but I was trying to read the whole thing into memory, they do the hex searching.  Here was the original code which worked for smaller 1-2gb files but nothing bigger than 3-4GB:

char *getBuffer(FILE *fptr, long long size) {
    char *bfr;
    size_t result;
    printf("size of file in allocate buffer:  %lld\n", size);
        //size here is 6448619520

    bfr = (char*) malloc(sizeof(char) * size);
    if (bfr == NULL) {
        printf("Error, malloc failed..\n");
        exit(EXIT_FAILURE);
    }
        //positions fptr to offset location which is 0 here.
    fseek(fptr, 0, SEEK_SET);
        //read the entire input file into bfr
    result = fread(bfr, sizeof(char), size, fptr);
    printf("result = %lld\n",  (long long) result);
    if(result != size)
    {
        printf("File failed to read\n");
        exit(5);
    }
    return (bfr);

}

The new code now fixes that and will now have to break it up into chunks
first, then do the hex searching.  Here is the somewhat working code I put
together tonight to make it work:

 while((rdbytes = fread(filebuffer, sizeof(unsigned char), sizeof(filebuffer), fptr)) > 0) {
       // printf("%d bytes read and number of runs:  %d.\n", rdbytes, runs);
        bufferptr = filebuffer;

        if(rdbytes > (BUFSIZE -1)) {
            for(z = 0; z < BUFSIZE; z++) {
                for (k = 0; k < BUFSIZE; k++) {
                    if (*bufferptr == buffer[cursearch]) {
                        cursearch++;
                        if (cursearch > (counter - 1)) {
                            cursearch = 0;
                            startingoffset = 0;
                            startingoffset = k - (counter - 1);
                            if(runs < 2) {
                                printf("Hex string %s was found at byte location:  %d\n",
                                       hexstring, startingoffset);
                            } else{
                                printf("Hex string %s was found at byte location:  %d\n",
                                       hexstring, startingoffset+BUFSIZE);
                            }
                       
                            matches++;
                        }
                    } else {
                        cursearch = 0;
                    }
                    bufferptr++;
                }
                printf("[%d]: %02X\n", offsetctr, filebuffer[z] & 0xff);
                offsetctr++;
            }
        }

This is somewhat of a workaround where  BUFSIZE is a const value and I read
in to the buffer 2 or 4k of bytes, do the analysis and hex searching, then
print out the matches.  However, I will now have to address the problem
of finding a string hit on the last byte within a buffer.  Looks like
I am going to have to test follow on buffers.  I know there is a better way
in doing all this but I am tired and can't really think straight!! I was 
fortunate to get some help from the guys at stackoverflow.com.

A new feature has been added that allows the end-user to search for hex byte patterns within your forensics images (or any binary file). It allows for you to enter any number of bytes for your hex pattern. It returns via stdout all locations within your binary file where that byte hex pattern was found and its corresponding byte offset location. Specifically, I want to incorporate this functionality against large dd imaged VMFS partitions which would grab all master boot record sectors and analyze them for the user.  Below is a snapshot of the features of mbrChunker 0.3.15:

root@redbox:/data/projects/C/mbrChunker-0.3-release# ./mbrChunker -h
Program: mbrChunker
Author:  JD Durick <jd@vmforensics.org>
Website: http://vmforensics.org

Description:  mbrChunker is a *nix-based program that allows you to mount raw disk images
as well as perform analysis on partition tables within the master boot record.  Furthermore,
mbrChunker can search for hex patterns within any type of binary file
Usage: ./mbrChunker -i <path to dd image> -v <output path of vmdk>...

OPTIONS:
   -i image file name that you will be analyzing
   -v VMDK metadata file that you will be creating for your dd2vmdk conversion
   -a creation of a metadata file to be created (file location required)
   -x search for hex code patterns within your file (not 0x80, just 80)
Examples:
./mbrChunker -x <hex bytes> -i /data/images/testsector.img
./mbrChunker -i <raw image path location> -v <vmdk output file location>
./mbrChunker -i <raw image path location> -a <output of analysis file>

The code is still raw but workable and if it helps someone else, then great.

enjoy...

Related articles by Zemanta

• Seagate 3TB Drive Raises Storage to New Heights (pcworld.com)
• Help me recover a GPT table! (ask.metafilter.com)
• Recently Released EASEUS Partition Master 6.0.1 Warmly Welcomed as Reliable GPT Disk Partition Manager (prnewswire.com)

I don't know how I missed this but in Nov. 2009, ENISA came out with the Cloud Computing Risk Assessment guide.  Below is a snippet on the document and what it actually entails:

ENISA, supported by a group of subject matter expert comprising representatives from Industries, Academia and Governmental Organizations, has conducted, in the context of the Emerging and Future Risk Framework project, an risks assessment on cloud computing business model and technologies. The result is an in-depth and independent analysis that outlines some of the information security benefits and key security risks of cloud computing. The report provide also a set of practical recommendations.Produced by ENISA with contributions from a group of subject matter expert comprising representatives from Industry, Academia and Governmental Organizations, a risk assessment of cloud computing business model and technologies. This is an in-depth and independent analysis that outlines some of the information security benefits and key security risks of cloud computing. The report provide also a set of practical recommendations. It is produced in the context of the Emerging and Future Risk Framework project.   You can download it HERE.

As you know from previous post, I talked about the Virtual Disk Development Kit (VDDK) whihe is a collection of C libraries, code samples, utilities, and documentation to help you create or access VMware virtual disk storage.  After downloading them on my ubuntu box, I gave the sample program a try to see how the thing works before I start modifiying it for my own use.  Below are my steps for fixing/patching the sample program that uses the VDDK libraries.

Fixing VDDK sample program and make sure it compiles correctly

1.  Install VDDK 1.1 from http://www.vmware.com/support/developer/vddk/
2.  Find the sample source in /usr/share/doc/vmware-vix-disklib/sample
3.  You should find the following:

root@*****:/data/vmwareproject/CODE/sampleprogram# ll /usr/share/doc/vmware-vix-disklib/sample
total 48
-r--r--r-- 1 root root   158 2009-12-04 08:02 Makefile
-r--r--r-- 1 root root 41718 2009-12-04 08:02 vixDiskLibSample.cpp
root@*****:/data/vmwareproject/CODE/sampleprogram#

4.  Make sure you edit your /etc/ld.so.conf file, here is what is should look like:


root@stalker:/data/vmwareproject/CODE/sampleprogram# more /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf
include /usr/lib/vmware-vix-disklib/lib32

root@stalker:/data/vmwareproject/CODE/sampleprogram#


5.  After editing the /etc/ld.so.conf file, run "ldconfig"
6. cp -r /usr/share/doc/vmware-vix-disklib/sample  to /tmp
7.  cd /tmp
8.  run "make"
9.  There is a problem with the vixDiskLibSample.cpp file and does not contain all the header files...here is the error you should get...
10..  Run make
 
Results:

root@stalker:/data/vmwareproject/CODE/sampleprogram# make
g++ -o vix-disklib-sample `pkg-config --cflags --libs vix-disklib` vixDiskLibSample.cpp
vixDiskLibSample.cpp: In function 'int PrintUsage()':
vixDiskLibSample.cpp:506: error: 'printf' was not declared in this scope
vixDiskLibSample.cpp: In function 'int main(int, char**)':
vixDiskLibSample.cpp:587: error: 'strdup' was not declared in this scope
vixDiskLibSample.cpp:606: error: 'printf' was not declared in this scope
vixDiskLibSample.cpp: In function 'int ParseArguments(int, char**)':
vixDiskLibSample.cpp:674: error: 'strcmp' was not declared in this scope
vixDiskLibSample.cpp: In function 'void DoFill()':
vixDiskLibSample.cpp:971: error: 'memset' was not declared in this scope
vixDiskLibSample.cpp: In function 'void DoDumpMetadata()':
vixDiskLibSample.cpp:1092: error: 'strlen' was not declared in this scope
vixDiskLibSample.cpp: In function 'void DumpBytes(const unsigned char*, size_t, int)':
vixDiskLibSample.cpp:1184: error: 'printf' was not declared in this scope
vixDiskLibSample.cpp:1199: error: 'printf' was not declared in this scope
make: *** [vix-disklib-sample] Error 1


11.  To fix this, create a patch and execute the patch on your file.  The steps below outline this for you.

root@******:/data/vmwareproject/CODE/sampleprogram# more create_patch_file.txt
Script started on Fri 04 Dec 2009 10:21:27 AM EST
root@******:/data/vmwareproject/CODE/sampleprogram# diff -uN vixDiskLibSample.cpp vixDiskLibSample_new.cpp > vixDiskLibSample.patch
root@@******:/:/data/vmwareproject/CODE/sampleprogram# more vixDiskLibSample.patch
--- vixDiskLibSample.cpp    2009-12-04 10:17:49.187001708 -0500
+++ vixDiskLibSample_new.cpp    2009-12-04 10:18:28.086448746 -0500
@@ -23,6 +23,9 @@
 #include <string>
 #include <vector>
 #include <stdexcept>
+// JD Durick added this in to account for printf and memset function calls - 12/04/09
+#include <stdio.h>
+#include <string.h>
 
 #include "vixDiskLib.h"
 
root@@******:/:/data/vmwareproject/CODE/sampleprogram# ll
total 104
-rw-r--r-- 1 root root   361 2009-12-04 10:21 vixDiskLibSample.patch
-rw-r--r-- 1 root root     0 2009-12-04 10:21 create_patch_file.txt
-r--r--r-- 1 root root 41843 2009-12-04 10:18 vixDiskLibSample_new.cpp
-r--r--r-- 1 root root 41718 2009-12-04 10:17 vixDiskLibSample.cpp
drwxr-xr-x 2 root root  4096 2009-12-04 10:09 new/
drwxr-xr-x 2 root root  4096 2009-12-04 10:09 orig/
-r--r--r-- 1 root root   158 2009-12-04 10:00 Makefile
root@@******:/:/data/vmwareproject/CODE/sampleprogram# patch vixDiskLibSample.cpp < vi
xDiskLibSample.patch
patching file vixDiskLibSample.cpp
You have new mail in /var/mail/root
root@@******:/:/data/vmwareproject/CODE/sampleprogram# make
g++ -o vix-disklib-sample `pkg-config --cflags --libs vix-disklib` vixDiskLibSample.cpp
root@@******:/:/data/vmwareproject/CODE/sampleprogram# exit
exit

Script done on Fri 04 Dec 2009 10:24:07 AM EST
root@@******:/:/data/vmwareproject/CODE/sampleprogram#


At this point, everything should work fine.

Added some extra functionality to mbrChunker with version 0.2 - mbrChunker can now parse each of the primary partition entries, analyze each of the 16 byte entries for information such as boot indicator, starting head, starting sector, starting... cylinder, system ID, ending head, ending sector, ending cylinder, and total sectors. It also validates the partition types of each of the partition entries. As of now, the tool can convert DD images into a flat VMDK file and analyze each of the partition entries found within the master boot record (physical sector 0) of a hard drive.  If you want to give it a try, you can download mbrChunker-0.2.1.tar.gz off my website or from Freshmeat.Below are some links from my friend zemanta!

Below is a quick snapshot of the analysis portion of the tool:

View image

Related articles by Zemanta:

• Bigger is Not Always Better with Hard Drive Storage (pcworld.com)
• How to get Linux LVM and dm-crypt to play nice with Truecrypt? (ask.metafilter.com)
• How To Fix Bootloader Problems Due To GRUB (makeuseof.com)
• 7 Steps to Drive Striping in Windows 7 (brighthub.com)

In the /dev/random category, starting July 3rd, live poker is now available in Charles Town, WVA which means that I am about 1 hour from a LEGAL poker room.  It sure beats doing those 3.5 hour trips to Atlantic City!  Will be trying to make it up there in the next month or so.

I just got my copy of Virtualization and Forensics by Diane Barrett and Gregory Kipper and will be reading it over the weekend.  In the next few weeks or so, I will be commenting on the book as it was just released by Amazon on June 1st - given that there is not a whole lot of documentation on the subject, I am eager to see what this book actually entails.  To give you an idea of what the book talks about here is some of the areas within the TOC.  I do recommend buying the book if you are interested in this subject area.

Chapter 1:  How virtualization happens?
Chapter 2:  Server Virtualization
Chapter 3:  Desktop Virtualization
Chapter 4:  Portable virtualization, emulators, and appliances
Chapter 5:  Investigating Dead virtual environments
Chapter 6:  Investigation live virtual environments
Chapter 7:  Finding and imaging virtual environments
Chapter 8:  Virtual environments and compliance

and more....


As always, I like to provide some related articles to VM Forensics:

• Desktop Virtualization with MokaFive (readwriteweb.com)
• What Is Virtualization and How Does It Work for Your Small Business? (myventurepad.com)
• Wyse and Citrix Joint Desktop Virtualization Deployments Highlight Optimized Simplicity, Security, ROI and User Experience (eon.businesswire.com)
• Digital Forensics Case Leads: Data Exposed, Movie Piracy Sites shutdown and a 0day exploit hits more the 10,000 Computers (blogs.sans.org)
• OPEN TO ALL - Digital Forensics Awards Night - 8 July 2010 (blogs.sans.org)
• The SANS Institute's Digital Forensics Lethal Forensicator Coin (blogs.sans.org)

Just saw on the forums that the sleuthkit 3.1.3 is now available.  Additionally, Brian Carrier's slides from Sleuth Kit and Open Source Digital Forensics Conference in Chantilly, VA is now available.  For anyone who has been using forensics tools on *nix boxes, you have to have used TSK.

A while back, I was looking for data within dns traffic and wrote this perl script called dns view. Basically, it took in gzipped tcpdump files within a directory, read through the directory, opened the files, searched for various terms within the packet payload. I used a config file, checked for the header and verified that it was a tcpdump file by looking at the first 20 bytes of the tcpdump file. There is some hard coded lines in there so beware but thought I would post it in case it might help someone else search for data within the packet payload of DNS traffic. BTW, the code is beta at best and has not been tested thoroughly - but should work for you!

#!/usr/bin/perl ####################################################################### # Created on: 01/06/08 # File: dnsview.pl # Version: 0.1 # Description: reads pcap file (offline) to conduct both header and # packet analysis. Initally written for DNS packet captures but can # be applied to all types. # # @author(s) jd@vmforensics.org # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation, using version 2 # of the License. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # ###################################################### # the whole gang is here use strict; use warnings; use Net::Pcap; use File::Find; use File::Basename; use NetPacket::IP; use POSIX qw(strftime); use NetPacket::TCP; use NetPacket::UDP; use NetPacket::Ethernet; use NetPacket::ICMP; use Getopt::Std; # DNS port use constant DNS_PORT => 53; my ( $pcapfiles, $gzfiles, $corruptfiles, $sigfileread, $PacketsProcessed, $dnspackets ) = 0; my $sigfile = "headersig\.txt"; my $configfile = "searchterms.txt"; my @termlist = (); my ( $pcap, $err, $result, $net, $mask, $filter, $s, $s1, $d, $d1, $src, $dst, $pair, $num, $two, $key ) = ""; my ( %sigs, %opt, %sources, %dests, %toptalkers, @spair, %h1 ) = (); my %conf = readconfig($configfile); foreach $key ( sort keys %conf ) { print "$key"; } # command-line functionality getopts( "hd:", \%opt ); # bomb unless ( $opt{d} ) { printUsage(); } if ( $opt{h} ) { printUsage(); } # lets throw in a signal catcher if we want to stop the perl script # midway, it then calls givemestats() $SIG{'INT'} = \&givemestats; # make sure our directory is defined if ( defined $opt{d} ) { # read in configuration file (contains list of term we will be using) # @termlist = readconfig($configfile); if ( -e $sigfile ) { if ( readsigfile($sigfile) ) { print "Signature file sucessfully read.\n"; $sigfileread = 1; foreach ( keys %sigs ) { my @list = @{ $sigs{$_} }; } } } else { print "Signature file not read\n"; } # using find for recursion into lower levels of directories # goes as far as *nix will allow it, 255 directories find( \&search, $opt{d} ); # Test printf "%-30s\t%s\n", "Total valid pcap files", $pcapfiles; printf "%-30s\t%s\n", "Total GZIPPED files", $gzfiles; printf "%-30s\t%s\n", "Total corrupted files", $corruptfiles; printf "%-30s\t%s\n", "Total packets processed", $PacketsProcessed; printf "%-30s\t%s\n", "DNS packets processed", $dnspackets; &givemestats; } # packet capture descriptor $pcap, loop forever and call process_pkt # The header information is a reference to a hash containing the following fields. # * len - the total length of the packet. # * caplen - the actual captured length of the packet data. This corresponds to the snapshot length parameter passed to open_live(). # * tv_sec - seconds value of the packet timestamp. # * tv_usec - microseconds value of the packet timestamp. sub process_pkt { my ( $UserData, $Hdr, $Pkt ) = @_; my $EthObj = NetPacket::Ethernet->decode($Pkt); my $IPObj = NetPacket::IP->decode( $EthObj->{data} ); # add in loop for udp searching here (for terms) if ( $IPObj->{proto} == 17 ) { #UDP my $UDPObj = NetPacket::UDP->decode( $IPObj->{data} ); my $Payload = $UDPObj->{data}; if ( $UDPObj->{dest_port} == DNS_PORT ) { my $timestamp = strftime( "%Y-%m-%d %H:%M:%S", gmtime($Hdr->{'tv_sec'} )); foreach $key ( sort keys %conf ) { chomp($key); if ( $Payload =~ /$key/ ) { open( FILE, ">>/tmp/output/$key" ); print FILE "$IPObj->{src_ip},$UDPObj->{src_port},$IPObj->{dest_ip},$UDPObj->{dest_port},$UDPObj->{len}, $timestamp\n"; close(FILE); $dnspackets++; # $sources{ $IPObj->{src_ip} }++; # $dests{ $IPObj->{dest_ip} }++; # $toptalkers{ $IPObj->{src_ip} } # { $IPObj->{dest_ip} }++; } } } } else { #Generic packets, do nothing } $PacketsProcessed++; } # Function: search: goes through each directory looking for a certain filetype. # After it finds the file, it tests the file to see if it is in the headersig.txt # file. But for our use today, we will only be looking for all tcpdump, gzipped, # or "bad" signature files. After it determines if it is a tcpdump file, runs a comparison # of search terms, same goes for the gzipped file (but uses zcat piped to tcpdump -r) sub search { my $fpath = $File::Find::name; my $fname = basename($fpath); # hard-coded for now but can be passed in via command line later if ( $fname =~ /log*/ ) { my ( $hex, $resp ) = getsig($fpath); my $match = 0; # looping through parsed headersig file foreach my $key ( keys %sigs ) { # check to see if we match the hex value of the file with the hex # value found in the headersig file if ( $hex =~ m/^$key/ ) { # good, match, lets see if its a valid tcpdump file if ( grep( /pcap/i, @{ $sigs{$key} } ) ) { $pcapfiles++; print "PCAP-$fpath...\n"; $pcap = Net::Pcap::open_offline( $fpath, \$err ) || die("$err"); Net::Pcap::loop( $pcap, -1, \&process_pkt, 0 ); $match = 1; } # do we match a .gz file? We are assuming that when the .gz file # is unzipped, it is a valid pcap file, I need to add this in # as a secondary check later. if ( grep( /gz/i, @{ $sigs{$key} } ) ) { $gzfiles++; print "GZ-$fpath($hex) = @{$sigs{$key}}\n"; $match = 1; } } } # lets find those ones that did not match our list if ( $match == 0 ) { $corruptfiles++; print "File signature not listed - $fpath\n"; } } } # function to print out stats based on packets processed # signal handler function so if an interrupt is called, will print # out stats up to that point. sub givemestats { print "\nSTATISTICS\n"; print "Source IP Addresses:\n"; foreach my $ip ( ( sort { $sources{$b} <=> $sources{$a} } keys %sources )[ 0 .. 19 ] ) { printf "%-20s\t%s\n", $ip, $sources{$ip}; } print "\nDestination IP Addresses:\n"; foreach my $ip ( ( sort { $dests{$b} <=> $dests{$a} } keys %dests )[ 0 .. 19 ] ) { printf "%-20s\t%s\n", $ip, $dests{$ip}; } print "\nMost Frequent Talkers\n"; foreach $src ( keys %toptalkers ) { foreach $dst ( keys %{ $toptalkers{$src} } ) { push( @spair, "$src:$dst:$toptalkers{$src}{$dst}" ); } } foreach my $pair (@spair) { ( $s, $d, $num ) = split( ':', $pair ); $two = "$s:$d"; $h1{$two} = $num; } foreach ( ( sort { $h1{$b} <=> $h1{$a} } keys(%h1) )[ 0 .. 19 ] ) { ( $s1, $d1 ) = split( ':', $_ ); printf "%-20s\t%-20s\t%s\n", $s1, $d1, $h1{$_}; } Net::Pcap::close($pcap); exit(0); } # reads in list of terms we want to search for...assuming we will have a huge list here sub readconfig { my $configfilename = shift; my %conf = (); my @termlist = (); my ( $keyword, $arg1 ); if ( -e $configfilename ) { open( FH, $configfilename ) || die " Could not open $configfilename : $! \n "; while ( my $terms = <FH> ) { # skip lines that begin w/ # or are blank next if ( $terms =~ m/^#/ || $terms =~ m/^\s+$/ ); $conf{$terms} = 1; } } # lets return our list of search terms back return %conf; } # gets the extension of a passed in filename sub getext { my $file = shift; my $ext; my @filelist = split( /\./, $file ); ( @filelist > 1 ) ? ( $ext = $filelist[ @filelist - 1 ] ) : ( $ext = " none " ); return $ext; } # reads the header signature file and parses out the file into three parts sub readsigfile { my $file = shift; if ( -e $file ) { open( FH, $file ) || die " Could not open $file : $! \n "; while (<FH>) { # skip lines that begin w/ # or are blank next if ( $_ =~ m/^#/ || $_ =~ m/^\s+$/ ); chomp; my ( $sig, $tag ) = ( split( /,/, $_, 3 ) )[ 0, 1 ]; my @list = split( /;/, $tag ); # split on "; foreach (@list) { $_ =~ s/\s//; #remove space $_ =~ s/\.//; # remove period } # %sigs is a global variable $sigs{$sig} = [@list]; } close(FH); return 1; } else { return undef; } } # simple USAGE routine sub printUsage { print " Usage: \tpcapquery.pl [d] <location of tcpdump directory> - [o] <output directory> \n "; print "\tDate: 01/06/08- <jdurick\@mitre.org> \n "; print "\nOptions : \n "; print "\t-h - print help (optional) \n "; print "\t -d - directory \n "; print "\t -o - output directory\n"; exit; } # gets the first 20 bytes of the binary header # returns the hex and whether or not it was a success or not (0 or 1) sub getsig { my $file = shift; my $success = 0; my $hex; eval { if ( open( FH, $file ) ) { binmode(FH); my $bin; sysread( FH, $bin, 20 ); close(FH); $hex = uc( unpack( "H*", $bin ) ); $success = 1; } }; return ( $hex, $success ); } __END__