WHAAAA0101 0000 0011T !!! (aka Extracting files out of pcaps with Foremost & discussing TShark)

My SANS SEC503 NetWars coin 

So I thought I would use this as a chance to try out something Dr. Ullrich had mentioned in class: that a tool I use often for forensic recovery, also can extract files from tcp/ip streams present in pcap files--that tool is Foremost--I had just simply not realized that or noticed this feature in the foremost documentation.

IN SUMMARY

This did work, however my idea to extract an audio stream was not as simple as I had thought.   I'll discuss only the simplest part of the capture below, but basically I have an Apple TV (maybe will write quite a bit more about that in future posts, it is a lot of fun to monitor!) captive through a Linux bridge, which I captured about 30 seconds of packets on, so the nature of the the device, or some already-finished handshake from the radio server probably complicated the excersize.   I'd habitually have otherwise used TCPDump if I wasn't wanting to explore TShark a bit.

The strangely altered jpeg, and inabiliti of foremost to extract an audio file could of course be caused by a number of things, for example encryption or simply just incomplete streams.  It is possible with TShark to go to a lot further lengths to ensure you get all the traffic matching specific streams or tcp sessions, I've ignored that here for this post, but is something I am currently also researching.  The HTTP 404 pages which I captured in that same pcap, came out perfectly, so my results may also be about file types supported by Foremost.

Since it was very simple to run Foremost, I'm also just discussing here various commands I ran on the same capture with TShark, to confirm and otherwise just analyze the same packet captures, and really just give a complete exploration of the tshark man pages.

THE PROCESS

I've got an on-going write up about different free tools for forensics and which standard Linux distributions they are in, such as those in Debian like Foremost and Sleuthkit, and I'll re-post that here when I am finished.  As a huge fan of the command line, I had used TShark a bit before, but not realized it was actually part of Wireshark and uses the same packet dissection code.

You can use Foremost to extract files from TCP streams for many purposes, of course, but really simple uses might be to decode unencrypted e-mail attachments or pull some simple files out of a stream if you didn't have Bro handy. For example today I'm going to to try and pull out a short clip of music, which I had streamed from Klassik Radio of Germany, on the Schiller Lounge channel. A full direct list of all the streams used to be here. I later tried it again with the stream on SomaFM from "DefCon Radio," another favorite, because of the amazing relaxing music played there, not even the infosec name! As you'll see below, both did not work, but instead I got some trippy graphics & html files, but I did prove Foremost works for this, in general, and as well I could find a way at a later date to extract music streams if I really wanted to (in this case, it was just the first thing I had thought of to write this post and show off my medal :)).

Many people use wireshark for extractions, which requires a bit of clicking around, and the most useful of all the tools I know of is probably to use a file-extraction script in Bro to extract files from packet streams, but the utter speed of Foremost would make this a great choice to have in your kitbox, similarly to using Tshark for packet analysis.  It blazed the entire pcap I had nearly instantly, and that was on only a Chromebook.

I captured about 30 seconds with tshark -n -vv -w [MyFullCaptureFile.pcap], in which music was playing on the Apple TV, I stopped the music, did an apt-get upgrade on the box, then started the music playing again.

I'd limited down my pcap first, to just the music, as would normally be a good practice for speed and resources, by using tcpdump out of habit:

tcpdump -l -r MyFullCaptureFile.pcap 'host 85.239.108.20' > radio.pcap

Of course you could use TShark to do the same thing, or even use TShark for the extraction with --extract-objects or other options.

THE EXTRACTION

So I fired off: 

foremost -i radio.pcap,

I also tried specifying files types like foremost -i radio.pcap -t mp4 but since there was not a file of this type present, for whatever reason, this did not help here, but it's how you'd limit the command, 

 again, this is applying the general principle of limiting down your input and using tools quickly.

This above did not work! There was no files in that reduced stream, that Foremost liked.

However, when I scanned my original entire capture though Foremost,

foremost -i MyFullCaptureFile.pcap

I very quickly found 3 files. Foremost creates by default a directory called "output" which contains a log, audit.txt, and files in folders sorted by file type. In my case it found html pages (404 returns) from the Debian update I had captured also within the same experimental session and the jpeg below (I am guessing it is some portion of the album art or the screensaver of the Now playing screen on Apple TV).

trippy

THE CAPTIAL AND LOWERCASE

I'd like to simply discuss some of the commands I used to explore the same packets, and as well commands I studied & practiced today in the man pages.   Reasons why I love TShark include it's speed and scriptability, but it is famous, among other things, for its use of the Berkley Packet filters for filter expressions.

The authors mention that as TShark progresses, expect more and more protocol fields to be allowed in read filters, so if you are a frequent user, look out for updated versions.

*The -G option is a special mode that simply causes Tshark to dump one of several types of internal glossaries and then exit. Actually makes loads of text spew to the console :)  Really, what this does is blast out internal settings of Tshark, but by default, if you don't specify one of the hundreds of "glossaries" then it selects for you an option called "fields," which dumps the contents of the registration database to stdout...

a perfect example is that when I wanted to get the names of possible "fields" for the -T fields output mentioned below, I do

tshark -G column-formats

*-f allows you then to input filters in the BPF Syntax (Based on the Berkley Packet Filter), which are handy to learn for a lot of reasons. So note that is unlike tcpdump, where you type more readable filters just at the end of the command like "tcpdump [expression]." I hadn't realized before, that Pre-defined capture filter names, as shown in the GUI menu item (from Wireshark) Capture->Capture Filters, can be used by prefixing the argument with "predef:". Example: -f "predef:MyPredefinedHostOnlyFilter" 

*I'll talk a bit below about how to use the -T "fields" [-e {list of fields}] option, because this really blew my mind the first few times I used it in class with Johannes. You use -e after transferring data into text, json, hex, and other formats with the "-T fields" option.  So a nice take-away here is that in TShark -E capital letter option is used to format the related lower case option:   

• don't the confuse this with the timestamp (-t ) option
• the -T option (perhaps think of as "table output"... gives specific, advanced formatting to the output of decoded packet data.    -T is then followed by

• the -e option which defines the fields &
• the -E option gives specific, advanced formatting to that chosen field output 
• in other words -T  [  -e <field> ] [ -E <field print option> ]

The way we used this in 503 was to cut nice, C-Suite worthy data out of large packet files, way faster than you could ever use Wireshark to do in most circumstances.

For example:  tshark -n -r radio.pcap  -T fields -e tcp.stream | uniq

to find a specific stream number, or optionally add in limitations like -Y 'tcp.src.port == 8000 and tcp.dstport == 80'.  In this example it just showed me the stream number of my small capture.

This of course could be scripted and even pumped directly into stuff like gnuplot or the R programming language to make charts, or even "e-mail a random set of 5 data sets from the previous day each morning to your boss" ;-)

Someone who can't script will soon be scripted out of their job someone once said! Shoutout to Matt Domko and the #python channel at the BrakeSec Slack... and as well as Mendel Cooper aka Grindell with the infamous advanced Bash Scripting guide at the LDP.

The example in the man page is to use:

tshark [-r input {or capture}] -T fields -e frame.number -e ip.addr -e udp -e _ws.col.Info
Note that giving a protocol rather than a single field will print multiple items of data about the protocol as a single field. Fields are separated by tab characters by default.

*The -z (Statistics) option also gives you a waterfall of options, almost Netflow like data (without having to go first to SilK format for example with rwp2yaf2silk, in a way it gives options or summaries that you see with capinfos or other commands; for example it showed me, at the end of my analysis some neat summary of packets with:

tshark -z conv, tcp -r myfile.pcap | tail

Another example we used in SEC503 was tshark -r myfile.pcap -z http_req,tree to see an ASCII "tree" representation of the http requests, very cool!  -z has a huge list of possible switches.

Congratulations to all the other SANS coin winners in Singapore and in Berlin! 

Infosecfeeds.org is up with the "planet" of Security #infosec podcasts (browser-based master feed of #security and #dfir pocasts)

I've really enjoyed a few blogs this year that were published directly on Linkedin, for example the Jiu Jitsui, chess and other blogs by Tim MV; so for the blog component of infosecfeeds.org, I'm testing out the "pages" function of Linkedin.

The homepage url for infosecfeeds.org should feature an aggregation of non-commercial (IT) Security podcasts, soon, if not it's not already up by the time you read this. I will update it and get some cron on it to regularly update it soon; there were also about 10 other feeds I wanted to initially include, including some dead-box-forensics focused shows I hope. InfosecFeeds.org is a planet based site, which is an open sourceproject under the python license. The original planet web app is featured on planet.debian.org for Planet Debian. I really love the format!

So the planet feed was just an idea to contribute to the community, but the main goal of the url was to include a blog link, which I will update after duplicating this blog, to blog.infosecfeeds.org for the entire journal (which I'll probably host somewhere else, or also on Github like the --The site can just as easily link for any other blogging site (for exposure alone, e.g. Blogger.com), or add that Ghost platform that I've wanted to see since Fab from Linux Outlaws touted it so highly (although he is a professional writer!)--and then just link/post the most career-relevant posts here), and dfir.infosecfeeds.org for only the most original research and computer-forensics focused content (which will be a link to here, if that is possible; the posts will all be "articles" here as well, to whichever degree of automation possisble.

This post actually I wouldn't even include there, but in a month or so, once I'm completely happy with the plant page of infosec podcasts, I think it is a great resource to the forensics and active defense communities as well, just as much as a post of original research.

The next super-relevant post should be a comparison of all the open-source linux "distros" already packaged up for Computer Forensics use. The post I mentioned above, will focus on several pre-packaged "forensic" computer systems, and which packages, and which major differences make up for example Sans' SIFT Workstation, REMnux, or any of these vs. just using Kali, or Black Arch, or building your own from the extensive Debian repositories, (tons were "#newinstretch"). I won't include Security Onion, which comes pre-packed with several IDS & IPS solutions; I'd read about that distro last spring in Amanda Berlin's Defensive Security Handbook. Off-topic I'd like to also include sometime soon also, a few off-topic posts about the early stages of the OWASP capture the flag.

Videos from B-Sides Springfield (and other videos from the MO/AR/#nwark Orzarks region) will also be posted soon, as I finish editing them. So whether here, or somewhere else, look soon for the videos from B-Sides Springfield (Missouri), posted one-by-one, as well as some initial blog-posts.

I plan to also include soon not only the first re-posts of other content from Bsides, but also a recap of the rough videos someone shared from Kansas City, including the special intro by Jack Daniels, and very inspiring, team & community-building opening from Chris Nickerson.

Approximately every 2nd or 3rd post, I hope to include the podcast form, which is a learning and case/study lab-based DFIR podcast called "Computer Forensics Podcast." I spent a lot of my life creating music and other content--also my current company is sound-related, even though I'm in networks & security--so don't think I'm jumping on a bandwagon here, but to help fill the void of forensics podcasts, and do sometimes an audio versions is not that much of a stretch for me at all! Any fans ending up back here in a year will agree, I promise. You should expect to hear projects, interviews, tutorial-like how-to's, (also original music from the community here in Berlin), all based around potentially vicious, authority-questioning attitude that every beginner, or always-learner will love. (Ok maybe not love, but brutally naive questions will never be off the table!) . For DFIR-related topics, I tend to approach things sometimes from an off-the-wall pov, whether it is a new topic, or just something I've already built a habit around, so I find that others usually find this very helpful, or at least funny. #knowthyself

I'm also busy studying for SANS SEC503 (GCIA) [I hope!], or whichever class I might get accepted to in Berlin on the work study program for SANS, and reading through Professor Mike O'Leary's book with the Brakeing Down Security Podcast's book club. Join us and read along and work on the labs together.

Peace & best wishes goes out to those experiencing the hurricane right now in Texas & across the south. Years from now we will not forget you!