What's going on.. ?

One team.

    Earlier this week I had referred a friend to my own work about understanding how timestamps in packets work and various command line options for working with them. I had also admired the length, ingenuity, and clarity of a recent post by my friend Carrie about cracking passwords with special characters. Those moments were inspiring enough that I knew it was finally time to share something more than a tweet's length about what I'd been up to for the last months. So in tribute to my no frills blog title "Computer Forensics blog - Searching to find what's really going on...," here is "what's been going on." The goal is just to briefly shout-out some projects I've been working on, and introduce several upcoming diaries that I've drafted over this past year and will share soon.

    I'm focusing professionally at the moment on the relationship between "advanced defense" strategies (the first couple that come to mind would be threat-hunting and deception tactics) and live detection and response. How far should we take defense-in-depth at the expense of having overlapping signatures? How can we decrease detection times? In an up-coming post later this week, you'll see that when Peter Kim (innovative red-teamer and author of The Hacker Playbook series of books) came live onto our Brakeing Down Security [sic] Book Club call for an interview and discussion, he said that an ideal maximum detection time for blue-teamers would be somewhere around 4 minutes for an initial access or specific action of an adversary. Beyond that a skilled technician would be long gone, pivoted on to the next box, or gone forever, tracks-covered and all.

    Speaking of threat-hunting, I had the opportunity to volunteer a few weeks ago at the Sans Threat-Hunting and Incidence Response Summit in New Orleans, which was a really intense experience. I worked at the summit, made a lot of new friends, and as well helped-out with Matt Bromiley's FOR508 class, which I had never attended before; and as well there my team won the FOR508 Challenge Coin, which was a very powerful experience because of the huge amount of competition in the room. We found a lot of forensic data early on in the event, but kept working hard and fast until the very last minute of the challenge. A lot of our success was due to the experienced forensicators on our team, but none of us would have won alone. So much more could be said about team-work, diligence, and humility! Although there was no hidden message to decrypt though on the coin, like on my SEC503 coin. You should expect to read a few reports from me eventually about things I've learned at the summit.

    I relocated my whole family in the past year, all the way from Berlin (you may have noticed the "Molecule Man" sculpture staring cameo in some of my past posts) back to the USA, because of an amazing work opportunity. As you can imagine this has brought also a lot of excitement to our lives, so the past year was heavily focused on family as well.

    I do still have interest in OSX (and unix in general), as well as the (gentoo-based) ChromeOS platform, which with the beta "Linux mode" is arguably the first affordable, ready-to-go Linux laptop anyone around the world can grab for even less than $100. The up-coming post I mentioned above with Peter Kim will launch the publication of a co-author/guest-series of blogs which I've worked on over the past year, but never published. There will be another post co-authored by a special guest, about using Linux on ChromeOS. There have also been a couple of other topics I've researched, for example the evolution of the packet filtering on OSX, which I found so much public information readily available on, that I don't need to blog about, but I'd love to share some of my favorite other-folks'-work on it soon. I've been also generally supporting the efforts of Mental Health Hackers. Hudson Harris gave a great, related talk at Sp4rkCon By the Bay last week--including a section about procrastination and anxiety--which was targeted at security professionals.

    I started up a Lean-In Circle at work, called the InfoSec Book Club. It's open to everyone, but the mission is to help fight anything that remains of gender-bias in InfoSec. This is based on the BrakeSec Book club, which I co-lead with David Cybuck. We're about to finish up Attacking Network Protocols there (we also just finished also an awesome "Summer Reading Series" led my David, including some author interviews, it was amazing to get to know Mastermind author Evan Ratliff as well). I've enjoyed contributing to the Atomic Red Team Project with several friends, thanks again to Carrie Roberts to introducing me to this project. It's been cool to also meet a few members of the Red Canary team, such great folks. I continue to remain active in the DShield/Internet Storm Center community and manage some honeypots. My goal this year for this blog is actually to migrate slowly out of these guest posts and catch-ups, and over the next year or so release more diaries about threat-hunting and other forensic investigations, getting back to the roots of why I started blogging. Additionally I've recently been trying to spread the word about Roberto and Jose Rodriguez's new efforts on building a community around their awesome open-source tools: many of which are now collected in a single place, The Hunters' Forge; I met them at the summit a few weeks ago, and I'm excited to see where that goes.

    I'm gearing-up to take the GCFA and another test over the next weeks, so that is my very short-term focus, and of course, adjusting to the season and the holidays with my family.  Contact me on Twitter or leave a comment below if any of this has given you an idea or inspiration in any way, or if you're working on any of the same topics. I look forward to learning with you, as well.

Packet-analysis and timestamping

It's not enough to just know your tools; a great engineer knows their tools inside out.

You're going to eventually have to analyze some packets in the cloud, or on any other headless system, or generally this may just be the fastest way to get a job done.  You might also not have access to the man or info pages--for example if you are working with a minimal system, whether that is on an Industrial Control System (ICS) or any other type of "IOT" embedded device system--or if you've even installed just the "core," "cli-extra," or similar version of a distribution on purpose. Either way you better have read the manual.

One thing everyone notices on their first spin through the tcpdump and tshark man pages are the huge amounts of options for timestamping, and how they are completely different in the two often-used-together programs.  I'll show you how to easily remember them.

Tcpdump is on most unix systems, or can be quickly installed.  To get more application-layer visibility, if you didn't already realize it you can install just tshark in most unix distributions without installing the very large, entire Wireshark package.  After installation 'tshark -V -r [file]' will give you just about all the information that you get with a quick skim of the entire Wireshark GUI, and include also option '-x' to add the hexadecimal view that many of us know and love (this adds the packet in hexadecimal at the end of both the quick line-display mode, or the -V and -O [protocol] verbose output modes).  The "expert info" from the Wireshark menus is even there (it's fun to grep out!), and as well the -z [statistics] option will provide you with countless hours or more fun (of endless commands to learn).

Where do timestamps come from?

Short answer, the kernel.  For the real answer, as usual, look at the source.  For example, the file util-print.c has some of the code where timestamp precision is called, and various options are parsed, if you'd like to take a look, you can appreciate the complexity of it, but see a tiny bit of what is going on. 

While browsing the source, I also noticed that the folder also had some short awk programs intended as examples of ways to reduce tcpdump data when you're tracking particular network problems.  Nanosecond resolution is definitely supported.

The timestamp is taken from the kernel, and the resolution can be as accurate as the kernel allows, and your libpcap configuration.  This is beyond the scope of this post, but for neat discussion about nanoseconds and the current limitations of this on Golang see google's github page.  Python seems to be able to handle it.

For a more in-depth discussion, which is really relevant to this post, I'd send you to read this post on the evidence.au forensics blog.

They note that "The accuracy of a timestamp depends on many factors including the performance of the system used to capture packets, its operating system, configuration, and more. On a Microsoft Windows computer the WinPCAP library is synced with the computer clock only at the beginning of the packet capture."

Command line tips

Here's how to remember the two different types of timestamp commands:
TCPDUMP "All the t's" and ...TSHARK "tame the shark, with a single't'"

I've not given any terminal standard output here, save one example.  You have to do that yourself, now grab a few seconds of traffic with 'tcpdump -w [output file.pcap]' and then try each as you read along; it will enhance your memory of the topic.

TCPDUMP "All the t's"

The only real trick to tcpdump, which is easy to remember once you've done it a few times, is that a single t remove the timestamp.  In other words, 'tcpdump -t -n -r [packets.pcap]' gives you output with no timestamps, and no dns resolution [-n].  Both programs also use all lower-case t's for time options.

         tcpdump timestamps
-t      Don't print a timestamp on each dump line.
-tt     Print the timestamp, as seconds since January 1, 1970, 00:00:00, UTC, and fractions of a second since that time, on each dump line.
-ttt    delta (micro-second resolution)
-tttt   date + hours, minutes, seconds, and fractions of a second
-ttttt  delta (micro-second resolution) between current and first line

TSHARK "Tame the shark, with a single 't' "

So basically you type "-t" add add "r" (relative), "a" (absolute), and "u" (for UTC, similar to the older Greewich mean time), and everything else really makes sense.  If you need help with relative and absolute time in packet captures, give it a quick Google search.

You write them just like I've typed out belw, with no spaces, as if each were a different option.  As a tcpdump fan, this will confuse you at first, but the after you realize they are additive, then it makes even more sense the tcpdump and "all the t's," above.  You can get a very handy list by simply typing only "-t," or getting one wrong, which I do all the time!

          tshark timestamps
 -t       get the funny help reminder
 -ta      absolute time
 -tad     absolute including date [YYYY-MM-DD] -tadoy   absolute date but with special YYYY/[day of year]
 -td      delta [change of time between packets]
 -tdd     delta [change of time between *displayed only* packets]
 -te      epoch date [seconds elapsed from January 1, 1970, ...look out 19-Jan 2038!]
 -tu      absolute UTC
 -tuud    absolute UTC including date [YYYY-MM-DD]
 -udoy    absolute UTC date but with special YYYY/[day of year]

So other than remembering that you start with the single "t" (tame the shark with a single 't') and then add options that you want, which make sense, the only one hard to remember--at least for me--is the y, so just remember that "y" gives you the strange year-plus-day-of-year view, which is a more of a special request.

Excersize: 

Perform one example of each of these commands, and note the differences.  Which is your favorite?  Mine's below.

This is my favorite command of the last few days:
#   #By greping the expert mode you can quickly find all sorts of interesting info & anomalous
tshark -V -x -tad -n -r [path to your file.pcap] | grep -i expert
#
# the -V option adds the details, as discussed above, and -x includes hex
# -tad gives a timestamp including a human-readable date, yyyy-mm-dd
# -n leaves out name resolution, to prevent among other things mistakes or adversary surveillance
# grep -i is  great habit to develop on your fingers, it ignores case in the searched term

Grep for a regular expression, but display entire "frame" entry, only

Here's a bonus tip:  to get exactly the full display of one packet (referred to as "frames" in TShark), pipe the command through "grep -A 18 -B 50 [regex]" or "grep -A 13 -B 50 [regex]" if you are not using the -x hexadecimal option.



I've been wanting to practice writing more directly, and where possible more briefly.  So I'd better end it here on this one.  I hope that it's helped you better remember the command-line syntax for working with packets, and depended both our knowledge on the topic and where timestamps come from, namely the kernel.

Thanks to a new friend for sharing this great post as well, from the philosophical questions every day blog, about self-evaluation of creative content, as a means of defeating so-called "imposter syndrome."  Here's a few bonus links about so-called threat-hunting.  Happy hunting!

3 quick tips for sec pro folks: firefox screenshots on the command line, clearing the memcache & swap on linux, & rss syntax of common blogging sites, OSX "raw disk device" aka why dd is slow on Macs

Here's a micropost of several one-liners that you might also find useful:

1. Firefox screenshots from the command line
2. Clearing the memory cache & swap
3. Time-saver: Syntax of RSS feeds of popular blogging sites
4. OSX raw disk device (why dd is so slow on OSX)

Create screenshots of webpages without opening Firefox.

Folks on twitter said this works even more efficiently (i.e. in pentesting) that other larger tools.  From
https://twitter.com/Cherokeejb_/status/996166771739709440, and Credit @botherder, (@amnesty / @citizenlab / @swborders ) for originally sharing this.


Since Firefox 57, the -screenshot flag allows you to take screenshots of websites.

The basic usage is:  /path/to/firefox -headless -screenshot https://reddit.com/r/PositiveNews


This creates a full-height screenshot of the site (here "Positive News" on Reddit), in the active directory called screenshot.png, with a viewport width of 800px.

You can omit -headless when using -screenshot, as it is implied:

I found the entire "headless Firefox" manual here on the Mozilla developer website. It's mostly used for automated testing of different types of web applications and sites.

Clearing the memory cache

I lead one of two weekly discussion groups about security books, which is hosted out of the BrakeSec Slack website of the Brakeing down Security podcast [sic].   The other, "main" group is hosted by David Cybuck.

Here's a link to Chris Sanders giving a nice shout out this week, to our latest session:  https://twitter.com/chrissanders88/status/1002188191779184642

The topic came up, off hand, while we were talking about Cyber Risk Measurement last month, and somehow the question came up about different types of RAM in a linux system, and how to clear them.  Yes, application, "used" memory would be typically cleared after a restart or other power or magnetic event--that is why in forensics you take the volatile things first.  But there are also several ways of clearing the "RAM disk cache."  Don't forget also the swap file on the disk. The references are directly below.

1. Clear PageCache only:   # sync; echo 1 > /proc/sys/vm/drop_caches


2. Clear dentries and inodes:  # sync; echo 2 > /proc/sys/vm/drop_caches


3. Clear PageCache, dentries and inodes:  # sync; echo 3 > /proc/sys/vm/drop_caches 

4.  Clear the swap space (on disk):  # swapoff -a && swapon -a

As with everything on Linux, there are probably several other ways of doing this!

Basic, interesting links on this topic:

https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/

https://www.linuxatemyram.com/

RSS syntax of blogger, medium.com, and wordpress

I shared this originally on twitter, and it inspired this extra "mini-post" today.

Knowing sites' rss syntax is very useful to me, because I love RSS.  Actually I'm hoping to create research time to really learn the in's and out's of RSS and XML, as well as json, as these seem constantly useful in infosec.    I love lean, text-based websites, especially for reading.

Sometimes the browser doesn't give these feeds up to you so easily, especially if you are blocking javascript, as many security folks do!  Sometimes even grep'ing the html source takes a few tries.

You can add a blog to your reader this way, or read it on a low-resource-bearing webpage by knowing this syntax.  All you need is the person's name or site name used on the site.  Here are three examples (substitute below "$user" with the name of the blog).

Medium.com

https://medium.com/feed/ @"$user" 

For example, one of my newer favorite blogs, focused generally on "red teaming" and the management, psychology, and strategy of great teams, Tim MalcomVetter's blog:

https://medium.com/feed/@malcomvetter/

WordPress

https://"$user".wordpress.com/feed/

An example here is one of the BrakeSec slack members, Aaron Gilliland, aka Amish Hacker:

https://amish22hacker.wordpress.com/feed

He's documenting here his path into learning more about security and red-teaming... way to go man!

Blogspot (Blogger.com, Google)

https://"$user".blogspot.com/feeds/posts/default

For example this blog right here, that you are reading now:

https://cherokeejb.blogspot.com/feeds/posts/default

A lot of sites use this blogger engine as well, just with their own domains, so the syntax is very similar.

...there is also a feed for comments; leave one here with your own nice one-liner tricks!

Bonus:  OSX "raw" disk device

Have you ever noticed how slow writing usb or media cards using dd, on Macs is?  That is because you are probably using the wrong device!   I actually thought until recently this was some form of DRM that OSX was deploying, because it took so long, until recently, I had never seen one even finish.

For example, use dd if=/path/to/yourlinuximage.iso of=/dev/rdisk2 bs=4194000

Type first diskutil list , or use the Disk Utility application to find the name of the disk you want to write to.

To write a linux image onto the raw disk device, for example of an inserted SD card that you want to write on a raspberrypi, instead of writing it to, e.g. /dev/disk2 in the example above.  Use the block size appropriate for your media.   This will go much faster.  This is because on many BSD-based systems, the disk devices are a few abstraction layers away from the actual disk metal; it is different than linux.  You learn something everyday!

Linux Privilege Escalation Preview: Last week's "Top" Vulnerability

with a logo like this, who can resist to learn more

An interesting report by Qualys

Last week a rather intriguing alert came across my news feeds via the Debian Security advisory list:

"The Qualys Research Labs discovered multiple vulnerabilities in procps, a set of command line and full screen utilities for browsing procfs..." 

I'd like to use this as an example today of how many strange, unknown attack surfaces must exist on the Unix ecosystem today, and how we must think creatively and diligently to defend against them.

When I saw this report, my initial reaction was "...a set of command line utilities for browsing procfs, awesome, how could I have never seen that before!" ...but then I realized that this is just the official project's way of describing top, ps, w, free, and some other programs which every Unix professional should know in and out (this includes you, Mac folks).  Then it hit me how ultimately critical these vulnerabilities could be, which was not evident just by reading the Debian security bullitin.

The /proc directory on every Linux machine is full of files used to explore and learn more about your system in real time; actually it is literally the kernel manifesting itself into the "runtime," so this is the computer coming to life to interface with the human if that's how you want to visualize it. You can read information out of it to create other applications and even modify it directly. In some ways it's like the Windows registry.  Most actions you take on Linux actually can be achieved in an alternative form by writing over a file somewhere within /proc. An common example is that to create a linux router, you must enable packet forwarding using the sysctl command, but you can also do that temporarily by typing at the command prompt " echo 1 > proc/sys/net/ipv4/ip_forward ." If you look at that file, it is simply a "1" or a "0." It is storing the state of the ip_forward value of your system, so that you can read it and other tools can access it.  One of the first Linux systems I built was on a Powerbook G4 (powerpc arch), where you had to do a similar echo command in order to make the system hibernate or standby.

The power of Proc - why are you reading this..?

1. Procps includes "ps, top, vmstat, sysctl, w, kill, skill, nice, snice, pgrep, free, tload, uptime, watch" according to the sourceforge page.  These tools are a must-know for security engineers. You can literally login and within seconds find everything you need to know (or completely destroy a system) with these commands.  A red team can attack with them or quickly gain knowledge about a system, and a blue team needs to defend against them or use them to analyze a headless system.
2. This is an amazing example of the huge amount of work by thousands of individuals that goes into open source software in general, but specifically the Unix ecosystem. Don't take this for granted.
3. These bugs were of the extra quirky sort, just imagine how many hundreds more must exist in Linux or even OSX. The top vuln was labeled by the CVS database as "low impact." Bayesian probability fans may point out to you at this moment that "high medium low" is not the best way to assess risks with hundreds of random factors in your very valuable corporate environment. The point is:  think creatively about security. The article linked just above is from Adam Shostack, and is a really fun read.
4. The entire project has a heavy does of humor hidden around every turn, just read the man page for top for example. The contributors include "just a sampler" of "hidden tricks" including the Big Bird trick and the Bouncing Window trick. I have to say that I didn't know about these. Here's another example from the project website's FAQ: "Why does ps get signal 17? No ps release has ever had this problem. Most likely your system has been broken into. You might want to install a more recent version of the OS. If you'd rather take your chances, simply upgrade procps."

An example of these attacks, would you have thought of this?

This is the explanation of how top works, from the man page:

"... If the $HOME and $XDG_CONFIG_HOME variables are not present, top will try to write the personal configuration file in the current directory, subject to permissions."

There was a bug found in the way that the program looks for the config file if it is not there.  An unprivileged attacker might even have a way of removing the constant, first!  These situations would mean that you can kill processes, renice (adjust the relative priority) processes, reorder jobs, and more;  you could do anything top can do.   Some of the other attacks let the attacker block admins from stopping processes, hide malicious processes altogether, and even escape containers!

Today I wanted to just use this as a challenge to creativity, research, and continual learning, but if you are interested in POC here are the links to the attacks:

https://www.qualys.com/2018/05/17/procps-ng-audit-report-advisory.txthttps://gitlab.com/procps-ng/procps/commit/b45c4803dd176f4e3f9d3d47421ddec9bbbe66cd

Humility and Creativity

I asked around online, in which cases would the $HOME constant not be set, and it seems to be a rare occasion, but due to the flaws in comparative scoring like CVE, this rarity may be the only reason the vulnerability was listed as "low impact." There might also be a away to remove the constant, like I mentioned above.  Qualys mentions that is also the case when there is simply an empty $HOME environment variable (which might be a mistake by an admin or even a slightly more common scenario). I pointed out that even in the dash shell (default in Debian if you don't set Bash), it is set. So as @espen mentioned to me, this is defined in the POSIX standard, so it would almost always be set, but in the world of ever-growing IOT devices and systems that are designed primarily to offer a service and never be "logged into," this may be silently becoming more of an issue.  Think too of systems like busybox, which I see getting attacked by bots on my honeypots on a daily basis. Think of all those accounts on Linux, even your Kali box for pentesting, that you have no idea what they are form even more so on systemd: learn what they are for.


In my mind this creates a number or probable attack surfaces to defend against. Assume breach. A saavy attacker--and with Unix I always assume there is a veteran who knows more than me--might know of a way to compromise a system account without the shell set, and "own" your system. This is the main takeaway today. Humility and creativity.  The Blue Team doesn't want to simply ask "what if," we want to ask "what if," but then be able to confidently say "that is not possible," or at least say "that is extremely unlikely," with much greater granularity than assigning a high, medium, or low value on a traffic light to each vulnerability summary.


Take a further look around the top man page if you are interested. The global configuration file also is quite interesting in the same way. It is a perfect example of how hundreds of such bugs must exist in so many programs core to the Linux ecosystem, which literally runs the Internet.

Mitre agrees

From https://attack.mitre.org/wiki/Linux_Techniques there are only a few examples given of similar attacks, actually Mitre ATT&CK provides excellent coverage of Exploitation for Privilege Escalation overall, but just basically mentions software flaws as ways an adversary could gain unauthorized credentials, or evade defenses, without giving many examples.  In a future post, I plan to go through all the linux priv-esc cases from ATT&CK which I was not aware of.  I also have some new packet analysis cases to go through, as always.


A really cool example is given though of "a rare attack" which comes from classic research by Skape at Hick.org. They explained (in 2003!) the general concept of injection into a running process or the proc system. "/proc/[pid]/mem provides access to the memory of the process and can be used to read/write arbitrary data to it. This technique is very rare due to its complexity" according to Mitre.


In closing remember, that however hard to pull off, these attacks exist, and this is where vulnerabilities like the procps set last week come from. The other attacks were even much more serious than top; I just chose top to focus on since almost everyone has used it at one time or another.  It only takes one person to engineer an exploit for these to become simple for anyone to use.





A special thanks goes to Espen Rønnevik (@espen) who gave me a great tip about POSIX via the infosec.exchange. It always pays to have great *nix folks around.