OpenCloudWeekend

So I discovered at OpenCloudWeekend on 11/5 that I actually know enough about OpenFlow to present for an hour.  I guess I'll have to create another SlideShare now just in case.  :-)

BayThreat talk teaser

I'm presenting a talk at BayThreat on December 10/11 in Mountain View, CA.  Here's the teaser.

IP Spoofing is a favorite InfoSec Cocktail Party topic.  It's a versatile attack against a node in a local network segment, such as coffee-shop public WiFi.  Cleartext connections can be hijacked through injected data, and TLS/SSL can be closed through injected control bits (e.g. RST).

I will present a proof-of-concept tool on Linux to provide TCP packet signing between previously unaffiliated nodes.  Signed TCP will prevent spoofing by providing cryptographic assurance of the sender.

Designed for incremental adoption and common use, the tool establishes an ephemeral relationship between the nodes during the TCP setup.  Authentication per se is not necessary - all that is required to protect the integrity of the session is validation that the remote node doesn't change mid-stream.

Oddly enough, I haven't found anything else that provides this kind of solution.  Hopefully the presentation will be high on questions and low on face-palms.

Exploding IPs, imploding infosec

I'm getting a funny feeling in my belly about the near future of data centers.  It's my impression that we're starting to see a lot more individual nodes than before, driven by the ease of virtualization.  Specifically:

1. Virtualization is leading to larger numbers of individually addressable nodes. Each server is capable of hosting multiple guest images. Therefore, manageable addresses are no longer limited by number of physical devices, but only by the speed at which servers can be deployed. Given that there is strong market push in this direction, deployment is receiving increasing amounts of automation, allowing almost arbitrary MAC (moves, adds, changes) – sometimes completely automated.
2. Virtualization is leading to increased specialization. Guest images are templatized to deploy services quickly: instead of building a server by installing an OS, patches, applications, and localized configuration, do it once, then clone it as needed. The overhead of a separate OS (and >= 1 IP address) per guest is outweighed by the simplicity of decreased administrative and management cost.
3. Increased demand for performance has led to horizontal scaling and clusters, whereby additional speed, capacity, etc. can be added just by adding more servers.  Web app too slow?  Just spin up another compute node!  Conversely, increased demand for cost savings by reduction of power (and cooling) during periods of decreased demand is leading to dynamic power management, turning off computers when not in use. Coupled with virtualization’s ability to add specialized servers dynamically, it is inevitable that fully automated deployment and retirement of servers will become commonplace. 

As a security guy, especially a security management guy, this makes me really twitchy.  It's not just the massive proliferation of IP addresses: it's the increasingly complex task of applying security policy to those addresses.

Historically, we would have handled this with "meaningful" IP addresses: pre-allocate a range of addresses per application.  Anything with that address is "known" to have that purpose.  However, a massive proliferation of dynamic changes makes pre-allocation difficult, as any range by definition has a fixed number of addresses, and it will be harder to predict how many addresses will be necessary - especially if they're being added and retired automatically.

Filesystems could be analogous to IP address allocation.  Windows users are long-accustomed to occasional de-fragmentation of the hard drive for speed, since blocks of large files don't always get stored consecutively.  Many newer filesystems (especially on Linux) are avoiding the defrag problem by embracing it: don't save a file at the first available location, save it way out on the disk to try to allow for file growth.  Returning to IP, current practice has us looking for contiguous blocks of addresses for a given set of related systems, but address fragmentation will occur despite our best efforts.  If address management becomes fully automated, with abstracted tools to provide access to each IP node, then the allocation headache goes away.  Network management will (eventually) embrace it.

For large address pools, e.g. desktops/laptops/users/etc, DHCP has been the way to go.  I believe that DHCP will be the de-facto solution for server-type addresses.  Spin up another front-end server in the cluster, give it a DHCP address, and programmatically add the address to the load balancer via the API.  For back-end, many cluster solutions already support auto-add.  From a network perspective, it will work reliably and quickly - and that's the #1 criterion.  However, from a security perspective, there's very little chance of being able to make a firewall change with the same automation.  Even if it's technically supported, making security changes is "scary", so it won't fly politically.

Firewall policies are already too big to manage.  If contiguity of addresses goes away, the policies become an order of magnitude larger if the policy is centrally managed.

So... what's the solution?  I see a couple of things as possible:

1. Abstraction of policies.  Remove references to IP addresses from the firewall rules, and let the system figure out what the actual elements are.  This method tiptoes past the unease about automated firewall policy changes by hiding the dynamic elements behind a reassuringly static view of the policy.  Is it a believable enough reassurance?
2. Dispersed policies.  VMware has a few tools which apparently allow for firewalling of each VM guest individually (VMsafe API and vShield 2.0).  The implication is that a centralized firewall is less necessary, as each guest has its own firewall.  My concern is that managing 1 firewall per VM may not be any easier than managing a central policy.
3. Deployment solutions like HyTrust, which limit VM guest deployment to particular VM hosts.  IMHO it's a related but different problem, which essentially creates multiple DMZ networks, minus the network security aspect.

In short: I don't know, but it scares me.

Stupid IPtables Tricks - more tricks w/ recent match

(For a tutorial on the mechanics on how to create state machines with the iptables recent match and recency tables, see my previous blog post.)

1. "Reverse IP tables"
The concept here is to perform the equivalent of port knocking, but instead of opening a firewall pinhole based on inbound connections, the pinhole is opened based on outbound connections.  This technique is especially useful in binding together different apps or services which are running at the same time.

In this example, MySql is setting up database synchronization.  The local master creates an outbound connection to the remote slave, but the remote slave also creates an inbound connection back to the local master.  Normally, the local inbound MySql port is closed.  IPtables will only open the inbound port after the master expresses its implicit trust in the remote slave by creating the outbound connection.

iptables -A OUTPUT -p tcp --dport 3306 -m recent --name dbout --rdest -j ACCEPT
iptables -A INPUT -p tcp --dport 3306 -m recent --name dbout --rcheck -j ACCEPT

The take-away here is that it's possible to make the iptables aware of the state of multiple connections both inbound and outbound.

---

2. "Log Clog"
When logging firewall events, keep in mind Ranum's Laws of Log Analysis:

---

Ranum's first law of Log Analysis:
- Never keep more than you can conceive of possibly looking at
Ranum's second law of Log Analysis:
- The number of times an uninteresting thing happens is an interesting thing
Ranum's third law of Log Analysis:
- Keep everything you possibly can except for where you come into conflict with the First Law
(Posted to firewall-wizards mailing list Oct 01,2004)

---

When using iptables, there isn't a lot of logging sophistication, which means that there's a relatively low limit on the First Law. Thankfully, there's a hack on the Second Law which expands that limit - reduce the logging on what is absolutely uninteresting. I recommend Windows broadcasts. (I actually recommend throwing away Windows, but that's a different blog post.)

Throwing away all instances would be an absolute violation of the Second (and Third) Laws.  Therefore, the key is to throw away most of the instances. Log the first instance, then suppress the rest.

1. iptables -A INPUT -p udp --dport 138 -m recent --name udp-138 --rcheck --seconds 300 -j DROP
2. iptables -A INPUT -p udp --dport 138 -m recent --name udp-138 --set
3. iptables -A INPUT -j LOG --log-prefix “INPUT-drop”
4. (optional) iptables -A INPUT -j DROP

Line b adds the source IP address to the "udp-138" recency table. It enables the filter in line a, which quietly drops the packet. In this example, the suppression lasts for 300 seconds (aka 5 minutes). Line c logs the packet, and line d drops it. An alternate version of line a would use the "hitcount" match instead of the "seconds" match.

iptables -A INPUT -p udp --dport 138 -m recent --name udp-138 --rcheck --hitcount 100 -j DROP

The advantage is that every log event represents a known larger number of events, so logging statistics can can still detect differences in numbers of "uninteresting" events.

---

3. "Auth port"
So you have a service which, for reasons you'd rather not discuss, has no auth, but still needs to be used. With iptables, if you can create a wrapper around the client to do auth, you can use the recent table to make the primary port inherit the auth. It's highly imperfect - auth is tied to the IP address, so IP spoofing FTW - but it's better than leaving the port wide open.

The key to success here is to make the auth service iptables-aware - and it has to have a portion running as root.  :-(  Once the auth is successful, the "old" way is to insert a new iptables rule.  The "new" way is to add the IP address to a recency table which is already referenced by a rule.

1. iptables -A INPUT -d [server] -p tcp --dport 443 -j ALLOW
2. iptables -A INPUT -d [server] -p tcp --dport 80 -m recent --name authok --rcheck -j ALLOW

Quick show of hands, what's missing here?  There is not a rule to add the IP address to the "authok" recency table.  Instead, the auth app can add the IP to the table from userspace.  Each recency table is exposed in the /proc filesystem in /proc/net/xt_recent/[name]. Therefore, once the auth happens, the service can simply # echo [IP] >> /proc/net/xt_recent/[name] and the address will be added as though it were accepted by a rule.

Take-away: recency tables can be read and written from user space, without having to modify the actual IPtables policy, or going anywhere near the iptables binary.

---

4. "Quick & Dirty IDS"
Doing a web search for "iptables -m recent" will return a result on the first page to do IP address blocking, probably something like this page:

1. iptables -A FORWARD -m recent --name noplacelike --rcheck --seconds 60 -j DROP
2. iptables -A FORWARD -i eth0 -d 127.0.0.0/8 -m recent --name noplacelike --set -j DROP

Line a drops the packet quietly if it was blocked (by line b) within the last 60 seconds. Obviously, blocking an IP address forever simply isn't realistic. However, instead of blocking for only 60 seconds, it's possible to use a sliding window.

iptables -A FORWARD -m recent --name noplacelike --update --seconds 60 -j DROP

After identifying the attacker (in line b), using "--update" instead of "--rcheck" will both check and re-set the address in the recency table.  As long as the attacker continues to connect to the target system, if the previous connection is within 60 seconds of the previous connection, the attacker's system will continue to be blacklisted.

---

That's it for the "sane" IPtables tricks with the "recent" match. Next blog entry: some of the crazy potential uses of iptables.

Stupid IPtables Tricks - the short & dense version - port knocking intro

Until I can find somewhere to host my slides, I'll post the "core" of the talk here.

I like using IPtables to solve problems because it adds a bump between the kernel and the application/service.  If you can "solve" a problem using IPtables, you can implement that solution for any app without re-coding or re-configuring.

Of course, best practice is to use good auth & encryption with standard solutions, like SSL/TLS.  Enough about that for now.

Most of the tricks I know are based on the '-m recent' filter, which stores an IP address and timestamp(s) in a named recency table.  Don't confuse this with the state tables like ESTABLISHED or chains like INPUT.  It's a different beast altogether.

Trick 1: Port knocking with IPtables.
In this trick, create a finite state machine using multiple new chains and recency tables.  The way it works is that the INPUT chain checks to see if the IP address is in the recency table for a state, moves to the associated chain, and adds the address to the recency table for the next state.  Here's a 4 state example:

1. icmp ping
2. TCP echo
3. TCP port 8000
4. TCP port 3306

1. icmp ping:

iptables -A INPUT -p icmp --icmp-type 8 -m recent --name seenping --set -j ACCEPT

The filters in this line are "-p icmp --icmp-type 8", icmp echo-request (ping), and can be narrowed down more with -s or -d options.  The final action for this line is "-j ACCEPT", to send the packet through.  The differentiator is "-m recent --name seenping --set", which adds the source IP address of the packet to the recency table called "seenping".

2. tcp echo

1. iptables -N ADD_ECHO
2. iptables -A INPUT -p tcp --dport echo -m recent --name seenping --rcheck -j ADD_ECHO
3. iptables -A ADD_ECHO -m recent --name seenecho --set -j ACCEPT

Line a: Create a new chain called ADD_ECHO to represent the 2nd state in the finite state machine.  It's necessary to use an additional chain for each state, as the "-m recent" syntax only allows for a single recency table reference per command.  Fortunately, recency tables references can be isolated into different chains, as in lines b and c.

Line b: This is the filter command.  If the packet matches the next state, TCP echo, with the source IP address listed in the "seenping" recency table, then move processing to the ADD_ECHO chain.

Line c: the ADD_ECHO chain only does 2 things: add the source IP address to the "seenecho" recency table (priming the filter for the next state), and allowing the packet.  Priming the next state is handled by "-m recent --name seenecho --set".

The action in line c is '-j ACCEPT'.  During my talk, there was a great question: does the action have to be an ACCEPT?  The answer is, no, it can be whatever you want.  There can even be no action listed in this policy.  If iptables reaches the end of the ADD_ECHO chain without finding a terminating action (.e.g ACCEPT, DROP, REJECT), it will automatically pop the stack back to the calling chain.  Additionally, returning to the calling chain can done explicitly via the "-j RETURN" action target.

3. Back-end port: TCP 8000

1. iptables -N ADD_BACKEND
2. iptables -A INPUT -p tcp --dport 8000 -m recent --name seenecho --rcheck --seconds 10 -j ADD_BACKEND
3. iptables -A INPUT -m recent --name seenecho --remove
4. iptables -A ADD_BACKEND -m recent --name seenbackend --set -j ACCEPT

Line a: Create iptables chain called "ADD_BACKEND" to allow state transition to the next state, from 8000 to 3306.

Line b: Filter the incoming packets and trigger the execution of the ADD_BACKEND chain. The filter here is "-p tcp --dport 8000", combined with the source IP address listed in the "seenecho" recency chain. The action is to jump to the "ADD_BACKEND" chain.

The new parameter in line b is the "--seconds" filter. An attacker might be port-scanning the system and accidentally open the state machine up to this state.  Placing a time restriction is a simple method of reducing false positives on the port knock.

Line c is the next line in the INPUT chain after the state transition filter.  If the packet is NOT a match, line c will clear the IP address out of the "seenecho" recency table.  If the client IP host does ANYTHING other than try to connect to TCP port 8000 after sending the TCP echo, the state machine will reject its actions as invalid, and the port knock will fail.  What happens from here isn't specified in this example, so the client could re-send a TCP echo to reset the state and timer for "seenecho".  (Note that the IP is never explicitly purged from "seenping", which is probably bad practice.)  Alternatively, the IP address could be banned for a period of time using another IP tables rule or trick.

Line d is the state progression command, same as step 2.

4. TCP port 3306

iptables -A INPUT -p tcp --dport 3306 -m recent --name seenbackend --rcheck -j ACCEPT

The closing state of the state table allows the client IP address to connect to the mysql port.  Since there's no "next" state, there's no need for another recency table, nor for another chain.

An alternate version of this step would be to create a chain specifically for the "opened" state to aid in overall policy maintenance through separation of rules into discrete chains.

1. iptables -N MYSQL_OPENED
2. iptables -A INPUT -m recent --name seenbackend --rcheck -j MYSQL_OPENED
3. iptables -A MYSQL_OPENED -p tcp --dport 3306 -j ACCEPT

If there is additional  packet-munging you want to perform once the port knocking is done, use this second approach, and replace line c with whatever you want at this stage of the policy.

Note that, for each stage of this example, state progression happened blindly as the first rule in a chain.  If the overall iptables policy is modified to include additional references to that chain, there will be additional methods of prgressing through the port knock.  Whether or not this is a good idea is up to you.

So, there you have it.  That's a 4-stage port knock with false positive rejection done entirely in IPtables, using 3 recency tables for state storage, plus 3 chains (beyond INPUT and ACCEPT) for state progression.

If you have read this far, you have all of the knowledge necessary for the next batch of Stupid IPtables Tricks, which I'll put in my next blog post.

To really make sure you understand the recency tables, check out the excellent example in the iptables tutorial called recent-match.txt.  If I hadn't slogged through it, I wouldn't know how this stuff works.

Stupid IPtables tricks - teaser

On 7/28/2010 at 6:00pm at BSides Las Vegas, I'll be giving a talk called "Stupid IPtables Tricks". Here's the quick overview.

Update / Clarification: the techniques I'll discuss can't all be done "natively" in iptables.  Some of them leverage other tools, like Dans Guardian for URL filtering.  IPtables itself isn't good at L7, but it can do lovely manipulation below that.

IPtables is the standard Linux firewall, happily implementing policy at L3/L4. As such, it looks like an older-generation firewall, stuck firmly in the Stateful Packet Filter realm of technology. It knows about UDP and TCP, but has no clue about HTTP, much less your userid on FaceTube.

However, iptables has one thing going for it that a lot of turn-key commercial products don't - it conforms to the Linux idea of being programmable and extensible. Beyond the basic TCP 3-way handshake, have-I-seen-this-circuit statefulness, it allows for nearly arbitrary reactions to and actions on packets. Furthermore, it's got some hooks that allow for blurring the lines between the network layer and the application layer.

This talk will focus on creating IPtables policies with weird abilities that aren't always found in other firewall implementations. Examples include:

• Using external auth to tie a user to an IP, then implementing different policies based on that IP
• URL filtering, with different policies per-group
• SNAT with different IP address based on user group
• Outbound routing through high-bandwidth, low-latency, etc.
• Slowing down network access for low-priority users
• De-clogging logs
• Why must Windows discovery broadcasts be so chatty?
• Poor-man's blocking IDS
• Ignore inbound IP for a while if it seems naughty
• Implementing port knocking on the gateway
• Anti-spam IP blacklisting and graylisting
• Tying iptables policies to the underlying applications on the same host
• Allow outbound ssh only if it's really from the ssh process
• Allow inbound connections to arbitrary / dynamic ports for specific applications
• Whoever decided dynamic ports were a good idea had no bloody experience with implementing security!

The talk will also discuss some classic limitations of L3/L4 policy implementation, especially session hijacking and "joe jobs" through IP spoofing.

This talk will not focus on ways to bypass iptables. That's just about every other talk this week in Vegas.

Why am I giving this talk?  My job is to retrofit security onto NMS appliances for a large networking vendor.  My personal goal is to keep 80% of you from cracking my department's product within the first week of its release, and to convince my team to make my job unnecessary by writing code that's secure.

Existing solutions to SSL untrusted CA

Current solutions to why you can't trust SSL:

Certlock	Christoper Soghoian and Sid Stamm have written a academic paper about this problem and imply that they will release a Firefox add-on which performs a TOFU (Trust on First Use) cache of the cert, with additional checking for country of certification.
Conspiracy	Kai Engert released this Firefox addon to display country of certification.
DoubleCheck	Mansoor Alicherry and Angelos D. Keromytis have a Firefox addon which performs a second check of the cert via TOR. Academic paper here.
Perspectives	Another tool for external cert verification using "Notary servers", by Dan Wendlandt et al. Probably the closest implementation to what I'm proposing. Academic paper here.
Certificate Patrol	Firefox addon to display changes to SSL certs, plus display info on certs for first-visit sites.