Stupid IPtables Tricks - more tricks w/ recent match

(For a tutorial on the mechanics on how to create state machines with the iptables recent match and recency tables, see my previous blog post.)

1. "Reverse IP tables"
The concept here is to perform the equivalent of port knocking, but instead of opening a firewall pinhole based on inbound connections, the pinhole is opened based on outbound connections.  This technique is especially useful in binding together different apps or services which are running at the same time.

In this example, MySql is setting up database synchronization.  The local master creates an outbound connection to the remote slave, but the remote slave also creates an inbound connection back to the local master.  Normally, the local inbound MySql port is closed.  IPtables will only open the inbound port after the master expresses its implicit trust in the remote slave by creating the outbound connection.

iptables -A OUTPUT -p tcp --dport 3306 -m recent --name dbout --rdest -j ACCEPT
iptables -A INPUT -p tcp --dport 3306 -m recent --name dbout --rcheck -j ACCEPT

The take-away here is that it's possible to make the iptables aware of the state of multiple connections both inbound and outbound.

---

2. "Log Clog"
When logging firewall events, keep in mind Ranum's Laws of Log Analysis:

---

Ranum's first law of Log Analysis:
- Never keep more than you can conceive of possibly looking at
Ranum's second law of Log Analysis:
- The number of times an uninteresting thing happens is an interesting thing
Ranum's third law of Log Analysis:
- Keep everything you possibly can except for where you come into conflict with the First Law
(Posted to firewall-wizards mailing list Oct 01,2004)

---

When using iptables, there isn't a lot of logging sophistication, which means that there's a relatively low limit on the First Law. Thankfully, there's a hack on the Second Law which expands that limit - reduce the logging on what is absolutely uninteresting. I recommend Windows broadcasts. (I actually recommend throwing away Windows, but that's a different blog post.)

Throwing away all instances would be an absolute violation of the Second (and Third) Laws.  Therefore, the key is to throw away most of the instances. Log the first instance, then suppress the rest.

1. iptables -A INPUT -p udp --dport 138 -m recent --name udp-138 --rcheck --seconds 300 -j DROP
2. iptables -A INPUT -p udp --dport 138 -m recent --name udp-138 --set
3. iptables -A INPUT -j LOG --log-prefix “INPUT-drop”
4. (optional) iptables -A INPUT -j DROP

Line b adds the source IP address to the "udp-138" recency table. It enables the filter in line a, which quietly drops the packet. In this example, the suppression lasts for 300 seconds (aka 5 minutes). Line c logs the packet, and line d drops it. An alternate version of line a would use the "hitcount" match instead of the "seconds" match.

iptables -A INPUT -p udp --dport 138 -m recent --name udp-138 --rcheck --hitcount 100 -j DROP

The advantage is that every log event represents a known larger number of events, so logging statistics can can still detect differences in numbers of "uninteresting" events.

---

3. "Auth port"
So you have a service which, for reasons you'd rather not discuss, has no auth, but still needs to be used. With iptables, if you can create a wrapper around the client to do auth, you can use the recent table to make the primary port inherit the auth. It's highly imperfect - auth is tied to the IP address, so IP spoofing FTW - but it's better than leaving the port wide open.

The key to success here is to make the auth service iptables-aware - and it has to have a portion running as root.  :-(  Once the auth is successful, the "old" way is to insert a new iptables rule.  The "new" way is to add the IP address to a recency table which is already referenced by a rule.

1. iptables -A INPUT -d [server] -p tcp --dport 443 -j ALLOW
2. iptables -A INPUT -d [server] -p tcp --dport 80 -m recent --name authok --rcheck -j ALLOW

Quick show of hands, what's missing here?  There is not a rule to add the IP address to the "authok" recency table.  Instead, the auth app can add the IP to the table from userspace.  Each recency table is exposed in the /proc filesystem in /proc/net/xt_recent/[name]. Therefore, once the auth happens, the service can simply # echo [IP] >> /proc/net/xt_recent/[name] and the address will be added as though it were accepted by a rule.

Take-away: recency tables can be read and written from user space, without having to modify the actual IPtables policy, or going anywhere near the iptables binary.

---

4. "Quick & Dirty IDS"
Doing a web search for "iptables -m recent" will return a result on the first page to do IP address blocking, probably something like this page:

1. iptables -A FORWARD -m recent --name noplacelike --rcheck --seconds 60 -j DROP
2. iptables -A FORWARD -i eth0 -d 127.0.0.0/8 -m recent --name noplacelike --set -j DROP

Line a drops the packet quietly if it was blocked (by line b) within the last 60 seconds. Obviously, blocking an IP address forever simply isn't realistic. However, instead of blocking for only 60 seconds, it's possible to use a sliding window.

iptables -A FORWARD -m recent --name noplacelike --update --seconds 60 -j DROP

After identifying the attacker (in line b), using "--update" instead of "--rcheck" will both check and re-set the address in the recency table.  As long as the attacker continues to connect to the target system, if the previous connection is within 60 seconds of the previous connection, the attacker's system will continue to be blacklisted.

---

That's it for the "sane" IPtables tricks with the "recent" match. Next blog entry: some of the crazy potential uses of iptables.

Stupid IPtables Tricks - the short & dense version - port knocking intro

Until I can find somewhere to host my slides, I'll post the "core" of the talk here.

I like using IPtables to solve problems because it adds a bump between the kernel and the application/service.  If you can "solve" a problem using IPtables, you can implement that solution for any app without re-coding or re-configuring.

Of course, best practice is to use good auth & encryption with standard solutions, like SSL/TLS.  Enough about that for now.

Most of the tricks I know are based on the '-m recent' filter, which stores an IP address and timestamp(s) in a named recency table.  Don't confuse this with the state tables like ESTABLISHED or chains like INPUT.  It's a different beast altogether.

Trick 1: Port knocking with IPtables.
In this trick, create a finite state machine using multiple new chains and recency tables.  The way it works is that the INPUT chain checks to see if the IP address is in the recency table for a state, moves to the associated chain, and adds the address to the recency table for the next state.  Here's a 4 state example:

1. icmp ping
2. TCP echo
3. TCP port 8000
4. TCP port 3306

1. icmp ping:

iptables -A INPUT -p icmp --icmp-type 8 -m recent --name seenping --set -j ACCEPT

The filters in this line are "-p icmp --icmp-type 8", icmp echo-request (ping), and can be narrowed down more with -s or -d options.  The final action for this line is "-j ACCEPT", to send the packet through.  The differentiator is "-m recent --name seenping --set", which adds the source IP address of the packet to the recency table called "seenping".

2. tcp echo

1. iptables -N ADD_ECHO
2. iptables -A INPUT -p tcp --dport echo -m recent --name seenping --rcheck -j ADD_ECHO
3. iptables -A ADD_ECHO -m recent --name seenecho --set -j ACCEPT

Line a: Create a new chain called ADD_ECHO to represent the 2nd state in the finite state machine.  It's necessary to use an additional chain for each state, as the "-m recent" syntax only allows for a single recency table reference per command.  Fortunately, recency tables references can be isolated into different chains, as in lines b and c.

Line b: This is the filter command.  If the packet matches the next state, TCP echo, with the source IP address listed in the "seenping" recency table, then move processing to the ADD_ECHO chain.

Line c: the ADD_ECHO chain only does 2 things: add the source IP address to the "seenecho" recency table (priming the filter for the next state), and allowing the packet.  Priming the next state is handled by "-m recent --name seenecho --set".

The action in line c is '-j ACCEPT'.  During my talk, there was a great question: does the action have to be an ACCEPT?  The answer is, no, it can be whatever you want.  There can even be no action listed in this policy.  If iptables reaches the end of the ADD_ECHO chain without finding a terminating action (.e.g ACCEPT, DROP, REJECT), it will automatically pop the stack back to the calling chain.  Additionally, returning to the calling chain can done explicitly via the "-j RETURN" action target.

3. Back-end port: TCP 8000

1. iptables -N ADD_BACKEND
2. iptables -A INPUT -p tcp --dport 8000 -m recent --name seenecho --rcheck --seconds 10 -j ADD_BACKEND
3. iptables -A INPUT -m recent --name seenecho --remove
4. iptables -A ADD_BACKEND -m recent --name seenbackend --set -j ACCEPT

Line a: Create iptables chain called "ADD_BACKEND" to allow state transition to the next state, from 8000 to 3306.

Line b: Filter the incoming packets and trigger the execution of the ADD_BACKEND chain. The filter here is "-p tcp --dport 8000", combined with the source IP address listed in the "seenecho" recency chain. The action is to jump to the "ADD_BACKEND" chain.

The new parameter in line b is the "--seconds" filter. An attacker might be port-scanning the system and accidentally open the state machine up to this state.  Placing a time restriction is a simple method of reducing false positives on the port knock.

Line c is the next line in the INPUT chain after the state transition filter.  If the packet is NOT a match, line c will clear the IP address out of the "seenecho" recency table.  If the client IP host does ANYTHING other than try to connect to TCP port 8000 after sending the TCP echo, the state machine will reject its actions as invalid, and the port knock will fail.  What happens from here isn't specified in this example, so the client could re-send a TCP echo to reset the state and timer for "seenecho".  (Note that the IP is never explicitly purged from "seenping", which is probably bad practice.)  Alternatively, the IP address could be banned for a period of time using another IP tables rule or trick.

Line d is the state progression command, same as step 2.

4. TCP port 3306

iptables -A INPUT -p tcp --dport 3306 -m recent --name seenbackend --rcheck -j ACCEPT

The closing state of the state table allows the client IP address to connect to the mysql port.  Since there's no "next" state, there's no need for another recency table, nor for another chain.

An alternate version of this step would be to create a chain specifically for the "opened" state to aid in overall policy maintenance through separation of rules into discrete chains.

1. iptables -N MYSQL_OPENED
2. iptables -A INPUT -m recent --name seenbackend --rcheck -j MYSQL_OPENED
3. iptables -A MYSQL_OPENED -p tcp --dport 3306 -j ACCEPT

If there is additional  packet-munging you want to perform once the port knocking is done, use this second approach, and replace line c with whatever you want at this stage of the policy.

Note that, for each stage of this example, state progression happened blindly as the first rule in a chain.  If the overall iptables policy is modified to include additional references to that chain, there will be additional methods of prgressing through the port knock.  Whether or not this is a good idea is up to you.

So, there you have it.  That's a 4-stage port knock with false positive rejection done entirely in IPtables, using 3 recency tables for state storage, plus 3 chains (beyond INPUT and ACCEPT) for state progression.

If you have read this far, you have all of the knowledge necessary for the next batch of Stupid IPtables Tricks, which I'll put in my next blog post.

To really make sure you understand the recency tables, check out the excellent example in the iptables tutorial called recent-match.txt.  If I hadn't slogged through it, I wouldn't know how this stuff works.