Saturday, February 25, 2017

Building tunnels for early adoption of new tech

Imagine there's a cool new tech that potentially solves a real problem when using the Internet. In this case, do you ever find yourself dealing with wifi so bad that you pull out your phone and tether instead1? Coded TCP (aka TCP/NC or CTCP)2 can potentially solve it. However, the solution requires an end-to-end change: it would have to be adopted by every website you visit. How do we deal with this?

The problem turns out to be similar to IPv6, which is incompatible with the IPv4 that's historically been used. Running native IPv6 required end-to-end support, including your WiFi router, your Internet router (cable modem, DSL, etc.), your Internet provider, and the website. If a website only supported IPv6, there was no way for an IPv4 client to reach it. The early adoption solutions generally used tunneling like Teredo3. In Teredo, the client would create an IPv6 packet, then wrap it in IPv4 and deliver it to a "middlebox", which would extract the IPv6 payload and deliver it natively to the website.

Network coding could similarly be deployed using tunneling and a middlebox. The solution looks a little bit different, but the end result is the same: the client machine can use the advantages of the new protocol on the Internet in a way that's transparent to the local network and to the website.

Conceptually, I imagine that CTCP would be encapsulated using UDP between the client and the middlebox, since the primary advantage is reducing retransmissions on the client's local WiFi network. The middlebox would then L4 proxy the data stream to the end website, extracting the data transparently from the client CTCP and re-encoding it to the server's legacy TCP. In this sense, it's the opposite of Teredo: CTCP's value is the client-middlebox connection, and Teredo's value was the middlebox-server connection.

There are of course some difficulties in this scenario.
  • Encapsulated traffic is difficult for firewalls to protect, as the real addressing info is buried inside a wrapper. Additionally, there may be L7 security concerns from HSTS, HTTP/2, etc., stemming from the recoding at L4 instead of L3.
  • For graceful deprecation, what happens if CTCP becomes more common and some sites don't need a middlebox connection? How does the client recognize the difference and decide which connection to use?
  • Google's QUIC protocol replaces TCP's congestion control and retransmissions with its own mechanism, which they claim is similarly designed to work with heavy packet loss. That would potentially remove the majority use case for CTCP
  • CTCP is a trade secret, so there are financial and legal barriers in the way of widespread adoption.

Still, this would be a fun thing to prototype.



1. If you use your cell phone to tether when the WiFi is bad, don't tether via WiFi, as you're just adding more traffic to the already poor wireless spectrum, and even if it works for you, it makes the problem worse for everyone around you. If you have to tether, do it via USB (RNDIS on the PC), as it doesn't create more wireless traffic, and it lets your phone charge while you use it.

2. Coded TCP uses linear network encoding to spread payload data across multiple packets, so a lost packet can be reconstructed on the receiving end.

3. Teredo introduced some serious security problems, as it allowed IPv6 to bypass the firewall on the client's network, so it became best practice to turn it off. Ironically, many networks today which natively support IPv6 still don't apply firewall policy to the protocol, providing a similar level of insecurity.


Friday, November 6, 2015

When will we see NFV as a Service?

NFVaaS - while it sounds like a joke about mixing acronyms between cloud ('x' as a Service) and SDN, NFV is already a serious mix of the two technologies - and I think there's a lot of promise in taking this joke a lot farther.

SDN provides centrally controlled packet steering - a planned path, through specified forwarding engines, between endpoints. Cloud provides orchestrated 'service' deployment, automating the setup of VMs/containers/whatever. NFV mixes the two concepts, to use cloud orchestration to deploy middleboxes into the SDN packet path, so it's possible to steer connections through firewalls, NAT, IDS, DLP, load balancers, caching engines, etc. The key point is that the services are consumed not by users, but by packets/sessions, thus making NFV both into "cloud for networking" and "networking for cloud". For the rest of this post, I'm going to simplify NFV into the concept of "orchestrating network middleboxes".

So how is NFVaaS different from just NFV? The XaaS term implies wrapping something up in a turn-key way so that it's easy to consume. DBaaS gives me an interface to a running database onto which I can apply schema and do some SCRUD, instead of setting up a server and installing software. IaaS gives me virtual bare metal. NFVaaS therefore goes beyond applying middleboxes as part of delivering or munging sessions for some other application - the whole point of NFVaaS is that I want middleboxes through which to push my own packets. If I have a network that needs a firewall, what I have is a problem that could be solved by NFVaaS, deploying a firewall through which I can push my packets.

To put this another way, if my network use is primarily as a client (rather than running a web site, etc.) then the primary service I consume is "access". To add value to that service, I want to add middleboxes, but I want it "cloud-style": usable in minutes, with costs relating to my use rather than acquisition and maintenance. NFV provides the ability for someone to deliver that service to me.

Some technically-minded readers may be asking, "how do you insert something into the packet path?" There are a few different ways, but what I personally prefer is using a VPN. Right now I'm renting a VPS which I use as a VPN concentrator and cloud storage server, with the intention of adding IP blocking based on an open theat intelligence feed. At some point I might add Squid as a transparent proxy running ClamAV. There's a lot I can do to create my own "clean pipe" service, and the value to me greatly exceeds the VPS cost. However, it took me a while to set up what I have, and will take a lot more time to add (and tune) the additional capabilities. If I could get all of this as a service for a reasonable cost, I'd recommend it to all of my non-technical friends and family.

NFV gets a lot of attention in big networks like Service Providers, Data Centers, and Large Enterprise - in short, anyone running their own private clouds - but just because it runs in a cloud doesn't mean that it doesn't have value to everyone else. I'm definitely looking forward to renting time on a IDS.

Thursday, April 2, 2015

Who will own the virtual network?

There are two network virtualization products being pushed by heavyweight vendors. In this corner: VMware NSX. In that corner: Cisco ACI.

The debate is ongoing how to add agility to networking. The overlay model, like NSX, says that the network can be simple as long as the apps are smart. The controller model, like ACI, says that the network should be aware of the apps.

On the overlay side, the strength is in its deployment simplicity: end nodes communicate with each other, but the network itself (the "underlay") doesn't require any adjustment. Its weakness is the flip side: there's no way to tune the underlay, so there's no way around bottlenecks and other potential artifacts of the architecture.

On the controller side, the strength is the potential of the network to adjust to support whatever flow patterns the app requires. There are two weaknesses: first, if the app doesn't integrate with the controller API, then the controller has to interpret the data it's carrying and make a best guess on optimal configuration; second, the "smart" abilities generally require new hardware, which greatly slows down the rate of adoption.

In the short term (1.5-3 years), it's likely that NSX will be deployed much more than ACI. NSX doesn't "technically" require the server team to involve the network team in the roll-out, and the agility that's grown from virtualization into modern devops will lead both to an increased tolerance for risk and a demand for better network delivery. (As a network guy, I'm used to hearing the network always getting blamed for any problem.) The fertile ground will be large applications hosted either classically or in private cloud, especially where the app is multi-tiered (with lots of local east-west traffic), but may grow as NSX support increases across data centers.

The question is whether ACI will take hold, and where. The fertile ground will likely be single-owner data centers with monetized services, where ACI will be seen as an investment to increase capacity, and API integration can be pushed top-down into the hosted systems. However, the Internet Giants (Google, Facebook, etc.) have demonstrated that they're willing to create their own network technology rather than purchase big-ticket vendor products, and analysts and pundits are declaring that everyone else will move to cloud hosting. Even though that prediction is hyperbole, it does put downward pressure on the ACI target markets.

Keep in mind that 3 years ago, the "future of networks" was OpenFlow, so any future predictions that aren't wildly inaccurate will likely be due to blind luck. :-)

What do you think? That's what the comments are for down below.

Monday, August 5, 2013

A sonnet about the TSA

This was my entry for the Liquid Matrix Podcast contest for a pass to BSides Las Vegas. I didn't win, but wanted to share it with you.

The Iron Wisdom of the TSA
inspired by a Vendor DERP from Liquid Matrix episode 0x2C

The airline tells us that their onboard staff
are there for safety, not for in-flight ease.
And yet SouthWest will brief us with a laugh,
and once airborne, serve beverages to please.

But safety on the ground from TSA:
before you soar, they make your spirits sink.
Remove your shoes, your liquids throw away!
No wonder on the plane you need a drink.

Your courtesy is answered with their scorn,
suspicion is delivered without tact.
So cover up, the scanners churn out porn!
There's theater security to act.

The mission of the TSA thus laid:
"Security" keeps passengers afraid.

Monday, June 3, 2013

Want innovation? Reduce the cost of education!

Today's post is inspired by the discussion over US visas for foreign workers, and the argument from high-tech companies that there aren't enough new US grads in STEM (Science, Tech, Engr, Math) to fill the open positions. Specifically, I'm reacting to the "data" in this article, which says what we've all suspected for the last 15 years: US companies can't find qualified people who will are willing to work for what the positions pay.

My gut reaction (no data to back this up other than personal experience) is that new US grads have a disadvantage that a lot of foreign grads historically haven't: student loan debt. If a new US grad leaves school with $50k-$150 in student loans, that's a strong disincentive to take risks. Instead, a monthly loan bill will demand that students quickly find a reliable source of income. Additionally, cost of living in traditional areas where STEM companies congregate is high. In my case, it was the Silicon Valley in 1996, when I was "lucky" to pay $1200/mo for a 2-bedroom apartment.

The "mythical" Silicon Valley startup mentality had the motto, "work any 80 hours a week you want," and didn't care what your education level was as long as you got the job done. Now, startups are less work-intensive, but early stage positions still offer very little income. If these companies want to attract the "best and brightest", one way to do it is to increase applicants' openness to risk.

A relatively recent idea has been the "hacker hostel" model, in which startup incubators come with not only office space, but sleep space. While that reduces the monthly expenditure for a new hire, it has its own disadvantages, including being appealing primarily to early-stage companies (read: the co-founders are the ones sleeping on the couch).

A more long-reaching solution would be to drive the cost of education back down. Forbes reported in 2012 that education has risen 500% since 1985, while the consumer price index was only 115%. There's a pretty picture here (note: data from same source). While there was some re-structuring a couple of years ago, in which some colleges switched from mixed financial aid to grant-only, that story has fallen out of the media, and costs haven't really been affected. Furthermore, the global economic downturn has reduced spending on eduction (at least in the US), driving tuition & fees higher, even in state schools.

A reduction in the cost of education is a likely driver to innovation specifically because it will reduce the p2p driver that killed so many startups. In business, p2p for a VC-backed firm was "path to profitability", insisting on rapid repayment of investment by bringing products to market early. In education, I'm using the term as "path to payment", repayment of student loans. If we want graduates to work for less money, then let's make it possible for them to do so.

Tuesday, January 22, 2013

Urgency and activity

I've noticed that I tend to get work done best when I'm nearly out of time to do it. Presumably, I'm not the only one who works like this, but it definitely causes... let's call it "friction" with the people I work with who seem to work at a constant pace.

I wonder sometimes if my life would be different if I could maintain that constant pace. Given my irregular output, I see myself as a miracle worker who would thrive in an environment of being called in to get stuff done in an emergency. However, my co-workers tend to see me as unreliable, which has occasionally been career-limiting.

A few years ago, the team I was part of was introduced to Scrum, an Agile methodology. At the time, we were told in training that one of the goals was to even out the crisis humps, replacing the "mad rush" at the deadline with a series of smaller deadlines, so there'd be no all-nighters to get things done. Looking back now, I wonder if it was a compromise to try to get "bursty" workers to increase their output, or at least make their output more predictable to schedule-keepers.

Driving home today, I wondered about what a crazy frenetic work environment would be like, in which goals for the day were handed out in the morning. After a while I know it would be maddening, and that very few people would survive working there. However, if they could put the right team together, imagine the output!

Some of this thought is driven by just having read The Phoenix Project, a novel about DevOps. It's a great read, and if you're an Amazon Prime member with a Kindle, you can "borrow" it for free. That's what I did before buying it. :-)

What it got me thinking about is how to overcome "impossible" challenges. In the book, one such challenge is going from a deploy every 9 weeks to a deploy of 10 times per day. It's called "impossible" by some characters, but, since the novel is a polemic about DevOps, of course they figure it out quickly. The key is to challenge your assumptions about what's possible. At my current job, we recently had a semi-successful meeting where some team members argued that what was being asked for was "against the laws of physics". That to me is a sign that you need to approach the problem differently, set aside the "impossible" label, and look at what obstacles are in the way. If you can re-route around those obstacles, or wave a magic wand to streamline them (by several orders of magnitude), then the solution becomes possible, and the new challenge is driven by the excitement of doing what you knew yesterday was impossible.

I guess the challenge for me now is to figure out how to find something impossible every day... make each day a mini-sprint, and see what happens to my output.

Wednesday, November 28, 2012

Blogging: inconsistent quality or frequency?

One of my tasks at my company is to create blog entries, both on internal and external blogs. (See some of my other posts for links to what I've written.) I've noticed an interesting difference between writing for myself and writing for my company:

When I write for myself, my output is determined by when I get an idea that I want to share. Results are created by inspiration.

When I write for my company, my output is determined by our schedule. If I don't have an idea, I get one from our marcom organization, and I work at it until it makes sense, even if I personally don't see a lot of value in the content. Conversely, if I have an idea which is burning inside my head, I have to fight to get it included in our publishing schedule, with the occasional exception as a special post.

There are good reasons for scheduled output: when trying to generate a follower base, predictable frequency makes it easier for readers to know when something new will go up. It also means that, for infrequent readers, there will almost always be something new to read.

However, it's hard to train my brain to get inspired on demand. A lot of us know this feeling, and there's a fabulous write-up about it from The Oatmeal. In short, content on a schedule leads to inconsistency of output quality.

It's tempting to put together a bogus formula which asserts that quality ideas are a non-linear, which can lead either to inconsistent output frequency or to inconsistent output quality. I'll leave that as an exercise to the reader.

However, I'll also point out that there's a disconnect in perceived quality between the information producer vs the information consumer. Neil Gaiman expressed this eloquently in his Make Good Art speech. This also appears to be a common experience: I write something that I think is awful, and other people love it, and vice-versa.

Given this disconnect, I feel comfortable for now working with a content production frequency schedule. My internal discomfort at writing can spur me to produce output that's much better than I'd anticipated. It's also useful to get ideas for technical blog posts from people who have a different understanding of technology that I do, because it triggers the instinct to correct the misinformation - which leads to output on a schedule.

That being said, I still don't like the process, but I'm okay with the results.

P.S. Some readers of this post will notice that the title of this post is a question. When I started writing, I had a completely different title, but I changed it part-way through, with the implication that it's an either-or. Now, I look at it and realize that I'm compliant with Betteridge's Law, and the answer to the question is in fact "No".

Monday, August 13, 2012

IPv4 "Offshore Account" Predictions

IP addresses are a necessary resource for using the Internet, especially for service providers like web hosting companies.  Given that unused addresses are becoming scarce, I predict that we'll start seeing businesses invest in Latin America and in Africa specifically to acquire IP addresses there.


The setup

On July 31 this year, ARIN adopted a new policy to allow inter-region transfers of IP address allocations.  That may be news to some people, that IPv4 addresses aren't like normal property that can be bought and sold at will.  According to ARIN CEO John Curran, this is because "how we use [...] IP addresses affects all networks". Interestingly, as of August 8, the only other regional internet registrar (RIR) with a compatible policy is APNIC, which reached address exhaustion on April 19, 2011.

A quick note on address exhaustion: this means they have less than 1 /8 block left, not that they are completely out of addresses.  A /8 potentially contains 65536 /24 blocks - although that number will be smaller if an organization can convince the RIR to allocate a larger block than a /24.  APNIC currently has 0.9183 /8 blocks, which roughly translates to about 60000 /24 blocks.

Why is it important that APNIC and ARIN have compatible regional transfer policies?  That means that it's possible to move IP address allocations between them.  Right now, the obvious motive is to ease the IP crunch in APNIC from ARIN, which has 3.4561 /8 blocks.  However, on the FAQ page for this new policy, ARIN states that "There are inter-RIR transfer policy proposals in several other regions at the moment".  Assuming that other RIRs have similar mentality to ARIN, it's likely that ARIN is establishing a policy now to allow for inbound IP transfers once ARIN reaches exhaustion in early 2013.


The sources

Current projections for remaining IP blocks are available in this nifty gadget courtesy of INTEC, although its numbers differ from the data provided by Internet guru Geoff Huston.


The big question is what ARIN will consider a necessary part of the policy to be "compatible".  ARIN's transfer policy imposes a 12-month before-and-after waiting period on transfer sources within the ARIN region: the source must have had the IP addresses for over 12 months before the transfer, and can't receive any more addresses from ARIN for another 12 months after the transfer.  However, the policy also states that "Source entities outside of the ARIN region must meet any requirements defined by the RIR where the source entity holds the registration."


The possibilities

1. Direct IP exporters

If another region has a much less restrictive policy, there's the possibility of a new business model for a company in that region to apply for IP address blocks, then sell them.  It's a centuries-old practice for a developing nation to sell its raw resources to overseas buyers.

2. Foreign shell companies

If other regions are planning to adopt strict restrictions like ARIN on source organizations for IP transfer, the logical step is for global organizations to found shell companies right now in the LACNIC and AfriNIC regions.  Then, once the hold-down timers expire - e.g. the shell company has had its IP addresses for 12 months or whatever the local policy is - then the parent company would either initiate a transfer, or just acquire the shell under ARIN's mergers and acquisitions policy.

3. Foreign providers

If an address transfer isn't feasible, the next likely business model would be for "boutique" hosting companies to spring up in the LACNIC and AfriNIC areas.  Bandwidth prices there are high but falling, leading to the potential for hosting companies there to rent out destination IP addresses, or, more importantly, blocks of IP addresses.  

To reduce bandwidth usage (and reduce costs), there are techniques available both at the IP layer and at higher layers.

Within IP address advertising, there's no current technical restriction to prevent geographic relocation via BGP advertising, in effect becoming a semi-legitimate use of IP Hijacking.  Even anti-hijacking technologies like RPKI could be co-opted, either via disabling (like the current status of the anti-spam SPF technology) or by simple delegation within the PKI.

Within higher layer protocols like HTTP, techniques like redirects are a time-tested method of providing a fixed landing point with dynamically located content.  The hosted site would contain just enough information to pull the real content from a CDN or other external high-bandwidth low-cost source.


The timeframe

Given the 12-month limitation from ARIN for transfers, plus the projection of ARIN address exhaustion only 6-12 months away, look for large organizations to start this kind of "IP address offshoring" very soon.  In fact, given that the newly adopted ARIN policy was first proposed in February 2011, it's likely that some global organizations have already started this process.

Just for grins, check out this map of Chinese investment in Africa - then remember that China is in APNIC, and has long been short on IP addresses.

Monday, June 4, 2012

Flame - something only a government could build?

The NY Times (June 3 2012) published an article saying that Kaspersky Labs has declared Flame to be something only a government could build.  I disagree.

Look at the purported history of Stuxnet.  The most comprehensive story so far comes from David Sanger - again in the NY Times - saying that it was a project started under Bush-43, continued under Obama-44, co-developed with Israel's Unit 8200.  The larger story implies that the development was broken down into modules coded separately, with the programmers potentially unaware of the nature of the project.  This sounds a lot like the development process of the virus in Neal Stephenson's 1992 novel Snow Crash, in which (ironically) the US Government is the contractor creating the virus for the villain.  The Government was chosen as the development partner because they are the only organization paranoid enough not to trust their programmers with any "big picture" view.

Flame is a bloated beast, 20MB in size, with business logic in Lua calling compiled C++ modules.  While there's nothing unusual about that structure - even large web sites are using hybrid combinations of languages like Scala and Java - it does add credence to the idea that this is code that was not written by one person, nor potentially even one organization.  Flame could easily have been developed according to "Snow Crash" paranoid secret separation of information principles.

We've heard statements before about some things only being possible by government-level operations.  My favorite example was when members of the l0pht testified before Congress about potentially taking down the Internet in 30 minutes.  Government reaction was astonishment - they'd assumed that no private nor commercial organization had such capabilities, and had to go back and re-write their threat models.

In short: while Flame is a sophisticated toolkit for malware, it's not something that only a government could build.  However, the way that it's put together is the way that a government would build it.

In researching this post, I finally found some information that hadn't been exaggerated bouncing around the media echo chamber in a BBC News article that actually quotes someone at Kaspersky Labs. Vitaly Kamluk, "chief malware expert" per the BBC, is quoted thusly:
"Currently there are three known classes of players who develop malware and spyware: hacktivists, cybercriminals and nation states. Flame is not designed to steal money from bank accounts. It is also different from rather simple hack tools and malware used by the hacktivists. So by excluding cybercriminals and hacktivists, we come to conclusion that it most likely belongs to the third group."
 My reading of this statement is that Flame seems consistent, based on its actions, with something that isn't consistent with non-government malware.  However, that headline won't sell news stories.

Monday, April 30, 2012

What the Hyperspeed "Bullet Time" guys might be thinking

Background: "Bullet Time" article in New Scientist

When this article hit Twitter, lots of us thought that it was waving-of-hands security, no more worthy of contemplation than the Evil Bit joke RFC.  However, like the Evil Bit, there might be some deeper truth to what they're proposing.

The journal article which contains the proposal focuses on creating low-bandwidth high-speed links within MPLS networks.  In that sense, it's an example of RFC 1925 truth 11, re-introducing an old idea to solve a new problem.  Prioritization of network control traffic was built into the TOS flags in IP, which were later replaced with DSCP.  It's not a bad idea, just one which has never found its "killer app", so has not had widespread adoption.  (Whether QoS has widespread adoption is a discussion left as an exercise to the reader.)

The basic concept of "Hyperspeed", the name given to the proposal, is that network control type traffic be pre-provisioned into existing infrastructure.  The New Scientist article focuses on a security use case in which cloud-based traffic examination (I'm putting words into their mouth here, they gracefully avoid using the c-word) detects potentially malevolent traffic and pre-warns the recipient network of a potential attack.  However, I suspect that the NS article suffers from the same media hype and distortion which surrounds a lot of technical discussion.

Taken on its own, there is a reasonable argument for this type of approach.  Adding additional layers of network defense can interfere with traffic.  Whether we security folks like it or not, speed trumps security when providing services.  Therefore, if it is economically feasible to inject high-speed low-latency analysis at a centralized point upstream of the target network, it would be useful to have a coordination protocol which automatically triggers deeper levels of analysis.

Anyone who has deployed a blocking IDS will immediately recognize that tuning the system is tricky.  Thresholds must be localized for the target network to avoid false positives blocking legitimate connections.  It might therefore be useful to have pre-set blocking tiers, and when the network terror level rises from Cookie Monster to Bert, either dynamically reconfigure the IDS or simply change the packet path to traverse additional layers like Riverhead Networks used to do.

The problem with Hyperspeed, based on what little we know so far, is that it provides yet another tool that doesn't fully solve the problem.  It uses upstream detection to metaphorically set the Evil Bit.  Downstream consumers of this information would theoretically be in a position to apply their own local policies in response to a potential attack.  However, this pass-the-buck security model relies on those downstream targets having their own expertise and automated systems.  Anyone who might find value in an early warning is arguably already prepared, and anyone who has no defense plan will only hear "hey, maybe something bad might happen," which is always true when connected to the Internet.

In short, don't look for this proposal to go anywhere outside of research networks.  The ideas will likely be implemented soon, but the idea is nothing new.

Wednesday, April 25, 2012

"Internet Doomsday" modest proposal

Most Both readers of this blog likely know about the "DNS Changer" malware which, appropriately enough, changed the DNS settings of the systems it infected.  Now those DNS servers are being run by the FBI, who are planning to shut them down July 9 2012 (after pushing the date back at least once).  The concern is that anyone using those DNS servers will not be able to resolve IP addresses, thus making the Internet "go away" for them.

It seems like there's a simple solution here, using a common (albeit unpopular) technique.  Many ISPs, when their DNS is queried for a non-existent address, will return a fake response which, through a hand-waving combination of DNS and HTTP, redirects the user's browser to a web page at that ISP.  Those pages typically say "That site doesn't exist, did you mean this other one, and BTW here are some ads."  This same technique is also used very effectively by OpenDNS for internet filtering.  (The good kind, not the evil kind.)

So... what if the FBI set up a PSA (Public Service Announcement) captive portal solution using this technology?  It's easy enough to set up: cache client IP addresses with a 1-hour sliding timer, i.e. each DNS query resets the clock.  If the client IP is in the list, send the query to the real DNS.  If the client IP isn't in the list, forward the query to the PSA DNS with a 0 TTL.  The browser will load & display the PSA page.

The PSA page should include a brief description of what happened - malware blah blah cleaned blah blah make this change before July 9 blah blah click here to see the official FBI page - as well as directions on re-setting the DNS to its proper value, most likely the DHCP DNS setting.  At an obvious location in the page, there's a link to the website the user was trying to visit.  (Easy to rebuild using the host: HTTP header and the page request.)  Click the link, the client DNS resolves via the real DNS, and the user merrily goes onto the Internet.

Standard traffic stat tools, like the DNS server log, should show whether it's working: each client IP should show up in the web server access log, and over time, the number of clients served should drastically decrease.

It seems like someone is missing something obvious: is it the FBI, or is it me?

Wednesday, December 21, 2011

An intro to networking

There are a lot of resources floating around about networking and OpenFlow, but I plan to publish a couple of blog entries on some use cases, so here's my crack at creating a primer.

The primary job of a network forwarding device is to receive and forward packets (L3) or frames (L2). The primary decision is whether to forward, and out which destination interface. There are different methods to make this decision based on the technology, but arguably the most common implementations deal with forwarding ethernet and TCP/IP, and use a Forwarding information base (FIB).

First, packets vs. frames. Packets are IP (L3), and are transported across ethernet (or other technologies) inside frames (L2). The frame contains the packet. There is global-scope addressing information in the packet for the whole trip (across multiple routers), and there is local-scope addressing information in the frame for each hop between routers. The packet addressing doesn't change across the whole trip, but the frame is re-created every time the packet is forwarded between routers.

(Over-simplification warning!) In TCP/IP over ethernet, switches are used for local delivery of traffic, and routers are used for remote delivery. A computer is connected to a switch, along with lots of other computers, and the switch is also connected to a router. If TCP/IP were the postal service, routers would be post offices, and switches would be the postmen.
  • When a computer wants to send a packet to an IP address, it uses the subnet mask to calculate whether the destination IP is local (in the same subnet) or remote (in a different subnet). An analogy is a 5-digit ZIP code to specify the local post office, and a 9-digit ZIP code to specify the individual house.
  • If a packet's source and destination IP addresses (L3) are in different subnets, the sending computer will set the frame's destination MAC to the router's MAC. The switch reads the MAC addresses of the frame, and sends the frame to the router. The router extracts the packet from the frame, throws the frame information away, reads the packet's destination IP address, makes a forwarding decision, and wraps the packet in a new frame with a destination MAC address to that next router (and its own MAC address as the source). The next router does the same thing: pull the packet from the frame, make a forwarding decision, and create a new frame to send to the next router. The final router, local to the destination IP address, sets the frame's destination MAC to the address for the destination computer. The switch (between the final router and the destination computer) forwards the frame to the computer.
  • However, if the source and destination IP are the same subnet, the sending computer sets the frame's MAC address directly to the destination computer's MAC. The switch will forward the frame directly to its destination, and a router isn't needed.

Generally, routers move packets between IP networks or subnets, and rely on routing protocols to make the decision based on aggregated collections of contiguous IP addresses, defined by applying increasingly larger subnet masks to the IP addresses. A router will check its FIB to figure out whether the destination IP is on one of its local interfaces, or whether it should give the packet to a neighbor router, which then makes the same decision using its own FIB. Routing protocols allow the routers to tell their neighbors about both their locally-connected subnets and their knowledge of larger collections of subnets.  That information is added to the routing table(s), and compiled into the FIB. Anything not in this advertised collection of routes falls into the "default" category, and is forwarded to the router's default gateway. In this way, packets move from areas with little knowledge of the destination towards areas of greater knowledge, and finally to the specific destination node.

Generally, switches move frames between MAC addresses within the same broadcast domain - either connected to the same switch, or to other switches connected to that switch. A switch "learns" where MAC addresses are located based on watching the source interface and MAC address of each frame they forward, then adds that knowledge into the FIB to be used in forwarding future frames.   If a switch doesn't have the MAC address in its FIB, it will typically "flood" the frame, sending a copy out all of its interfaces (except where it received it). Given that most TCP/IP traffic is part of a back-and-forth conversation ("flow"), the destination node will usually send a response, and the switch will learn where the node is, to use in future forwarding decisions. Traditionally, there have not been protocols that switches use to notify each other where the MAC addresses are, but this is starting to change in data center environments through protocols like TRILL (Transparent Interconnect of Lots of Links) and SPB (Shortest-Path Bridging).

It's generally easier to set up a switch than a router, both from the perspective of the actual network forwarding device and from the perspective of the computers they interconnect.  It's also much easier to make architectural changes to the switching infrastructure (adding, removing, etc) than to the routing infrastructure.  Each computer has an IP address and an associated subnet mask, so changing the routing architecture (or, more correctly, the IP subnetting architecture) can require making changes to every computer.  A common sentiment among network designers is therefore, "Switch where you can, route where you must."

What's interesting to me about both routers and switches is that they make forwarding decisions based on information in the FIB.  New technologies are starting to become available which change the way the FIB is created in the routers or switches, resulting in more efficiency and/or more control over how packets flow.

Monday, November 7, 2011

OpenCloudWeekend

So I discovered at OpenCloudWeekend on 11/5 that I actually know enough about OpenFlow to present for an hour.  I guess I'll have to create another SlideShare now just in case.  :-)

Tuesday, November 30, 2010

BayThreat talk teaser

I'm presenting a talk at BayThreat on December 10/11 in Mountain View, CA.  Here's the teaser.

IP Spoofing is a favorite InfoSec Cocktail Party topic.  It's a versatile attack against a node in a local network segment, such as coffee-shop public WiFi.  Cleartext connections can be hijacked through injected data, and TLS/SSL can be closed through injected control bits (e.g. RST).

I will present a proof-of-concept tool on Linux to provide TCP packet signing between previously unaffiliated nodes.  Signed TCP will prevent spoofing by providing cryptographic assurance of the sender.

Designed for incremental adoption and common use, the tool establishes an ephemeral relationship between the nodes during the TCP setup.  Authentication per se is not necessary - all that is required to protect the integrity of the session is validation that the remote node doesn't change mid-stream.

Oddly enough, I haven't found anything else that provides this kind of solution.  Hopefully the presentation will be high on questions and low on face-palms.

Exploding IPs, imploding infosec

I'm getting a funny feeling in my belly about the near future of data centers.  It's my impression that we're starting to see a lot more individual nodes than before, driven by the ease of virtualization.  Specifically:
  1. Virtualization is leading to larger numbers of individually addressable nodes. Each server is capable of hosting multiple guest images. Therefore, manageable addresses are no longer limited by number of physical devices, but only by the speed at which servers can be deployed. Given that there is strong market push in this direction, deployment is receiving increasing amounts of automation, allowing almost arbitrary MAC (moves, adds, changes) – sometimes completely automated.
  2. Virtualization is leading to increased specialization. Guest images are templatized to deploy services quickly: instead of building a server by installing an OS, patches, applications, and localized configuration, do it once, then clone it as needed. The overhead of a separate OS (and >= 1 IP address) per guest is outweighed by the simplicity of decreased administrative and management cost.
  3. Increased demand for performance has led to horizontal scaling and clusters, whereby additional speed, capacity, etc. can be added just by adding more servers.  Web app too slow?  Just spin up another compute node!  Conversely, increased demand for cost savings by reduction of power (and cooling) during periods of decreased demand is leading to dynamic power management, turning off computers when not in use. Coupled with virtualization’s ability to add specialized servers dynamically, it is inevitable that fully automated deployment and retirement of servers will become commonplace. 
As a security guy, especially a security management guy, this makes me really twitchy.  It's not just the massive proliferation of IP addresses: it's the increasingly complex task of applying security policy to those addresses.

Historically, we would have handled this with "meaningful" IP addresses: pre-allocate a range of addresses per application.  Anything with that address is "known" to have that purpose.  However, a massive proliferation of dynamic changes makes pre-allocation difficult, as any range by definition has a fixed number of addresses, and it will be harder to predict how many addresses will be necessary - especially if they're being added and retired automatically.

Filesystems could be analogous to IP address allocation.  Windows users are long-accustomed to occasional de-fragmentation of the hard drive for speed, since blocks of large files don't always get stored consecutively.  Many newer filesystems (especially on Linux) are avoiding the defrag problem by embracing it: don't save a file at the first available location, save it way out on the disk to try to allow for file growth.  Returning to IP, current practice has us looking for contiguous blocks of addresses for a given set of related systems, but address fragmentation will occur despite our best efforts.  If address management becomes fully automated, with abstracted tools to provide access to each IP node, then the allocation headache goes away.  Network management will (eventually) embrace it.

For large address pools, e.g. desktops/laptops/users/etc, DHCP has been the way to go.  I believe that DHCP will be the de-facto solution for server-type addresses.  Spin up another front-end server in the cluster, give it a DHCP address, and programmatically add the address to the load balancer via the API.  For back-end, many cluster solutions already support auto-add.  From a network perspective, it will work reliably and quickly - and that's the #1 criterion.  However, from a security perspective, there's very little chance of being able to make a firewall change with the same automation.  Even if it's technically supported, making security changes is "scary", so it won't fly politically.

Firewall policies are already too big to manage.  If contiguity of addresses goes away, the policies become an order of magnitude larger if the policy is centrally managed.

So... what's the solution?  I see a couple of things as possible:
  1. Abstraction of policies.  Remove references to IP addresses from the firewall rules, and let the system figure out what the actual elements are.  This method tiptoes past the unease about automated firewall policy changes by hiding the dynamic elements behind a reassuringly static view of the policy.  Is it a believable enough reassurance?
  2. Dispersed policies.  VMware has a few tools which apparently allow for firewalling of each VM guest individually (VMsafe API and vShield 2.0).  The implication is that a centralized firewall is less necessary, as each guest has its own firewall.  My concern is that managing 1 firewall per VM may not be any easier than managing a central policy.
  3. Deployment solutions like HyTrust, which limit VM guest deployment to particular VM hosts.  IMHO it's a related but different problem, which essentially creates multiple DMZ networks, minus the network security aspect.
In short: I don't know, but it scares me.

Monday, August 9, 2010

Stupid IPtables Tricks - more tricks w/ recent match

(For a tutorial on the mechanics on how to create state machines with the iptables recent match and recency tables, see my previous blog post.)

1. "Reverse IP tables"
The concept here is to perform the equivalent of port knocking, but instead of opening a firewall pinhole based on inbound connections, the pinhole is opened based on outbound connections.  This technique is especially useful in binding together different apps or services which are running at the same time.

In this example, MySql is setting up database synchronization.  The local master creates an outbound connection to the remote slave, but the remote slave also creates an inbound connection back to the local master.  Normally, the local inbound MySql port is closed.  IPtables will only open the inbound port after the master expresses its implicit trust in the remote slave by creating the outbound connection.

iptables -A OUTPUT -p tcp --dport 3306 -m recent --name dbout --rdest -j ACCEPT
iptables -A INPUT -p tcp --dport 3306 -m recent --name dbout --rcheck -j ACCEPT

The take-away here is that it's possible to make the iptables aware of the state of multiple connections both inbound and outbound.


2. "Log Clog"
When logging firewall events, keep in mind Ranum's Laws of Log Analysis:

Ranum's first law of Log Analysis:
- Never keep more than you can conceive of possibly looking at
Ranum's second law of Log Analysis:
- The number of times an uninteresting thing happens is an interesting thing
Ranum's third law of Log Analysis:
- Keep everything you possibly can except for where you come into conflict with the First Law
(Posted to firewall-wizards mailing list Oct 01,2004)

When using iptables, there isn't a lot of logging sophistication, which means that there's a relatively low limit on the First Law. Thankfully, there's a hack on the Second Law which expands that limit - reduce the logging on what is absolutely uninteresting. I recommend Windows broadcasts. (I actually recommend throwing away Windows, but that's a different blog post.)

Throwing away all instances would be an absolute violation of the Second (and Third) Laws.  Therefore, the key is to throw away most of the instances. Log the first instance, then suppress the rest.
  1. iptables -A INPUT -p udp --dport 138 -m recent --name udp-138 --rcheck --seconds 300 -j DROP
  2. iptables -A INPUT -p udp --dport 138 -m recent --name udp-138 --set
  3. iptables -A INPUT -j LOG --log-prefix “INPUT-drop”
  4. (optional) iptables -A INPUT -j DROP
Line b adds the source IP address to the "udp-138" recency table. It enables the filter in line a, which quietly drops the packet. In this example, the suppression lasts for 300 seconds (aka 5 minutes). Line c logs the packet, and line d drops it. An alternate version of line a would use the "hitcount" match instead of the "seconds" match.

iptables -A INPUT -p udp --dport 138 -m recent --name udp-138 --rcheck --hitcount 100 -j DROP

The advantage is that every log event represents a known larger number of events, so logging statistics can can still detect differences in numbers of "uninteresting" events.

3. "Auth port"
So you have a service which, for reasons you'd rather not discuss, has no auth, but still needs to be used. With iptables, if you can create a wrapper around the client to do auth, you can use the recent table to make the primary port inherit the auth. It's highly imperfect - auth is tied to the IP address, so IP spoofing FTW - but it's better than leaving the port wide open.

The key to success here is to make the auth service iptables-aware - and it has to have a portion running as root.  :-(  Once the auth is successful, the "old" way is to insert a new iptables rule.  The "new" way is to add the IP address to a recency table which is already referenced by a rule.
  1. iptables -A INPUT -d [server] -p tcp --dport 443 -j ALLOW
  2. iptables -A INPUT -d [server] -p tcp --dport 80 -m recent --name authok --rcheck -j ALLOW
Quick show of hands, what's missing here?  There is not a rule to add the IP address to the "authok" recency table.  Instead, the auth app can add the IP to the table from userspace.  Each recency table is exposed in the /proc filesystem in /proc/net/xt_recent/[name]. Therefore, once the auth happens, the service can simply # echo [IP] >> /proc/net/xt_recent/[name] and the address will be added as though it were accepted by a rule.

Take-away: recency tables can be read and written from user space, without having to modify the actual IPtables policy, or going anywhere near the iptables binary.

4. "Quick & Dirty IDS"
Doing a web search for "iptables -m recent" will return a result on the first page to do IP address blocking, probably something like this page:
  1. iptables -A FORWARD -m recent --name noplacelike --rcheck --seconds 60 -j DROP
  2. iptables -A FORWARD -i eth0 -d 127.0.0.0/8 -m recent --name noplacelike --set -j DROP
Line a drops the packet quietly if it was blocked (by line b) within the last 60 seconds. Obviously, blocking an IP address forever simply isn't realistic. However, instead of blocking for only 60 seconds, it's possible to use a sliding window.

iptables -A FORWARD -m recent --name noplacelike --update --seconds 60 -j DROP

After identifying the attacker (in line b), using "--update" instead of "--rcheck" will both check and re-set the address in the recency table.  As long as the attacker continues to connect to the target system, if the previous connection is within 60 seconds of the previous connection, the attacker's system will continue to be blacklisted.


That's it for the "sane" IPtables tricks with the "recent" match. Next blog entry: some of the crazy potential uses of iptables.

Sunday, August 8, 2010

Stupid IPtables Tricks - the short & dense version - port knocking intro

Until I can find somewhere to host my slides, I'll post the "core" of the talk here.

I like using IPtables to solve problems because it adds a bump between the kernel and the application/service.  If you can "solve" a problem using IPtables, you can implement that solution for any app without re-coding or re-configuring.

Of course, best practice is to use good auth & encryption with standard solutions, like SSL/TLS.  Enough about that for now.

Most of the tricks I know are based on the '-m recent' filter, which stores an IP address and timestamp(s) in a named recency table.  Don't confuse this with the state tables like ESTABLISHED or chains like INPUT.  It's a different beast altogether.

Trick 1: Port knocking with IPtables.
In this trick, create a finite state machine using multiple new chains and recency tables.  The way it works is that the INPUT chain checks to see if the IP address is in the recency table for a state, moves to the associated chain, and adds the address to the recency table for the next state.  Here's a 4 state example:
  1. icmp ping
  2. TCP echo
  3. TCP port 8000
  4. TCP port 3306
  1. icmp ping:
iptables -A INPUT -p icmp --icmp-type 8 -m recent --name seenping --set -j ACCEPT

The filters in this line are "-p icmp --icmp-type 8", icmp echo-request (ping), and can be narrowed down more with -s or -d options.  The final action for this line is "-j ACCEPT", to send the packet through.  The differentiator is "-m recent --name seenping --set", which adds the source IP address of the packet to the recency table called "seenping".
  1. tcp echo
  1. iptables -N ADD_ECHO
  2. iptables -A INPUT -p tcp --dport echo -m recent --name seenping --rcheck -j ADD_ECHO
  3. iptables -A ADD_ECHO -m recent --name seenecho --set -j ACCEPT
Line a: Create a new chain called ADD_ECHO to represent the 2nd state in the finite state machine.  It's necessary to use an additional chain for each state, as the "-m recent" syntax only allows for a single recency table reference per command.  Fortunately, recency tables references can be isolated into different chains, as in lines b and c.

Line b: This is the filter command.  If the packet matches the next state, TCP echo, with the source IP address listed in the "seenping" recency table, then move processing to the ADD_ECHO chain.

Line c: the ADD_ECHO chain only does 2 things: add the source IP address to the "seenecho" recency table (priming the filter for the next state), and allowing the packet.  Priming the next state is handled by "-m recent --name seenecho --set".

The action in line c is '-j ACCEPT'.  During my talk, there was a great question: does the action have to be an ACCEPT?  The answer is, no, it can be whatever you want.  There can even be no action listed in this policy.  If iptables reaches the end of the ADD_ECHO chain without finding a terminating action (.e.g ACCEPT, DROP, REJECT), it will automatically pop the stack back to the calling chain.  Additionally, returning to the calling chain can done explicitly via the "-j RETURN" action target.
  1. Back-end port: TCP 8000
  1. iptables -N ADD_BACKEND
  2. iptables -A INPUT -p tcp --dport 8000 -m recent --name seenecho --rcheck --seconds 10 -j ADD_BACKEND
  3. iptables -A INPUT -m recent --name seenecho --remove
  4. iptables -A ADD_BACKEND -m recent --name seenbackend --set -j ACCEPT
Line a: Create iptables chain called "ADD_BACKEND" to allow state transition to the next state, from 8000 to 3306.

Line b: Filter the incoming packets and trigger the execution of the ADD_BACKEND chain. The filter here is "-p tcp --dport 8000", combined with the source IP address listed in the "seenecho" recency chain. The action is to jump to the "ADD_BACKEND" chain.

The new parameter in line b is the "--seconds" filter. An attacker might be port-scanning the system and accidentally open the state machine up to this state.  Placing a time restriction is a simple method of reducing false positives on the port knock.

Line c is the next line in the INPUT chain after the state transition filter.  If the packet is NOT a match, line c will clear the IP address out of the "seenecho" recency table.  If the client IP host does ANYTHING other than try to connect to TCP port 8000 after sending the TCP echo, the state machine will reject its actions as invalid, and the port knock will fail.  What happens from here isn't specified in this example, so the client could re-send a TCP echo to reset the state and timer for "seenecho".  (Note that the IP is never explicitly purged from "seenping", which is probably bad practice.)  Alternatively, the IP address could be banned for a period of time using another IP tables rule or trick.

Line d is the state progression command, same as step 2.
  1. TCP port 3306
iptables -A INPUT -p tcp --dport 3306 -m recent --name seenbackend --rcheck -j ACCEPT

The closing state of the state table allows the client IP address to connect to the mysql port.  Since there's no "next" state, there's no need for another recency table, nor for another chain.

An alternate version of this step would be to create a chain specifically for the "opened" state to aid in overall policy maintenance through separation of rules into discrete chains.
  1. iptables -N MYSQL_OPENED
  2. iptables -A INPUT -m recent --name seenbackend --rcheck -j MYSQL_OPENED
  3. iptables -A MYSQL_OPENED -p tcp --dport 3306 -j ACCEPT
If there is additional  packet-munging you want to perform once the port knocking is done, use this second approach, and replace line c with whatever you want at this stage of the policy.

Note that, for each stage of this example, state progression happened blindly as the first rule in a chain.  If the overall iptables policy is modified to include additional references to that chain, there will be additional methods of prgressing through the port knock.  Whether or not this is a good idea is up to you.

So, there you have it.  That's a 4-stage port knock with false positive rejection done entirely in IPtables, using 3 recency tables for state storage, plus 3 chains (beyond INPUT and ACCEPT) for state progression.

If you have read this far, you have all of the knowledge necessary for the next batch of Stupid IPtables Tricks, which I'll put in my next blog post.

To really make sure you understand the recency tables, check out the excellent example in the iptables tutorial called recent-match.txt.  If I hadn't slogged through it, I wouldn't know how this stuff works.

Monday, July 19, 2010

Stupid IPtables tricks - teaser

On 7/28/2010 at 6:00pm at BSides Las Vegas, I'll be giving a talk called "Stupid IPtables Tricks". Here's the quick overview.

Update / Clarification: the techniques I'll discuss can't all be done "natively" in iptables.  Some of them leverage other tools, like Dans Guardian for URL filtering.  IPtables itself isn't good at L7, but it can do lovely manipulation below that.

IPtables is the standard Linux firewall, happily implementing policy at L3/L4. As such, it looks like an older-generation firewall, stuck firmly in the Stateful Packet Filter realm of technology. It knows about UDP and TCP, but has no clue about HTTP, much less your userid on FaceTube.

However, iptables has one thing going for it that a lot of turn-key commercial products don't - it conforms to the Linux idea of being programmable and extensible. Beyond the basic TCP 3-way handshake, have-I-seen-this-circuit statefulness, it allows for nearly arbitrary reactions to and actions on packets. Furthermore, it's got some hooks that allow for blurring the lines between the network layer and the application layer.

This talk will focus on creating IPtables policies with weird abilities that aren't always found in other firewall implementations. Examples include:

  • Using external auth to tie a user to an IP, then implementing different policies based on that IP
    • URL filtering, with different policies per-group
    • SNAT with different IP address based on user group
    • Outbound routing through high-bandwidth, low-latency, etc.
    • Slowing down network access for low-priority users
  • De-clogging logs
    • Why must Windows discovery broadcasts be so chatty?
  • Poor-man's blocking IDS
    • Ignore inbound IP for a while if it seems naughty
  • Implementing port knocking on the gateway
  • Anti-spam IP blacklisting and graylisting
  • Tying iptables policies to the underlying applications on the same host
    • Allow outbound ssh only if it's really from the ssh process
    • Allow inbound connections to arbitrary / dynamic ports for specific applications
      • Whoever decided dynamic ports were a good idea had no bloody experience with implementing security!
The talk will also discuss some classic limitations of L3/L4 policy implementation, especially session hijacking and "joe jobs" through IP spoofing.

This talk will not focus on ways to bypass iptables. That's just about every other talk this week in Vegas.

Why am I giving this talk?  My job is to retrofit security onto NMS appliances for a large networking vendor.  My personal goal is to keep 80% of you from cracking my department's product within the first week of its release, and to convince my team to make my job unnecessary by writing code that's secure.