Flame - something only a government could build?

The NY Times (June 3 2012) published an article saying that Kaspersky Labs has declared Flame to be something only a government could build.  I disagree.

Look at the purported history of Stuxnet.  The most comprehensive story so far comes from David Sanger - again in the NY Times - saying that it was a project started under Bush-43, continued under Obama-44, co-developed with Israel's Unit 8200.  The larger story implies that the development was broken down into modules coded separately, with the programmers potentially unaware of the nature of the project.  This sounds a lot like the development process of the virus in Neal Stephenson's 1992 novel Snow Crash, in which (ironically) the US Government is the contractor creating the virus for the villain.  The Government was chosen as the development partner because they are the only organization paranoid enough not to trust their programmers with any "big picture" view.

Flame is a bloated beast, 20MB in size, with business logic in Lua calling compiled C++ modules.  While there's nothing unusual about that structure - even large web sites are using hybrid combinations of languages like Scala and Java - it does add credence to the idea that this is code that was not written by one person, nor potentially even one organization.  Flame could easily have been developed according to "Snow Crash" paranoid secret separation of information principles.

We've heard statements before about some things only being possible by government-level operations.  My favorite example was when members of the l0pht testified before Congress about potentially taking down the Internet in 30 minutes.  Government reaction was astonishment - they'd assumed that no private nor commercial organization had such capabilities, and had to go back and re-write their threat models.

In short: while Flame is a sophisticated toolkit for malware, it's not something that only a government could build.  However, the way that it's put together is the way that a government would build it.

In researching this post, I finally found some information that hadn't been exaggerated bouncing around the media echo chamber in a BBC News article that actually quotes someone at Kaspersky Labs. Vitaly Kamluk, "chief malware expert" per the BBC, is quoted thusly:

"Currently there are three known classes of players who develop malware and spyware: hacktivists, cybercriminals and nation states. Flame is not designed to steal money from bank accounts. It is also different from rather simple hack tools and malware used by the hacktivists. So by excluding cybercriminals and hacktivists, we come to conclusion that it most likely belongs to the third group."

 My reading of this statement is that Flame seems consistent, based on its actions, with something that isn't consistent with non-government malware.  However, that headline won't sell news stories.

What the Hyperspeed "Bullet Time" guys might be thinking

Background: "Bullet Time" article in New Scientist

When this article hit Twitter, lots of us thought that it was waving-of-hands security, no more worthy of contemplation than the Evil Bit joke RFC.  However, like the Evil Bit, there might be some deeper truth to what they're proposing.

The journal article which contains the proposal focuses on creating low-bandwidth high-speed links within MPLS networks.  In that sense, it's an example of RFC 1925 truth 11, re-introducing an old idea to solve a new problem.  Prioritization of network control traffic was built into the TOS flags in IP, which were later replaced with DSCP.  It's not a bad idea, just one which has never found its "killer app", so has not had widespread adoption.  (Whether QoS has widespread adoption is a discussion left as an exercise to the reader.)

The basic concept of "Hyperspeed", the name given to the proposal, is that network control type traffic be pre-provisioned into existing infrastructure.  The New Scientist article focuses on a security use case in which cloud-based traffic examination (I'm putting words into their mouth here, they gracefully avoid using the c-word) detects potentially malevolent traffic and pre-warns the recipient network of a potential attack.  However, I suspect that the NS article suffers from the same media hype and distortion which surrounds a lot of technical discussion.

Taken on its own, there is a reasonable argument for this type of approach.  Adding additional layers of network defense can interfere with traffic.  Whether we security folks like it or not, speed trumps security when providing services.  Therefore, if it is economically feasible to inject high-speed low-latency analysis at a centralized point upstream of the target network, it would be useful to have a coordination protocol which automatically triggers deeper levels of analysis.

Anyone who has deployed a blocking IDS will immediately recognize that tuning the system is tricky.  Thresholds must be localized for the target network to avoid false positives blocking legitimate connections.  It might therefore be useful to have pre-set blocking tiers, and when the network terror level rises from Cookie Monster to Bert, either dynamically reconfigure the IDS or simply change the packet path to traverse additional layers like Riverhead Networks used to do.

The problem with Hyperspeed, based on what little we know so far, is that it provides yet another tool that doesn't fully solve the problem.  It uses upstream detection to metaphorically set the Evil Bit.  Downstream consumers of this information would theoretically be in a position to apply their own local policies in response to a potential attack.  However, this pass-the-buck security model relies on those downstream targets having their own expertise and automated systems.  Anyone who might find value in an early warning is arguably already prepared, and anyone who has no defense plan will only hear "hey, maybe something bad might happen," which is always true when connected to the Internet.

In short, don't look for this proposal to go anywhere outside of research networks.  The ideas will likely be implemented soon, but the idea is nothing new.

"Internet Doomsday" modest proposal

Most Both readers of this blog likely know about the "DNS Changer" malware which, appropriately enough, changed the DNS settings of the systems it infected.  Now those DNS servers are being run by the FBI, who are planning to shut them down July 9 2012 (after pushing the date back at least once).  The concern is that anyone using those DNS servers will not be able to resolve IP addresses, thus making the Internet "go away" for them.

It seems like there's a simple solution here, using a common (albeit unpopular) technique.  Many ISPs, when their DNS is queried for a non-existent address, will return a fake response which, through a hand-waving combination of DNS and HTTP, redirects the user's browser to a web page at that ISP.  Those pages typically say "That site doesn't exist, did you mean this other one, and BTW here are some ads."  This same technique is also used very effectively by OpenDNS for internet filtering.  (The good kind, not the evil kind.)

So... what if the FBI set up a PSA (Public Service Announcement) captive portal solution using this technology?  It's easy enough to set up: cache client IP addresses with a 1-hour sliding timer, i.e. each DNS query resets the clock.  If the client IP is in the list, send the query to the real DNS.  If the client IP isn't in the list, forward the query to the PSA DNS with a 0 TTL.  The browser will load & display the PSA page.

The PSA page should include a brief description of what happened - malware blah blah cleaned blah blah make this change before July 9 blah blah click here to see the official FBI page - as well as directions on re-setting the DNS to its proper value, most likely the DHCP DNS setting.  At an obvious location in the page, there's a link to the website the user was trying to visit.  (Easy to rebuild using the host: HTTP header and the page request.)  Click the link, the client DNS resolves via the real DNS, and the user merrily goes onto the Internet.

Standard traffic stat tools, like the DNS server log, should show whether it's working: each client IP should show up in the web server access log, and over time, the number of clients served should drastically decrease.

It seems like someone is missing something obvious: is it the FBI, or is it me?

An intro to networking

There are a lot of resources floating around about networking and OpenFlow, but I plan to publish a couple of blog entries on some use cases, so here's my crack at creating a primer.

The primary job of a network forwarding device is to receive and forward packets (L3) or frames (L2). The primary decision is whether to forward, and out which destination interface. There are different methods to make this decision based on the technology, but arguably the most common implementations deal with forwarding ethernet and TCP/IP, and use a Forwarding information base (FIB).

First, packets vs. frames. Packets are IP (L3), and are transported across ethernet (or other technologies) inside frames (L2). The frame contains the packet. There is global-scope addressing information in the packet for the whole trip (across multiple routers), and there is local-scope addressing information in the frame for each hop between routers. The packet addressing doesn't change across the whole trip, but the frame is re-created every time the packet is forwarded between routers.

(Over-simplification warning!) In TCP/IP over ethernet, switches are used for local delivery of traffic, and routers are used for remote delivery. A computer is connected to a switch, along with lots of other computers, and the switch is also connected to a router. If TCP/IP were the postal service, routers would be post offices, and switches would be the postmen.

• When a computer wants to send a packet to an IP address, it uses the subnet mask to calculate whether the destination IP is local (in the same subnet) or remote (in a different subnet). An analogy is a 5-digit ZIP code to specify the local post office, and a 9-digit ZIP code to specify the individual house.
• If a packet's source and destination IP addresses (L3) are in different subnets, the sending computer will set the frame's destination MAC to the router's MAC. The switch reads the MAC addresses of the frame, and sends the frame to the router. The router extracts the packet from the frame, throws the frame information away, reads the packet's destination IP address, makes a forwarding decision, and wraps the packet in a new frame with a destination MAC address to that next router (and its own MAC address as the source). The next router does the same thing: pull the packet from the frame, make a forwarding decision, and create a new frame to send to the next router. The final router, local to the destination IP address, sets the frame's destination MAC to the address for the destination computer. The switch (between the final router and the destination computer) forwards the frame to the computer.
• However, if the source and destination IP are the same subnet, the sending computer sets the frame's MAC address directly to the destination computer's MAC. The switch will forward the frame directly to its destination, and a router isn't needed.

Generally, routers move packets between IP networks or subnets, and rely on routing protocols to make the decision based on aggregated collections of contiguous IP addresses, defined by applying increasingly larger subnet masks to the IP addresses. A router will check its FIB to figure out whether the destination IP is on one of its local interfaces, or whether it should give the packet to a neighbor router, which then makes the same decision using its own FIB. Routing protocols allow the routers to tell their neighbors about both their locally-connected subnets and their knowledge of larger collections of subnets.  That information is added to the routing table(s), and compiled into the FIB. Anything not in this advertised collection of routes falls into the "default" category, and is forwarded to the router's default gateway. In this way, packets move from areas with little knowledge of the destination towards areas of greater knowledge, and finally to the specific destination node.

Generally, switches move frames between MAC addresses within the same broadcast domain - either connected to the same switch, or to other switches connected to that switch. A switch "learns" where MAC addresses are located based on watching the source interface and MAC address of each frame they forward, then adds that knowledge into the FIB to be used in forwarding future frames.   If a switch doesn't have the MAC address in its FIB, it will typically "flood" the frame, sending a copy out all of its interfaces (except where it received it). Given that most TCP/IP traffic is part of a back-and-forth conversation ("flow"), the destination node will usually send a response, and the switch will learn where the node is, to use in future forwarding decisions. Traditionally, there have not been protocols that switches use to notify each other where the MAC addresses are, but this is starting to change in data center environments through protocols like TRILL (Transparent Interconnect of Lots of Links) and SPB (Shortest-Path Bridging).

It's generally easier to set up a switch than a router, both from the perspective of the actual network forwarding device and from the perspective of the computers they interconnect.  It's also much easier to make architectural changes to the switching infrastructure (adding, removing, etc) than to the routing infrastructure.  Each computer has an IP address and an associated subnet mask, so changing the routing architecture (or, more correctly, the IP subnetting architecture) can require making changes to every computer.  A common sentiment among network designers is therefore, "Switch where you can, route where you must."

What's interesting to me about both routers and switches is that they make forwarding decisions based on information in the FIB.  New technologies are starting to become available which change the way the FIB is created in the routers or switches, resulting in more efficiency and/or more control over how packets flow.

OpenCloudWeekend

So I discovered at OpenCloudWeekend on 11/5 that I actually know enough about OpenFlow to present for an hour.  I guess I'll have to create another SlideShare now just in case.  :-)

BayThreat talk teaser

I'm presenting a talk at BayThreat on December 10/11 in Mountain View, CA.  Here's the teaser.

IP Spoofing is a favorite InfoSec Cocktail Party topic.  It's a versatile attack against a node in a local network segment, such as coffee-shop public WiFi.  Cleartext connections can be hijacked through injected data, and TLS/SSL can be closed through injected control bits (e.g. RST).

I will present a proof-of-concept tool on Linux to provide TCP packet signing between previously unaffiliated nodes.  Signed TCP will prevent spoofing by providing cryptographic assurance of the sender.

Designed for incremental adoption and common use, the tool establishes an ephemeral relationship between the nodes during the TCP setup.  Authentication per se is not necessary - all that is required to protect the integrity of the session is validation that the remote node doesn't change mid-stream.

Oddly enough, I haven't found anything else that provides this kind of solution.  Hopefully the presentation will be high on questions and low on face-palms.

Exploding IPs, imploding infosec

I'm getting a funny feeling in my belly about the near future of data centers.  It's my impression that we're starting to see a lot more individual nodes than before, driven by the ease of virtualization.  Specifically:

1. Virtualization is leading to larger numbers of individually addressable nodes. Each server is capable of hosting multiple guest images. Therefore, manageable addresses are no longer limited by number of physical devices, but only by the speed at which servers can be deployed. Given that there is strong market push in this direction, deployment is receiving increasing amounts of automation, allowing almost arbitrary MAC (moves, adds, changes) – sometimes completely automated.
2. Virtualization is leading to increased specialization. Guest images are templatized to deploy services quickly: instead of building a server by installing an OS, patches, applications, and localized configuration, do it once, then clone it as needed. The overhead of a separate OS (and >= 1 IP address) per guest is outweighed by the simplicity of decreased administrative and management cost.
3. Increased demand for performance has led to horizontal scaling and clusters, whereby additional speed, capacity, etc. can be added just by adding more servers.  Web app too slow?  Just spin up another compute node!  Conversely, increased demand for cost savings by reduction of power (and cooling) during periods of decreased demand is leading to dynamic power management, turning off computers when not in use. Coupled with virtualization’s ability to add specialized servers dynamically, it is inevitable that fully automated deployment and retirement of servers will become commonplace. 

As a security guy, especially a security management guy, this makes me really twitchy.  It's not just the massive proliferation of IP addresses: it's the increasingly complex task of applying security policy to those addresses.

Historically, we would have handled this with "meaningful" IP addresses: pre-allocate a range of addresses per application.  Anything with that address is "known" to have that purpose.  However, a massive proliferation of dynamic changes makes pre-allocation difficult, as any range by definition has a fixed number of addresses, and it will be harder to predict how many addresses will be necessary - especially if they're being added and retired automatically.

Filesystems could be analogous to IP address allocation.  Windows users are long-accustomed to occasional de-fragmentation of the hard drive for speed, since blocks of large files don't always get stored consecutively.  Many newer filesystems (especially on Linux) are avoiding the defrag problem by embracing it: don't save a file at the first available location, save it way out on the disk to try to allow for file growth.  Returning to IP, current practice has us looking for contiguous blocks of addresses for a given set of related systems, but address fragmentation will occur despite our best efforts.  If address management becomes fully automated, with abstracted tools to provide access to each IP node, then the allocation headache goes away.  Network management will (eventually) embrace it.

For large address pools, e.g. desktops/laptops/users/etc, DHCP has been the way to go.  I believe that DHCP will be the de-facto solution for server-type addresses.  Spin up another front-end server in the cluster, give it a DHCP address, and programmatically add the address to the load balancer via the API.  For back-end, many cluster solutions already support auto-add.  From a network perspective, it will work reliably and quickly - and that's the #1 criterion.  However, from a security perspective, there's very little chance of being able to make a firewall change with the same automation.  Even if it's technically supported, making security changes is "scary", so it won't fly politically.

Firewall policies are already too big to manage.  If contiguity of addresses goes away, the policies become an order of magnitude larger if the policy is centrally managed.

So... what's the solution?  I see a couple of things as possible:

1. Abstraction of policies.  Remove references to IP addresses from the firewall rules, and let the system figure out what the actual elements are.  This method tiptoes past the unease about automated firewall policy changes by hiding the dynamic elements behind a reassuringly static view of the policy.  Is it a believable enough reassurance?
2. Dispersed policies.  VMware has a few tools which apparently allow for firewalling of each VM guest individually (VMsafe API and vShield 2.0).  The implication is that a centralized firewall is less necessary, as each guest has its own firewall.  My concern is that managing 1 firewall per VM may not be any easier than managing a central policy.
3. Deployment solutions like HyTrust, which limit VM guest deployment to particular VM hosts.  IMHO it's a related but different problem, which essentially creates multiple DMZ networks, minus the network security aspect.

In short: I don't know, but it scares me.