Building tunnels for early adoption of new tech

Imagine there's a cool new tech that potentially solves a real problem when using the Internet. In this case, do you ever find yourself dealing with wifi so bad that you pull out your phone and tether instead¹? Coded TCP (aka TCP/NC or CTCP)² can potentially solve it. However, the solution requires an end-to-end change: it would have to be adopted by every website you visit. How do we deal with this?

The problem turns out to be similar to IPv6, which is incompatible with the IPv4 that's historically been used. Running native IPv6 required end-to-end support, including your WiFi router, your Internet router (cable modem, DSL, etc.), your Internet provider, and the website. If a website only supported IPv6, there was no way for an IPv4 client to reach it. The early adoption solutions generally used tunneling like Teredo³. In Teredo, the client would create an IPv6 packet, then wrap it in IPv4 and deliver it to a "middlebox", which would extract the IPv6 payload and deliver it natively to the website.

Network coding could similarly be deployed using tunneling and a middlebox. The solution looks a little bit different, but the end result is the same: the client machine can use the advantages of the new protocol on the Internet in a way that's transparent to the local network and to the website.

Conceptually, I imagine that CTCP would be encapsulated using UDP between the client and the middlebox, since the primary advantage is reducing retransmissions on the client's local WiFi network. The middlebox would then L4 proxy the data stream to the end website, extracting the data transparently from the client CTCP and re-encoding it to the server's legacy TCP. In this sense, it's the opposite of Teredo: CTCP's value is the client-middlebox connection, and Teredo's value was the middlebox-server connection.

There are of course some difficulties in this scenario.

• Encapsulated traffic is difficult for firewalls to protect, as the real addressing info is buried inside a wrapper. Additionally, there may be L7 security concerns from HSTS, HTTP/2, etc., stemming from the recoding at L4 instead of L3.
• For graceful deprecation, what happens if CTCP becomes more common and some sites don't need a middlebox connection? How does the client recognize the difference and decide which connection to use?
• Google's QUIC protocol replaces TCP's congestion control and retransmissions with its own mechanism, which they claim is similarly designed to work with heavy packet loss. That would potentially remove the majority use case for CTCP
• CTCP is a trade secret, so there are financial and legal barriers in the way of widespread adoption.

Still, this would be a fun thing to prototype.

---

<sup>1. If you use your cell phone to tether when the WiFi is bad, don't tether via WiFi, as you're just adding more traffic to the already poor wireless spectrum, and even if it works for you, it makes the problem worse for everyone around you. If you have to tether, do it via USB (RNDIS on the PC), as it doesn't create more wireless traffic, and it lets your phone charge while you use it.↩

2. Coded TCP uses linear network encoding to spread payload data across multiple packets, so a lost packet can be reconstructed on the receiving end.↩

3. Teredo introduced some serious security problems, as it allowed IPv6 to bypass the firewall on the client's network, so it became best practice to turn it off. Ironically, many networks today which natively support IPv6 still don't apply firewall policy to the protocol, providing a similar level of insecurity.↩</sup>

When will we see NFV as a Service?

NFVaaS - while it sounds like a joke about mixing acronyms between cloud ('x' as a Service) and SDN, NFV is already a serious mix of the two technologies - and I think there's a lot of promise in taking this joke a lot farther.

SDN provides centrally controlled packet steering - a planned path, through specified forwarding engines, between endpoints. Cloud provides orchestrated 'service' deployment, automating the setup of VMs/containers/whatever. NFV mixes the two concepts, to use cloud orchestration to deploy middleboxes into the SDN packet path, so it's possible to steer connections through firewalls, NAT, IDS, DLP, load balancers, caching engines, etc. The key point is that the services are consumed not by users, but by packets/sessions, thus making NFV both into "cloud for networking" and "networking for cloud". For the rest of this post, I'm going to simplify NFV into the concept of "orchestrating network middleboxes".

So how is NFVaaS different from just NFV? The XaaS term implies wrapping something up in a turn-key way so that it's easy to consume. DBaaS gives me an interface to a running database onto which I can apply schema and do some SCRUD, instead of setting up a server and installing software. IaaS gives me virtual bare metal. NFVaaS therefore goes beyond applying middleboxes as part of delivering or munging sessions for some other application - the whole point of NFVaaS is that I want middleboxes through which to push my own packets. If I have a network that needs a firewall, what I have is a problem that could be solved by NFVaaS, deploying a firewall through which I can push my packets.

To put this another way, if my network use is primarily as a client (rather than running a web site, etc.) then the primary service I consume is "access". To add value to that service, I want to add middleboxes, but I want it "cloud-style": usable in minutes, with costs relating to my use rather than acquisition and maintenance. NFV provides the ability for someone to deliver that service to me.

Some technically-minded readers may be asking, "how do you insert something into the packet path?" There are a few different ways, but what I personally prefer is using a VPN. Right now I'm renting a VPS which I use as a VPN concentrator and cloud storage server, with the intention of adding IP blocking based on an open theat intelligence feed. At some point I might add Squid as a transparent proxy running ClamAV. There's a lot I can do to create my own "clean pipe" service, and the value to me greatly exceeds the VPS cost. However, it took me a while to set up what I have, and will take a lot more time to add (and tune) the additional capabilities. If I could get all of this as a service for a reasonable cost, I'd recommend it to all of my non-technical friends and family.

NFV gets a lot of attention in big networks like Service Providers, Data Centers, and Large Enterprise - in short, anyone running their own private clouds - but just because it runs in a cloud doesn't mean that it doesn't have value to everyone else. I'm definitely looking forward to renting time on a IDS.

Who will own the virtual network?

There are two network virtualization products being pushed by heavyweight vendors. In this corner: VMware NSX. In that corner: Cisco ACI.

The debate is ongoing how to add agility to networking. The overlay model, like NSX, says that the network can be simple as long as the apps are smart. The controller model, like ACI, says that the network should be aware of the apps.

On the overlay side, the strength is in its deployment simplicity: end nodes communicate with each other, but the network itself (the "underlay") doesn't require any adjustment. Its weakness is the flip side: there's no way to tune the underlay, so there's no way around bottlenecks and other potential artifacts of the architecture.

On the controller side, the strength is the potential of the network to adjust to support whatever flow patterns the app requires. There are two weaknesses: first, if the app doesn't integrate with the controller API, then the controller has to interpret the data it's carrying and make a best guess on optimal configuration; second, the "smart" abilities generally require new hardware, which greatly slows down the rate of adoption.

In the short term (1.5-3 years), it's likely that NSX will be deployed much more than ACI. NSX doesn't "technically" require the server team to involve the network team in the roll-out, and the agility that's grown from virtualization into modern devops will lead both to an increased tolerance for risk and a demand for better network delivery. (As a network guy, I'm used to hearing the network always getting blamed for any problem.) The fertile ground will be large applications hosted either classically or in private cloud, especially where the app is multi-tiered (with lots of local east-west traffic), but may grow as NSX support increases across data centers.

The question is whether ACI will take hold, and where. The fertile ground will likely be single-owner data centers with monetized services, where ACI will be seen as an investment to increase capacity, and API integration can be pushed top-down into the hosted systems. However, the Internet Giants (Google, Facebook, etc.) have demonstrated that they're willing to create their own network technology rather than purchase big-ticket vendor products, and analysts and pundits are declaring that everyone else will move to cloud hosting. Even though that prediction is hyperbole, it does put downward pressure on the ACI target markets.

Keep in mind that 3 years ago, the "future of networks" was OpenFlow, so any future predictions that aren't wildly inaccurate will likely be due to blind luck. :-)

What do you think? That's what the comments are for down below.

A sonnet about the TSA

This was my entry for the Liquid Matrix Podcast contest for a pass to BSides Las Vegas. I didn't win, but wanted to share it with you.

The Iron Wisdom of the TSA
inspired by a Vendor DERP from Liquid Matrix episode 0x2C

The airline tells us that their onboard staff
are there for safety, not for in-flight ease.
And yet SouthWest will brief us with a laugh,
and once airborne, serve beverages to please.

But safety on the ground from TSA:
before you soar, they make your spirits sink.
Remove your shoes, your liquids throw away!
No wonder on the plane you need a drink.

Your courtesy is answered with their scorn,
suspicion is delivered without tact.
So cover up, the scanners churn out porn!
There's theater security to act.

The mission of the TSA thus laid:
"Security" keeps passengers afraid.

Want innovation? Reduce the cost of education!

Today's post is inspired by the discussion over US visas for foreign workers, and the argument from high-tech companies that there aren't enough new US grads in STEM (Science, Tech, Engr, Math) to fill the open positions. Specifically, I'm reacting to the "data" in this article, which says what we've all suspected for the last 15 years: US companies can't find qualified people who will are willing to work for what the positions pay.

My gut reaction (no data to back this up other than personal experience) is that new US grads have a disadvantage that a lot of foreign grads historically haven't: student loan debt. If a new US grad leaves school with $50k-$150 in student loans, that's a strong disincentive to take risks. Instead, a monthly loan bill will demand that students quickly find a reliable source of income. Additionally, cost of living in traditional areas where STEM companies congregate is high. In my case, it was the Silicon Valley in 1996, when I was "lucky" to pay $1200/mo for a 2-bedroom apartment.

The "mythical" Silicon Valley startup mentality had the motto, "work any 80 hours a week you want," and didn't care what your education level was as long as you got the job done. Now, startups are less work-intensive, but early stage positions still offer very little income. If these companies want to attract the "best and brightest", one way to do it is to increase applicants' openness to risk.

A relatively recent idea has been the "hacker hostel" model, in which startup incubators come with not only office space, but sleep space. While that reduces the monthly expenditure for a new hire, it has its own disadvantages, including being appealing primarily to early-stage companies (read: the co-founders are the ones sleeping on the couch).

A more long-reaching solution would be to drive the cost of education back down. Forbes reported in 2012 that education has risen 500% since 1985, while the consumer price index was only 115%. There's a pretty picture here (note: data from same source). While there was some re-structuring a couple of years ago, in which some colleges switched from mixed financial aid to grant-only, that story has fallen out of the media, and costs haven't really been affected. Furthermore, the global economic downturn has reduced spending on eduction (at least in the US), driving tuition & fees higher, even in state schools.

A reduction in the cost of education is a likely driver to innovation specifically because it will reduce the p2p driver that killed so many startups. In business, p2p for a VC-backed firm was "path to profitability", insisting on rapid repayment of investment by bringing products to market early. In education, I'm using the term as "path to payment", repayment of student loans. If we want graduates to work for less money, then let's make it possible for them to do so.

Urgency and activity

I've noticed that I tend to get work done best when I'm nearly out of time to do it. Presumably, I'm not the only one who works like this, but it definitely causes... let's call it "friction" with the people I work with who seem to work at a constant pace.

I wonder sometimes if my life would be different if I could maintain that constant pace. Given my irregular output, I see myself as a miracle worker who would thrive in an environment of being called in to get stuff done in an emergency. However, my co-workers tend to see me as unreliable, which has occasionally been career-limiting.

A few years ago, the team I was part of was introduced to Scrum, an Agile methodology. At the time, we were told in training that one of the goals was to even out the crisis humps, replacing the "mad rush" at the deadline with a series of smaller deadlines, so there'd be no all-nighters to get things done. Looking back now, I wonder if it was a compromise to try to get "bursty" workers to increase their output, or at least make their output more predictable to schedule-keepers.

Driving home today, I wondered about what a crazy frenetic work environment would be like, in which goals for the day were handed out in the morning. After a while I know it would be maddening, and that very few people would survive working there. However, if they could put the right team together, imagine the output!

Some of this thought is driven by just having read The Phoenix Project, a novel about DevOps. It's a great read, and if you're an Amazon Prime member with a Kindle, you can "borrow" it for free. That's what I did before buying it. :-)

What it got me thinking about is how to overcome "impossible" challenges. In the book, one such challenge is going from a deploy every 9 weeks to a deploy of 10 times per day. It's called "impossible" by some characters, but, since the novel is a polemic about DevOps, of course they figure it out quickly. The key is to challenge your assumptions about what's possible. At my current job, we recently had a semi-successful meeting where some team members argued that what was being asked for was "against the laws of physics". That to me is a sign that you need to approach the problem differently, set aside the "impossible" label, and look at what obstacles are in the way. If you can re-route around those obstacles, or wave a magic wand to streamline them (by several orders of magnitude), then the solution becomes possible, and the new challenge is driven by the excitement of doing what you knew yesterday was impossible.

I guess the challenge for me now is to figure out how to find something impossible every day... make each day a mini-sprint, and see what happens to my output.

Blogging: inconsistent quality or frequency?

One of my tasks at my company is to create blog entries, both on internal and external blogs. (See some of my other posts for links to what I've written.) I've noticed an interesting difference between writing for myself and writing for my company:

When I write for myself, my output is determined by when I get an idea that I want to share. Results are created by inspiration.

When I write for my company, my output is determined by our schedule. If I don't have an idea, I get one from our marcom organization, and I work at it until it makes sense, even if I personally don't see a lot of value in the content. Conversely, if I have an idea which is burning inside my head, I have to fight to get it included in our publishing schedule, with the occasional exception as a special post.

There are good reasons for scheduled output: when trying to generate a follower base, predictable frequency makes it easier for readers to know when something new will go up. It also means that, for infrequent readers, there will almost always be something new to read.

However, it's hard to train my brain to get inspired on demand. A lot of us know this feeling, and there's a fabulous write-up about it from The Oatmeal. In short, content on a schedule leads to inconsistency of output quality.

It's tempting to put together a bogus formula which asserts that quality ideas are a non-linear, which can lead either to inconsistent output frequency or to inconsistent output quality. I'll leave that as an exercise to the reader.

However, I'll also point out that there's a disconnect in perceived quality between the information producer vs the information consumer. Neil Gaiman expressed this eloquently in his Make Good Art speech. This also appears to be a common experience: I write something that I think is awful, and other people love it, and vice-versa.

Given this disconnect, I feel comfortable for now working with a content production frequency schedule. My internal discomfort at writing can spur me to produce output that's much better than I'd anticipated. It's also useful to get ideas for technical blog posts from people who have a different understanding of technology that I do, because it triggers the instinct to correct the misinformation - which leads to output on a schedule.

That being said, I still don't like the process, but I'm okay with the results.

P.S. Some readers of this post will notice that the title of this post is a question. When I started writing, I had a completely different title, but I changed it part-way through, with the implication that it's an either-or. Now, I look at it and realize that I'm compliant with Betteridge's Law, and the answer to the question is in fact "No".