Heroku | New Dyno Networking Model

Today we're announcing a change to how networking on Heroku works. Dynos now get a dedicated, virtual networking interface instead of sharing a network interface with other dynos. This greatly improves the virtualization abstraction provided by Heroku and makes dynos behave more like dedicated servers or the laptop you're using to develop your app.

Background

Previously, network interfaces would be shared between a number of dynos. This broke the abstraction of a dyno as a standard Unix-style container with a network interface of its own and full disposal of the whole TCP port range.

The shared network interface also resulted in a low grade information leak where one dyno could obtain some information about connections made by other dynos. This information did not include any customer data or other customer identifying information. But it broke the core principle of tenant isolation. With the new networking model, dynos now have fully isolated network configurations. We’d like to thank John Leach for working with us to analyze this aspect of our old networking model and point out the weakness.

Networking improvements

We want Heroku dynos to resemble real machines as much as possible, with all the Heroku deployment and management goodness thrown in on top. A dyno should let you run any application or set of processes that you would run on your local machine or on an old-school server.

The new dyno networking model brings us closer to that goal: For local networking within a dyno, processes can now listen and connect on any port they want. This lets you use the local network interface to send data between processes running in a single dyno. Dyno network interfaces are firewalled off from the external network access, including from other Heroku dynos. For a web dyno, the only permitted external network access is on the TCP port specified in the $PORT environment variable. This is the port used by the Heroku router to forward HTTP requests to the dyno.

Challenges

We've been rolling out this change over the past month and it's now enabled for all apps, both new and existing. The new networking model does not require changes to applications but it has uncovered a few interesting bugs in some of the languages and frameworks used by applications running on Heroku. This is because library code now has to work correctly on systems that have a lot of (virtual) network interfaces and use network namespaces.

For example, listing network interfaces in Java using OpenJDK fails if any one network interface's index is greater than 255. Heroku engineers quickly narrowed down this problem and made a custom patched OpenJDK build available to our customers.

One problem we're still working on resolving is that Raindrops is not functional with the new networking stack. This is due to a Linux kernel bug. The bug has been addressed in the upstream kernel sources, but we’re still waiting for it show up in the branch that Heroku runs.

Check the Dev Center article on dynos for additional details.