Gotchas, Irritants and Warts in Go Web Development

Braintree has a tradition of trying out new languages and technology. For a recent project, we decided to try out Go, the increasingly-popular language originally developed by Rob Pike’s team at Google.

Working in Go was tons of fun; the language designers have put a lot of work into making the language a joy to use. We particularly appreciated the first-class focus the language puts on the parts of development that are uninteresting in theory but very important in practice, such as third-party library management, coding convention enforcement, and testing.

With that said, we ran into a series of gotchas, irritants, and warts while developing our application, which I’ll elaborate on below. Ultimately, these problems were severe enough that they convinced us that we couldn't justify the gains we saw from using Go -- in application speed and easy access to concurrency tools -- against the cost using Go imposed on the development team. At the moment, we are rewriting the application in Sinatra.

Overall, our impression is that while Go has a lot of promise, the language's ecosystem for developing web applications is still extremely immature. Developers thinking about putting Go web apps in production should be prepared to run into more problems than they would with an older language.

Here are some of the problems we ran into:

1. Database Library Immaturity

A few days after we deployed to our sandbox environment, an engineer on our infrastructure team noticed that the database error log for our QA environment was huge. Examining the log immediately revealed why: every few seconds, a database connection was timing out. At the time, the application was under almost no load -- we were only servicing our own internal tests. How could our queries be timing out?

After some investigation, we found the problem: one of our HA monitoring tools -- a simple "I am alive" route that our load balancing service pings once a second -- was running a SELECT 1 query to ensure that the application could connect to the database. (This is a standard practice at Braintree; you can learn more about how we run HA services with home grown and open source tools here.) This sounds fine, but Go does not return the connection to the pool until the result is checked.

We weren’t checking the value of the query, since all we wanted to do was establish database connectivity. Because of this, the connection was never returned to the pool, and it eventually timed out.

This could have been extremely bad. The database package in Go’s standard library caps the number of idle connections per pool at 2 -- new connections after those 2 are lazily generated when needed. If we hadn't caught this bug, we never would have had idle connections. We effectively would be running without a connection pool, which would have significantly slowed down IO.

This is a classic gotcha -- it’s totally undocumented, except for a slew of TODOs in the database/sql source code, and it can ruin application performance. We were very lucky to have discovered it before sending this product to production.

This wasn't the only problem we ran into with our database libraries. We started out using Gorp, a popular lightweight ORM. Unfortunately, after a few months developing with Gorp, we found a critical race condition Gorp's postgres support. While the author of Gorp was great and fixed the issue within a few days of learning about it, finding this bug in Gorp made us question the reliability of the rest of the libraries we were using.

2. Null values in your database are cumbersome and dangerous.

Go does not allow many basic types, such as strings or booleans, to be nil. Instead, when a type is initialized without a value, it defaults to the “zero value” for that type. This is frequently useful, but complicates database interactions, where null values are common.

When inserting rows, Go’s database/sql package and the pq postgres driver will insert zero values for uninitialized fields. As a side effect, in the course of normal operations, getting this data out of the database is safe. If you don’t want this behavior, the standard library provides a series of types -- NullString, NullBool, etc -- that can be configured to write as null, athough this can be cumbersome.

More dangerously, however, what happens if a migration adds a text field to a table and doesn’t specify a default value? The short answer is that any select statement on the table will fail unless the application explicitly specifies that the return value is a NullString. Gotcha! Again, this is barely documented; despite the many excellent tutorials on writing basic web applications in Go, we found very little in the way of best practices when it came to talking to the database.

If you’re willing to pollute your entire application with persistence logic, it’s possible to use NullString-style types in all your persisted structs. At Braintree, we instead adopted a policy of always specifying default values for columns on this project. This fixes the problem, but comes with its own complications -- specifically, it makes zero-downtime migrations more complicated.

3. You're forced to reinvent the wheel over and over again.

While working on this project, we were forced to either reinvent or heavily modify the Go ecosystem's solutions for logging, email alerting, route level testing, database migrations, and restarting gracefully. While some of this work was interesting, the reality is that all of these issues are solved problems in mature frameworks like Rails or Django, and after a certain point we could no longer justify continuing to invest engineering resources into this sort of work.

Where possible, we've tried to open-source the work we did on these issues. TestFlight provides a number of convenience methods for testing HTTP routes. Manners is a drop-in replacement for the standard library's HTTP server that shuts down gracefully; it can be combined with the pre-existing Goagain package to achieve Unicorn-style deployments without downtime. More projects are in the pipline.

Conclusion

While these problems make it difficult to write production-ready web applications in Go at the moment, the language itself is of such high quality that we are confident that these problems will eventually be overcome.

The point of writing this post was not to dissuade developers from using Go. Instead, we want to make sure that developers trying out this language go into it with their eyes open. Eventually, Go will be just as easy to use for web development as Ruby or Python -- but it isn’t there yet.